I'm using the neo4j Python driver to run batched data loads on a local Neo4j database. I have the following packages:
neo4j==4.4.1
neo4j-driver==4.4.1
I am using the apoc.periodic.iterate method. The call returns a Result object which contains a small dictionary with some data about the load. It looks like this:
{'batches': 1,
'total': 9,
'timeTaken': 0,
'committedOperations': 9,
...}
When the load is very small, I can extract this object from Result and save it. When it is larger, however, I cannot work with the Result object. I am able to print the address of the object. But if I try to run any method on it, or extract data from it in anyway, or return it from the function, my code hangs forever. However, the return data should always be the same size, because it's just a bit of metadata about the load.
from neo4j import GraphDatabase
driver = GraphDatabase.driver(URI, auth=(user_name, password))
address = "test.csv"
cql = '''
CALL apoc.periodic.iterate(
"LOAD CSV WITH HEADERS FROM '%s' AS row
WITH
row.field1 AS field1,
toFloat(row.field2) AS field2
RETURN *",
"MERGE (tx: Object {field1: field1})
SET
tx.field1 = field1,
tx.field2 = field2;
",
{batchSize: 10000, parallel: true, retries: 3})
''' % address
with driver.session() as session:
result = session.run(cql)
print(result)
log_data = result.data()[0] # this line hangs forever with large loads
Well, the problem disappears when switching from auto-commit transaction to transaction functions. So instead of the approach above, I did:
def transaction(tx):
return dict(tx.run(cql).single())
with driver.session() as session:
log_data = session.write_transaction(transaction)
https://neo4j.com/docs/driver-manual/1.7/sessions-transactions/
Related
How can I call stored procedures of sql server with sqlAlchemy?
Engines and Connections have an execute() method you can use for arbitrary sql statements, and so do Sessions. For example:
results = sess.execute('myproc ?, ?', [param1, param2])
You can use outparam() to create output parameters if you need to (or for bind parameters use bindparam() with the isoutparam=True option)
context: I use flask-sqlalchemy with MySQL and without ORM-mapping. Usually, I use:
# in the init method
_db = SqlAlchemy(app)
#... somewhere in my code ...
_db.session.execute(query)
Calling stored procedures is not supported out of the box: the callproc is not generic, but specific to the mysql connector.
For stored procedures without out params, it is possible to execute a query like
_db.session.execute(sqlalchemy.text("CALL my_proc(:param)"), param='something')
as usual. Things get more complicated when you have out params...
One way to use out params is to access the underlying connector is through engine.raw_connection(). For example:
conn = _db.engine.raw_connection()
# do the call. The actual parameter does not matter, could be ['lala'] as well
results = conn.cursor().callproc('my_proc_with_one_out_param', [0])
conn.close() # commit
print(results) # will print (<out param result>)
This is nice since we are able to access the out parameter, BUT this connection is not managed by the flask session. This means that it won't be committed/aborted as with the other managed queries... (problematic only if your procedure has side-effect).
Finally, I ended up doing this:
# do the call and store the result in a local mysql variabl
# the name does not matter, as long as it is prefixed by #
_db.session.execute('CALL my_proc_with_one_out_param(#out)')
# do another query to get back the result
result = _db.session.execute('SELECT #out').fetchone()
The result will be a tuple with one value: the out param. This is not ideal, but the least dangerous: if another query fails during the session, the procedure call will be aborted (rollback) as well.
Just execute procedure object created with func:
from sqlalchemy import create_engine, func
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite://', echo=True)
print engine.execute(func.upper('abc')).scalar() # Using engine
session = sessionmaker(bind=engine)()
print session.execute(func.upper('abc')).scalar() # Using session
The easiest way to call a stored procedure in MySQL using SQLAlchemy is by using callproc method of Engine.raw_connection(). call_proc will require the procedure name and parameters required for the stored procedure being called.
def call_procedure(function_name, params):
connection = cloudsql.Engine.raw_connection()
try:
cursor = connection.cursor()
cursor.callproc(function_name, params)
results = list(cursor.fetchall())
cursor.close()
connection.commit()
return results
finally:
connection.close()
Supposing you already have session created with sessionmaker(), you can use following function:
def exec_procedure(session, proc_name, params):
sql_params = ",".join(["#{0}={1}".format(name, value) for name, value in params.items()])
sql_string = """
DECLARE #return_value int;
EXEC #return_value = [dbo].[{proc_name}] {params};
SELECT 'Return Value' = #return_value;
""".format(proc_name=proc_name, params=sql_params)
return session.execute(sql_string).fetchall()
Now you can execute your stored procedure 'MyProc' with parameters simply like that:
params = {
'Foo': foo_value,
'Bar': bar_value
}
exec_procedure(session, 'MyProc', params)
Out of desperate need for a project of mine, I wrote a function that handles Stored Procedure calls.
Here you go:
import sqlalchemy as sql
def execute_db_store_procedure(database, types, sql_store_procedure, *sp_args):
""" Execute the store procedure and return the response table.
Attention: No injection checking!!!
Does work with the CALL syntax as of yet (TODO: other databases).
Attributes:
database -- the database
types -- tuple of strings of SQLAlchemy type names.
Each type describes the type of the argument
with the same number.
List: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html
sql_store_procudure -- string of the stored procedure to be executed
sp_args -- arguments passed to the stored procedure
"""
if not len(types) == len(sp_args):
raise ValueError("types tuple must be the length of the sp args.")
# Construch the type list for the given types
# See
# http://docs.sqlalchemy.org/en/latest/core/sqlelement.html?highlight=expression.text#sqlalchemy.sql.expression.text
# sp_args (and their types) are numbered from 0 to len(sp_args)-1
type_list = [sql.sql.expression.bindparam(
str(no), type_=getattr(sql.types, typ)())
for no, typ in zip(range(len(types)), types)]
try:
# Adapts to the number of arguments given to the function
sp_call = sql.text("CALL `%s`(%s)" % (
sql_store_procedure,
", ".join([":%s" % n for n in range(len(sp_args))])),
bindparams=type_list
)
#raise ValueError("%s\n%s" % (sp_call, type_list))
with database.engine.begin() as connection:
return connection.execute(
sp_call,
# Don't do this at home, kids...
**dict((str(no), arg)
for (no, arg) in zip(range(len(sp_args)), sp_args)))
except sql.exc.DatabaseError:
raise
It works with the CALL syntax, so MySQL should work as expected. MSSQL uses EXEC instead of call and a little differennt syntax, I guess. So making it server agnostic is up to you but shouldn’t be too hard.
Another workaround:
query = f'call Procedure ("{#param1}", "{#param2}", "{#param3}")'
sqlEngine = sqlalchemy.create_engine(jdbc)
conn = sqlEngine.connect()
df = pd.read_sql(query,conn,index_col=None)
I had a stored procedure for postgresql with following signature -
CREATE OR REPLACE PROCEDURE inc_run_count(
_host text,
_org text,
_repo text,
_rule_ids text[]
)
After quite a few error and trial, I found this is how to call the procedure from python3.
def update_db_rule_count(rule_ids: List[str], host: str, org: str, repo: str):
param_dict = {"host": host, "org": org, "repo": repo, "rule_ids": f'{{ {",".join(rule_ids)} }}'}
with AnalyticsSession() as analytics_db:
analytics_db.execute('call inc_run_count(:host, :org, :repo, :rule_ids)', param_dict)
analytics_db.commit()
I created my tables in Microsoft SQL Server Management Studio and then I created the connection with the database in Python.
I can fetch the whole records of a table which means that I do not have any problem with the connection.
Yet I want to create a login system and I am trying to compare the input which is entered by the user and the database record for login system.
Here is my code
def read(conn,name,password):
cursor = conn.cursor()
cursor.execute("select * from UserList")
if(name==cursor["Username"] & password == cursor["Password"]) :
print("A")
Hence, when I run it, it gives an error as
File "C:/Users/PycharmProjects/MyProject/Project", line 18, in <module>
read(conn,name,password)
File "C:/Users/PycharmProjects/MyProject/Project", line 6, in read
if(name==cursor["Username"] & password == cursor["Password"]) :
TypeError: 'pyodbc.Cursor' object is not subscriptable
What is the problem and which guide I should follow to solve this?
You need to use one of these cursor functions to load the results of the query into memory.
cursor.execute("select * from UserList")
# get all results
results = cursor.fetchall()
# get one result
results = cursor.fetchone()
# get multiple results in chunks
results = cursor.fetchmany(10)
Reference:
https://github.com/mkleehammer/pyodbc/wiki/Cursor
"fetchone" & "fetchmany" will load the results into memory in chunks. You would then need to loop until there are no more records available.
while True:
results = cursor.fetchmany(10)
if len(results)==0:
break
I am trying to retrieve data from a mysql server, run a code using the data retrieved, then push the data back to the database on different columns. I have managed to get all of that working as seen in my code attached.
However, new rows will be added constantly to the database and i don't want to have to load the entire database everytime. I want the code to only run on the new rows that have been added. Therefore, i am trying save the student_id and retrieve it somehow and the beginning of the code.
I tried different methods and i even tried pickle but i was left with little luck. The aim is to have the code run continuously on a remote server so i would really appreciate if someone can help me out as it will help me achieve my task.
Note: This is only dummy data and code but is pretty much the method of what i am trying to achieve.
import mysql.connector
import numpy as np
import time
import pickle
filename = 'saved_id'
infile = open(filename,'rb')
st_id = pickle.load(infile)
print(number)
conn = mysql.connector.connect(user="user", password='password',
host="demo.mysql.database.azure.com", port=3306)
conn.autocommit = True
cursor = conn.cursor()
cursor.execute("USE testdb;")
while True:
outfile = open(filename,'wb')
cursor.execute(f"SELECT * FROM student WHERE student_id >= {st_id};")
myresult = np.array(cursor.fetchall())
print (myresult)
for row in myresult:
mark_1 = np.random.randint(0,100)
mark_2 = np.random.randint(0,100)
mark_3 = np.random.randint(0,100)
iid = row[0]
pickle.dump(iid,outfile)
outfile.close()
cursor.execute(f"UPDATE student SET grade_1 = {mark_1}, grade_2 = {mark_2}, grade_3 = {mark_3} WHERE student_id = {iid};")
time.sleep(5)
I Think the problem is that when i initially run the code the pickle value doesn't have an input, and when i try to use a standard variable method it will not be defined which will result in some kind of error as well.
Please also do let me know if there are any aspects of my code that could be improved for speed, efficiency!
I have a php script that executes a python script and returns the data back and it is stored in mysql. It's working fine but when the data is stored in the database it inserts an additional blank row. My question would be, how could I make so that it stores only the actual data I recieve.
This is part of the python script
##ser.write(sync)
ser.write(counters)
a = ser.read(30)
state = binascii.hexlify(a)
asd = re.sub(rb'([0-9, a-z, A-Z])(?!$)', rb'\1,', state)
url = 'http://127.0.0.1/sastest/meters.php'
x = requests.post(url, data = asd)
print(asd)
And this is from the PHP
passthru("meters.py");
$incomingData = file_get_contents("php://input");
$qry1 = "INSERT INTO machtest(data)
values('".$incomingData."')";
mysqli_query($conn,$qry1);
From comments we discover the overall process:
When I call meters.php it activates meters.py. meters.py interrogates a devices and sends the data back to meters.php
Because PHP's passthru does not support output return but a similar function exec does as array object with each line as elements, use that instead and do not have Python post back a response. Of course, always run parameterization when interacting with databases and passing input values.
Python (meters.py)
ser.write(counters)
a = ser.read(30)
state = binascii.hexlify(a)
asd = re.sub(rb'([0-9, a-z, A-Z])(?!$)', rb'\1,', state)
print(asd)
PHP (meters.php)
// USE output ARG
exec(command = "meters.py", output = $incomingData);
// USE PARAMETERIZATION
$qry = "INSERT INTO machtest (data) VALUES (%s)";
$stmt = mysqli_prepare($qry);
mysqli_stmt_bind_param($stmt, "s", $incomingData[0]);
mysqli_stmt_execute($stmt);
See mysqli prepared statement docs
Alternatively, have Python run all processing including device and database interaction. Then, have PHP call the .py script:
Python (meters.py)
import mysql.connector # USE ANY MySQL DB-API. THIS IS AN EXAMPLE
...
### INTERROGATE DEVICE
ser.write(counters)
a = ser.read(30)
state = binascii.hexlify(a)
asd = re.sub(rb'([0-9, a-z, A-Z])(?!$)', rb'\1,', state)
### APPEND TO DATABASE
# OPEN CONNECTION AND CURSOR
conn = mysql.connector.connect(host='localhost', database='mydatabase',
user='root', password='pwd')
cur = conn.cursor()
# USE PARAMETERIZATION
qry = "INSERT INTO machtest (data) VALUES (%s)"
cur.execute(qry, (asd,))
conn.commit()
cur.close()
conn.close()
See MySQL cursor execute docs
PHP (meters.php)
// NO NEED FOR output
passthru(command = "meters.py");
I have a celery project connected to a MySQL databases. One of the tables is defined like this:
class MyQueues(Base):
__tablename__ = 'accepted_queues'
id = sa.Column(sa.Integer, primary_key=True)
customer = sa.Column(sa.String(length=50), nullable=False)
accepted = sa.Column(sa.Boolean, default=True, nullable=False)
denied = sa.Column(sa.Boolean, default=True, nullable=False)
Also, in the settings I have
THREADS = 4
And I am stuck in a function in code.py:
def load_accepted_queues(session, mode=None):
#make query
pool = session.query(MyQueues.customer, MyQueues.accepted, MyQueues.denied)
#filter conditions
if (mode == 'XXX'):
pool = pool.filter_by(accepted=1)
elif (mode == 'YYY'):
pool = pool.filter_by(denied=1)
elif (mode is None):
pool = pool.filter(\
sa.or_(MyQueues.accepted == 1, MyQueues.denied == 1)
)
#generate a dictionary with data
for i in pool: #<---------- line 90 in the error
l.update({i.customer: {'customer': i.customer, 'accepted': i.accepted, 'denied': i.denied}})
When running this I get an error:
[20130626 115343] Traceback (most recent call last):
File "/home/me/code/processing/helpers.py", line 129, in wrapper
ret_value = func(session, *args, **kwargs)
File "/home/me/code/processing/test.py", line 90, in load_accepted_queues
for i in pool: #generate a dictionary with data
File "/home/me/envs/me/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2341, in instances
fetch = cursor.fetchall()
File "/home/me/envs/me/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 3205, in fetchall
l = self.process_rows(self._fetchall_impl())
File "/home/me/envs/me/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 3174, in _fetchall_impl
self._non_result()
File "/home/me/envs/me/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 3179, in _non_result
"This result object does not return rows. "
ResourceClosedError: This result object does not return rows. It has been closed automatically
So mainly it is the part
ResourceClosedError: This result object does not return rows. It has been closed automatically
and sometimes also this error:
DBAPIError: (Error) (, AssertionError('Result length not requested
length:\nExpected=1. Actual=0. Position: 21. Data Length: 21',))
'SELECT accepted_queues.customer AS accepted_queues_customer,
accepted_queues.accepted AS accepted_queues_accepted,
accepted_queues.denied AS accepted_queues_denied \nFROM
accepted_queues \nWHERE accepted_queues.accepted = %s OR
accepted_queues.denied = %s' (1, 1)
I cannot reproduce the errror properly as it normally happens when processing a lot of data. I tried to change THREADS = 4 to 1 and errors disappeared. Anyway, it is not a solution as I need the number of threads to be kept on 4.
Also, I am confused about the need to use
for i in pool: #<---------- line 90 in the error
or
for i in pool.all(): #<---------- line 90 in the error
and could not find a proper explanation of it.
All together: any advise to skip these difficulties?
All together: any advise to skip these difficulties?
yes. you absolutely cannot use a Session (or any objects which are associated with that Session), or a Connection, in more than one thread simultaneously, especially with MySQL-Python whose DBAPI connections are very thread-unsafe*. You must organize your application such that each thread deals with it's own, dedicated MySQL-Python connection (and therefore SQLAlchemy Connection/ Session / objects associated with that Session) with no leakage to any other thread.
Edit: alternatively, you can make use of mutexes to limit access to the Session/Connection/DBAPI connection to just one of those threads at a time, though this is less common because the high degree of locking needed tends to defeat the purpose of using multiple threads in the first place.
I got the same error while making a query to SQL-Server procedure using SQLAlchemy.
In my case, adding SET NOCOUNT ON to the stored procedure fixed the problem.
ALTER PROCEDURE your_procedure_name
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for your procedure here
SELECT *
FROM your_table_name;
END;
Check out this article for more details
I was using an INSERT statment. Adding
RETURNING id
at the end of the query worked for me. As per this issue
That being said it's a pretty weird solution, maybe something fixed in later versions of SQLAlchemy, I am using 1.4.39.
This error occurred for me when I used a variable in Python
and parsed it with an UPDATE
statement using pandas pd.read_sql()
Solution:
I simply used mycursor.execute() instead of pd.read_sql()
import mysql.connector and from sqlalchemy import create_engine
Before:
pd.read_sql("UPDATE table SET column = 1 WHERE column = '%s'" % variable, dbConnection)
After:
mycursor.execute("UPDATE table SET column = 1 WHERE column = '%s'" % variable)
Full code:
import mysql.connector
from sqlalchemy import create_engine
import pandas as pd
# Database Connection Setup >
sqlEngine = create_engine('mysql+pymysql://root:root#localhost/db name')
dbConnection = sqlEngine.connect()
db = mysql.connector.connect(
host="localhost",
user="root",
passwd="root",
database="db name")
mycursor = db.cursor()
variable = "Alex"
mycursor.execute("UPDATE table SET column = 1 WHERE column = '%s'" % variable)
For me I got this error when I forgot to write the table calss name for the select function query = select().where(Assessment.created_by == assessment.created_by) so I had only to fix this by adding the class table name I want to get entries from like so:
query = select(Assessment).where(
Assessment.created_by == assessment.created_by)