Develop Reference

Python sqlalchemy and mySQL stored procedure always returns 0 (out param only)

I am trying to get the ROW_COUNT() from a MySQL stored procedure into python.
here is what I got, but I don't know what I am missing.
DELIMITER //
CREATE OR REPLACE PROCEDURE sp_refresh_mytable(
OUT row_count INT
)
BEGIN
DECLARE exit handler for SQLEXCEPTION
BEGIN
ROLLBACK;
END;
DECLARE exit handler for SQLWARNING
BEGIN
ROLLBACK;
END;
DECLARE exit handler FOR NOT FOUND
BEGIN
ROLLBACK;
END;
START TRANSACTION;
DELETE FROM mytable;
INSERT INTO mytable
(
col1
, col2
)
SELECT
col1
, col2
FROM othertable
;
SET row_count = ROW_COUNT();
COMMIT;
END //
DELIMITER ;
If I call this in via normal SQL like follows I get the correct row_count of the insert operation (e.g. 26 rows inserted):
CALL sp_refresh_mytable(#rowcount);
select #rowcount as t;
-- output: 26
Then in python/mysqlalchemy:
def call_procedure(engine, function_name, params=None):
connection = engine.raw_connection()
try:
cursor = connection.cursor()
result = cursor.callproc('sp_refresh_mytable', [0])
## try result outputs
resultfetch = cursor.fetchone()
logger.info(result)
logger.info(result[0])
logger.info(resultfetch)
cursor.close()
connection.commit()
connection.close()
logger.info(f"Running procedure {function_name} success!")
return result
except Exception as e:
logger.error(f"Running procedure {function_name} failed!")
logger.exception(e)
return None
finally:
connection.close()
So I tried logging different variations of getting the out value, but it is always 0 or None.
[INFO] db_update [0]
[INFO] db_update 0
[INFO] db_update None
What am I missing?
Thanks!

With the help of this answer I found the following solution that worked for me.
a) Working solution using engine.raw_connection() and cursor.callproc:
def call_procedure(engine, function_name):
connection = engine.raw_connection()
try:
cursor = connection.cursor()
cursor.callproc(function_name, [0])
cursor.execute(f"""SELECT #_{function_name}_0""")
results = cursor.fetchone() ## returns a tuple e.g. (285,)
rows_affected = results[0]
cursor.close()
connection.commit()
logger.info(f"Running procedure {function_name} success!")
return rows_affected
except Exception as e:
logger.error(f"Running procedure {function_name} failed!")
logger.exception(e)
return None
finally:
connection.close()
And with this answer I found this solution also:
b) Instead of using a raw connection, this worked as well:
def call_procedure(engine, function_name, params=None):
try:
with engine.begin() as db_conn:
db_conn.execute(f"""CALL {function_name}(#out)""")
results = db_conn.execute('SELECT #out').fetchone() ## returns a tuple e.g. (285,)
rows_affected = results[0]
logger.debug(f"Running procedure {function_name} success!")
return rows_affected
except Exception as e:
logger.error(f"Running procedure {function_name} failed!")
logger.exception(e)
return None
finally:
if db_conn: db_conn.close()
If there are any advantages or drawbacks of using one of these methods over the other, please let me know in a comment.

I just wanted to add another piece of code, since I was trying to get callproc to work (using sqlalchemy) with multiple in- and out-params.
For this case I went with the callproc method using a raw connection [solution b) in my previous answer], since this functions accepts params as a list.
It could probably be done more elegantly or more pythonic in some parts, but it was mainly for getting it to work and I will probably create a function from this so I can use it for generically calling a SP with multiple in and out params.
I included comments in the code below to make it easier to understand what is going on.
In my case I decided to put the out-params in a dict so I can pass it along to the calling app in case I need to react to the results. Of course you could also include the in-params which could make sense for error logging maybe.
## some in params
function_name = 'sp_upsert'
in_param1 = 'my param 1'
in_param2 = 'abcdefg'
in_param3 = 'some-name'
in_param4 = 'some display name'
in_params = [in_param1, in_param1, in_param1, in_param1]
## out params
out_params = [
'out1_row_count'
,'out2_row_count'
,'out3_row_count'
,'out4_row_count_ins'
,'out5_row_count_upd'
]
params = copy(in_params)
## adding the outparams as integers from out_params indices
params.extend([i for i, x in enumerate(out_params)])
## the params list will look like
## ['my param 1', 'abcdefg', 'some-name', 'some display name', 0, 1, 2, 3, 4]
logger.info(params)
## build query to get results from callproc (including in and out params)
res_qry_params = []
for i in range(len(params)):
res_qry_params.append(f"#_{function_name}_{i}")
res_qry = f"SELECT {', '.join(res_qry_params)}"
## the query to fetch the results (in and out params) will look like
## SELECT #_sp_upsert_0, #_sp_upsert_1, #_sp_upsert_2, #_sp_upsert_3, #_sp_upsert_4, #_sp_upsert_5, #_sp_upsert_6, #_sp_upsert_7, #_sp_upsert_8
logger.info(res_qry)
try:
connection = engine.raw_connection()
## calling the sp
cursor = connection.cursor()
cursor.callproc(function_name, params)
## get the results (includes in and out params), the 0/1 in the end are the row_counts from the sp
## fetchone is enough since all results come as on result record like
## ('my param 1', 'abcdefg', 'some-name', 'some display name', 1, 0, 1, 1, 0)
cursor.execute(res_qry)
results = cursor.fetchone()
logger.info(results)
## adding just the out params to a dict
res_dict = {}
for i, element in enumerate(out_params):
res_dict.update({
element: results[i + len(in_params)]
})
## the result dict in this case only contains the out param results and will look like
## { 'out1_row_count': 1,
## 'out2_row_count': 0,
## 'out3_row_count': 1,
## 'out4_row_count_ins': 1,
## 'out5_row_count_upd': 0}
logger.info(pformat(res_dict, indent=2, sort_dicts=False))
cursor.close()
connection.commit()
logger.debug(f"Running procedure {function_name} success!")
except Exception as e:
logger.error(f"Running procedure {function_name} failed!")
logger.exception(e)
Just to complete the picture, here is a shortened version of my stored procedure. After BEGIN I declare some error handlers I set the out params to default 0, otherwise they could also return as NULL/None if not set by the procedure (e.g. because no insert was made):
DELIMITER //
CREATE OR REPLACE PROCEDURE sp_upsert(
IN in_param1 VARCHAR(32),
IN in_param2 VARCHAR(250),
IN in_param3 VARCHAR(250),
IN in_param4 VARCHAR(250),
OUT out1_row_count INTEGER,
OUT out2_row_count INTEGER,
OUT out3_row_count INTEGER,
OUT out4_row_count_ins INTEGER,
OUT out5_row_count_upd INTEGER
)
BEGIN
-- declare variables, do NOT declare the out params here!
DECLARE dummy INTEGER DEFAULT 0;
-- declare error handlers (e.g. continue handler for not found)
DECLARE CONTINUE HANDLER FOR NOT FOUND SET dummy = 1;
-- set out params defaulting to 0
SET out1_row_count = 0;
SET out2_row_count = 0;
SET out3_row_count = 0;
SET out4_row_count_ins = 0;
SET out5_row_count_upd = 0;
-- do inserts and updates and set the outparam variables accordingly
INSERT INTO some_table ...;
SET out1_row_count = ROW_COUNT();
-- commit if no errors
COMMIT;
END //
DELIMITER ;

``Null value stopping rows from displaying

I want so show all the students in a database based on an address the user enters. Some have qualifications, some have not. Regardless of this I would still like to show their details but I am having a difficulty working around the students who have no quailifications.
The below code throws up an error where they have no qualifications but is working fine where they have qualifications.
KeyError: 'qualifications'
How can I work around this?
def call_query(address):
with MongoClient() as client:
db = client.get_database("test")
coll = db.get_collection("docs")
for col in coll.find({"details.address": { "$regex":address}}):
print (col["_id"] , ":", col["details"]["name"],":", col["details"]["age"], col["qualifications"], )

A simple way to work it around is to use a try except in your code :
def call_query(address):
with MongoClient() as client:
db = client.get_database("test")
coll = db.get_collection("docs")
for col in coll.find({"details.address": { "$regex":address}}):
try:
print (col["_id"] , ":", col["details"]["name"],":", col["details"]
["age"], col["qualifications"], )
except KeyError:
print (col["_id"] , ":", col["details"]["name"],":", col["details"]
["age"], "No qualification", )
If the exception is raised because of a None value then it will print "No qualification" for the student

In Python, a dictionary object has a get method that allows to set a default value in case the requested key is not found. So, you can simple col.get("qualifications", "") which returns empty string if there is no "qualifications" key.
Full code:
def call_query(address):
with MongoClient() as client:
db = client.get_database("test")
coll = db.get_collection("docs")
for col in coll.find({"details.address": { "$regex":address}}):
print (
col["_id"] , ":",
col["details"]["name"],
":",
col["details"]["age"],
col.get("qualifications", ""),
)

Using Neo4j Transactions with Python Bolt Driver

I have some code that successfully creates a new node using the Python Bolt Neo4j driver. However, I cannot create new relationships in the same transaction.
I am using Python 2.7 with the Neo4j Bolt drive 1.7.2.
with conn.session() as session:
uuid = getNewUUID()
tx = None
try:
tx = session.begin_transaction()
stmt = "CREATE (a:{type} {{{uuid_attrib}: $uuid, {name_attrib}: $name, {desc_attrib}: $desc, {has_phi_attrib}: $has_phi}}) RETURN a.{uuid_attrib}".format(
type=ENTITY_NODE_NAME, uuid_attrib=UUID_ATTRIBUTE,
name_attrib=NAME_ATTRIBUTE, desc_attrib=DESCRIPTION_ATTRIBUTE,
has_phi_attrib=HAS_PHI_ATTRIBUTE)
#print "EXECUTING: " + stmt
tx.run(stmt, uuid=uuid, name=name, desc=description, has_phi=hasPHI)
create_relationship(tx, uuid, DERIVED_FROM_REL, parentUUID)
create_relationship(tx, uuid, LAB_CREATED_AT_REL, labCreatedUUID)
create_relationship(tx, uuid, CREATED_BY_REL, createdByUUID)
tx.commit()
return uuid
here is the create_relationship method:
def create_relationship(tx, startuuid, rel_label, enduuid):
try:
stmt = "MATCH (a),(b) WHERE a.uuid = '$startuuid' AND b.uuid = '$enduuid' CREATE (a)-[r:{rel_label}]->(b) RETURN type(r)".format(
rel_label=rel_label)
temp_stmt = stmt
temp_stmt = temp_stmt.replace("$startuuid", startuuid)
temp_stmt = temp_stmt.replace("$enduuid", enduuid)
print "EXECUTING: " + temp_stmt
result = tx.run(stmt,startuuid=startuuid, enduuid=enduuid)
The code successfully creates the node in Neo4j. However, the relationships are never created. I expected the relationships to be added to the node. If I copy and paste the relationship CREATE commands into the bolt web interface, the CREATE command works.

I am not sure if this is the exact issue but it looks like transaction tx is passed as value and create_relationship function creates its own local copy of tx so original tx is not modified by the function create_relationship.
When you commit the tx, transactions from create_relationship function are not committed as these are not part of the tx.
You should consider running these transactions in the calling function itself instead of create_relationship, use create_relationship or similar function to create and return the statement and run these statement in the calling function.
Function to get statement:
def get_relationship_statement(startuuid, rel_label, enduuid):
stmt = "MATCH (a),(b) WHERE a.uuid = '$startuuid' AND b.uuid = '$enduuid' CREATE (a)-[r:{rel_label}]->(b) RETURN type(r)".format(
rel_label=rel_label)
temp_stmt = stmt
temp_stmt = temp_stmt.replace("$startuuid", startuuid)
temp_stmt = temp_stmt.replace("$enduuid", enduuid)
print "Statement: " + temp_stmt
return stmt
Replace
create_relationship(tx, uuid, DERIVED_FROM_REL, parentUUID)
with
tx.run(get_relationship_statement(uuid, DERIVED_FROM_REL, parentUUID),startuuid=uuid, enduuid=parentUUID)

python-ldap unable to do any basic search queries on open server

I have been trying to do some basic search queries, but I am unable to connect to an open LDAP server regardless. I tried a couple of servers, and none of them worked. I used Apache Directory Studio to make sure that the keyword was there but it did not work either way. I tried a variety of different code from different sources.
This was the first one I used
:
https://www.linuxjournal.com/article/6988
import ldap
keyword = "boyle"
def main():
server = "ldap.forumsys.com"
username = "cn=read-only-admin,dc=example,dc=com"
password = "password"
try:
l = ldap.open(server)
l.simple_bind_s(username,password)
print "Bound to server . . . "
l.protocol_version = ldap.VERSION3
print "Searching . . ."
mysearch (l,keyword)
except ldap.LDAPError:
print "Couldnt connect"
def mysearch(l, keyword):
base = ""
scope = ldap.SCOPE_SUBTREE
filter = "cn=" + "*" + keyword + "*"
retrieve_attributes = None
count = 0
result_set = []
timeout = 0
try:
result_id = l.search(base, scope, filter, retrieve_attributes)
while l != 1:
result_id = l.search(base, scope,filter, retrieve_attributes)
result_type, result_data = l.result(result_id, timeout)
if result_data == []:
break
else:
if result_type == ldap.RES_SEARCH_ENTRY:
result_set.append(result_data)
if len (result_set = 0):
print "No Results"
for i in range (len(result_set)):
for entry in result_set[i]:
try:
name = entry[1]['cn'][0]
mail = entry[1]['mail'][0]
#phone = entry[1]['telephonenumber'][0]
#desc = entry[1]['description'][0]
count = count + 1
print name + mail
except:
pass
except ldap.LDAPError, error_message:
print error_message
main()
Every time I ran this program, I received an error
{'desc': u"No such object"}
I also tried this
import ldap
try:
l = ldap.open("ldap.example.com")
except ldap.LDAPError, e:
print e
base_dn = "cn=read-only-admin,dc=example,dc=com"
search_scope = ldap.SCOPE_SUBTREE
retrieve_attributes = None
search_filter = "uid=myuid"
try:
l_search = l.search(base_dn, search_scope, search_filter, retrieve_attributes)
result_status, result_data = l.result(l_search, 0)
print result_data
except ldap.LDAPError, e:
print e
The error on this one was
{'desc': u"Can't contact LDAP server"}
I spent about 5 hours trying to figure this out. I would really appreciate it if you guys could give me some advice. Thanks.

There are several bogus things in there.
I will only comment your first code sample because it can be used by anyone with that public LDAP server.
l = ldap.open(server)
Function ldap.open() is deprecated since many years. You should use function ldap.initialize() with LDAP URI as argument instead like this:
l = ldap.initialize("ldap://ldap.forumsys.com")
l_search = l.search(..)
This is the asynchronous method which just returns a message ID (int) of the underlying OpenLDAP C API (libldap). It's needed if you want to retrieve extended controls returned by the LDAP server along with search results. Is that what you want?
As a beginner you probably want to use the simpler method LDAPObject.search_s() which immediately returns a list of (DN, entry) 2-tuples.
See also: python-ldap -- Sending LDAP requests
while l != 1
This does not make sense at all because l is your LDAPObject instance (LDAP connection object). Note that LDAPObject.search() would raise an exception if it gets an Integer error code from OpenLDAP's libldap. No need to do C-style error checks at this level.
filter = "cn=" + "" + keyword + ""
If keyword can be arbitrary input this is a prone to LDAP injection attacks. Don't do that.
For adding arbitrary input into a LDAP filter use function ldap.filter.escape_filter_chars() to properly escape special characters. Also avoid using variable name filter because it's the name of a built-in Python function and properly enclose the filter in parentheses.
Better example:
ldap_filter = "(cn=*%s*)" % (ldap.filter.escape_filter_chars(keyword))
base = ""
The correct search base you have to use is:
base = "dc=example,dc=com"
Otherwise ldap.NO_SUCH_OBJECT is raised.
So here's a complete example:
import pprint
import ldap
from ldap.filter import escape_filter_chars
BINDDN = "cn=read-only-admin,dc=example,dc=com"
BINDPW = "password"
KEYWORD = "boyle"
ldap_conn = ldap.initialize("ldap://ldap.forumsys.com")
ldap_conn.simple_bind_s(BINDDN, BINDPW)
ldap_filter = "(cn=*%s*)" % (ldap.filter.escape_filter_chars(KEYWORD))
ldap_results = ldap_conn.search_s(
"dc=example,dc=com",
ldap.SCOPE_SUBTREE,
ldap_filter,
)
pprint.pprint(ldap_results)

Django Model use of keyword IN

I am attempting to figure out how to use the IN keyword in a django model query.
I was attempting to replace:
db = database.connect()
c = db.cursor()
c.execute("SELECT MAX(Date) FROM `Requests` WHERE UserId = %%s AND VIN = %%s AND Success = 1 AND RptType in %s" % str(cls.SuperReportTypes), (userID, vin))
With this:
myrequests = Request.objects.filter(user=userID, vin = vin, report_type in cls.SuperReportTypes)
myrequests.aggregate(Max('Date'))
I get a:
SyntaxError: non-keyword arg after keyword arg (<console>, line 1)
When I remove the ending "report_type in cls.SuperReportTypes" the query functions properly.
I recognize that there is a way to do this after the query managing the result set but I was hoping to deal with this in such a way that MYSQL would do the execution.

field__in=seq

You are using the wrong in statement
https://docs.djangoproject.com/en/dev/ref/models/querysets/#in