I'm trying to check if database exists in Python. I found that it can be done using the following sql statement: cur.execute(f'SHOW DATABASES LIKE {event["db_name"]}') However,after trying to execute it I get the following errors:
[ERROR] ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'TestLAU' at line 1")
Traceback (most recent call last):
File "/var/task/handler.py", line 63, in handler
cur.execute(f'SHOW DATABASES LIKE {event["db_name"]}')
File "/var/task/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/var/task/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/var/task/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/var/task/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/var/task/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/var/task/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/var/task/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/var/task/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
You miss an apostrophe in your statement and you can use % in your LIKE statement in order to match more results, try the following:
cur.execute(f"SHOW DATABASES LIKE '%{event["db_name"]}%;'")
more about LIKE operator:
LIKE 'a%' Finds any values that start with "a"
LIKE '%a' Finds any values that end with "a"
LIKE '%or%' Finds any values that have "or" in any position
LIKE '_r%' Finds any values that have "r" in the second position
LIKE 'a__%' Finds any values that start with "a" and are at least 3 characters in length
LIKE 'a%o' Finds any values that start with "a" and ends with "o"
Turns out the error was caused by the fact that I needed to surround the database name with quotes so the following SQL statement worked: cur.execute(f'SHOW DATABASES LIKE "{event["db_name"]}"')
Related
I am trying to update all rows of the specific column in MSSQL database using pymssql. But I encountered with this error:
" Traceback (most recent call last):
File "src\pymssql_pymssql.pyx", line 460, in pymssql._pymssql.Cursor.execute
File "src\pymssql_mssql.pyx", line 1104, in pymssql._mssql.MSSQLConnection.execute_query
File "src\pymssql_mssql.pyx", line 1135, in pymssql._mssql.MSSQLConnection.execute_query
File "src\pymssql_mssql.pyx", line 1268, in pymssql._mssql.MSSQLConnection.format_and_run_query
File "src\pymssql_mssql.pyx", line 1806, in pymssql._mssql.check_cancel_and_raise
File "src\pymssql_mssql.pyx", line 1852, in pymssql._mssql.raise_MSSQLDatabaseException
pymssql._mssql.MSSQLDatabaseException: (102, b"Incorrect syntax near ','.DB-Lib error message 20018, severity 15:\nGeneral SQL Server error: Check messages from the SQL Server\n") "
Here is my code:
params = [tuple(x) for x in all_list]
# print(params)
query = """UPDATE dbo.MYTABLE SET KeyWords=%s WHERE KeyWords=%s"""
for i in params:
val = ("NULL", i)
cursor.execute(query, val)
conn.commit()
print(cursor.rowcount)
Params variable is a list of tuples that are going to be the new values of my column. In the column, all values are filled with NULL and I want to update with the Params variable.
Any suggestion?
Thank you for help, in advance.
I am running a Pyramid + Zope transaction manager + SQLAlchemy + PostgreSQL. On some occasions, I have seen StaleDataError error on a Pyramid web application which should very trivial view of updating one row in a database. As the error happens outside the normal view boundary and is not repeatable, it is quite tricky to debug.
I guess this might have something to do with broken database connections or transaction lifecycle. However I don't know how to start debugging the system, so I am asking what could cause this and furthermore how one can pin down errors like this.
UPDATE statement on table 'xxx' expected to update 1 row(s); 0 were matched.
Stacktrace (most recent call last):
File "pyramid/tweens.py", line 20, in excview_tween
response = handler(request)
File "pyramid_tm/__init__.py", line 94, in tm_tween
reraise(*exc_info)
File "pyramid_tm/compat.py", line 15, in reraise
raise value
File "pyramid_tm/__init__.py", line 82, in tm_tween
manager.commit()
File "transaction/_manager.py", line 111, in commit
return self.get().commit()
File "transaction/_transaction.py", line 280, in commit
reraise(t, v, tb)
File "transaction/_compat.py", line 55, in reraise
raise value
File "transaction/_transaction.py", line 271, in commit
self._commitResources()
File "transaction/_transaction.py", line 417, in _commitResources
reraise(t, v, tb)
File "transaction/_compat.py", line 55, in reraise
raise value
File "transaction/_transaction.py", line 389, in _commitResources
rm.tpc_begin(self)
File "/srv/pyramid/trees/venv/lib/python3.4/site-packages/zope/sqlalchemy/datamanager.py", line 90, in tpc_begin
self.session.flush()
File "sqlalchemy/orm/session.py", line 2004, in flush
self._flush(objects)
File "sqlalchemy/orm/session.py", line 2122, in _flush
transaction.rollback(_capture_exception=True)
File "sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "sqlalchemy/util/compat.py", line 182, in reraise
raise value
File "sqlalchemy/orm/session.py", line 2086, in _flush
flush_context.execute()
File "sqlalchemy/orm/unitofwork.py", line 373, in execute
rec.execute(self)
File "sqlalchemy/orm/unitofwork.py", line 532, in execute
uow
File "sqlalchemy/orm/persistence.py", line 170, in save_obj
mapper, table, update)
File "sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
(table.description, len(records), rows))
This is most likely scenario:
You have 2 requests that first select an object and try to update/delete it in the datastore and you end up with a "race condition".
Lets say you want to do something like fetch an object and then update it.
If transaction takes some time and you do not select the object with "for update" thus locking the rows - if the object gets deleted in first request and 2nd transaction tries to issue update to row that is not present in the db anymore you can end up with this exception.
You can try doing some row locking to prevent this from happening - subsequent transaction will "wait" for the first operation to finish. Before it gets executed.
http://docs.sqlalchemy.org/en/rel_1_0/orm/query.html?highlight=for_update#sqlalchemy.orm.query.Query.with_for_update
and
http://docs.sqlalchemy.org/en/rel_1_0/orm/query.html?highlight=with_lockmode#sqlalchemy.orm.query.Query.with_lockmode
Describe some of the sqlalchemy machinery you can use to resolve this.
Another option:
TL,DR: if you have "first()" in some cases, you need to remove this in alchemy if you update multiple records
db.session.query(xxx).filter_by(field=value).first()
This command expects the update to affect only one line. And it should if your table has only one record with field=value. This is particularly the case if the field is your ID.
HOWEVER - if your ID is not unique, you might have multiple records with the same ID.
In this case, your can update all by removing "first()"
BTW, use the following to debug your SQL queries (which wouldn't have helped this time...)
import logging
logging.basicConfig()
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
I am using pandas.read_sql function with hive connection to extract a really large data. I have a script like this:
df = pd.read_sql(query_big, hive_connection)
df2 = pd.read_sql(query_simple, hive_connection)
The big query take a long time, and after it is executed, python returns the following error when trying to execute the second line:
raise NotSupportedError("Hive does not have transactions") # pragma: no cover
It seems there is something wrong with the connection.
Moreover, If I replace the second line with multirpocessing.Manager().Queue(), It returns the following error:
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 662, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 554, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
It seems this kind of error are related to exit function being messed up, in the connection.py. Moreover, when I changed the query in the first command to extract smaller data that doesn't take too long, Everything works fine. So I assume it may be that because it takes too long to execute the first query, something is improperly terminated. which caused the two error, both of which are so different in nature but both are related to broken connection issues.
I'm trying to correctly escape urls to enter into a mysql connection, but apparently I'm doing it wrong:
>>> import MySQLdb
>>> db = MySQLdb.connect(host="localhost",user="...",passwd="...",db="...")
>>> cur = db.cursor()
>>> cmd = "insert into S3_data (url) VALUES ('http://google.com')"
>>> cur.execute(cmd)
1
>>> cur.execute(MySQLdb.escape_string(cmd))
Traceback (most recent call last):
File "<pyshell#49>", line 1, in <module>
cur.execute(MySQLdb.escape_string(cmd))
File "C:\Python34\lib\site-packages\MySQLdb\cursors.py", line 220, in execute
self.errorhandler(self, exc, value)
File "C:\Python34\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
raise errorvalue
File "C:\Python34\lib\site-packages\MySQLdb\cursors.py", line 209, in execute
r = self._query(query)
File "C:\Python34\lib\site-packages\MySQLdb\cursors.py", line 371, in _query
rowcount = self._do_query(q)
File "C:\Python34\lib\site-packages\MySQLdb\cursors.py", line 335, in _do_query
db.query(q)
File "C:\Python34\lib\site-packages\MySQLdb\connections.py", line 280, in query
_mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '\\'http://google.com\\')' at line 1")
As you can see the command works ok, but the escaping fails. What am I missing here?
Also, does escape string handle multi-byte encodings?
Thanks
I believe if you use parameters you shouldn't have a problem.
cmd = "INSERT INTO S3_data (url) VALUES (%s)"
args = 'http://google.com'
cur.execute(cmd, args)
I'm trying to build a custom query and would like to apply .extra() after filter(). The statement looks like:
V.objects.filter(v_id__product__icontains=name)
Now it produces the valid SQL, which, however, does not have quotes around name:
WHERE `v_id`.`product` LIKE %xxx%
But when I add .extra() statement:
V.objects.filter(id__product__icontains=name).extra(where=[concat_str],params=[version,'%','%'])
, the query becomes invalid because there is no quotes aroung %xxx%:
WHERE `v_id`.`product` LIKE %xxx% AND 'yyy' LIKE CONCAT('%',version,'%')
All I need here is to add single quotes around %xxx%, to make it valid:
WHERE `vulnerabilities_cpeid`.`product` LIKE '%xxx%' AND 'yyy' LIKE CONCAT('%',version,'%')
However, I just do not know how to force Django to put %xxx% into single quotes when using icontains. Any help appreciated. Thank you in advance.
Full traceback:
INFO 2014-07-21 11:33:55,515 views: SELECT `vulnerabilities_vulnerability`.`identifier` FROM `vulnerabilities_vulnerability` INNER JOIN `vulnerabilities_vulnerability_cpe_id` ON (`vulnerabilities_vulnerability`.`id` = `vulnerabilities_vulnerability_cpe_id`.`vulnerability_id`) INNER JOIN `vulnerabilities_cpeid` ON (`vulnerabilities_vulnerability_cpe_id`.`cpeid_id` = `vulnerabilities_cpeid`.`id`) WHERE (`vulnerabilities_cpeid`.`product` LIKE %accountsservice% AND '0.6.15-2ubuntu9.7' LIKE CONCAT('%',version,'%'))
ERROR 2014-07-21 11:33:55,517 django.request: Internal Server Error: /vulndb/inventory/
Traceback (most recent call last):
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/sapegin/vulndb_mercurial/vulndb/vulndb/vulnerabilities/views.py", line 1650, in inventory
if ((vulnerabilities is not None) and (vulnerabilities.count() > 0)):
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/db/models/query.py", line 351, in count
return self.query.get_count(using=self.db)
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/db/models/sql/query.py", line 418, in get_count
number = obj.get_aggregation(using=using)[None]
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/db/models/sql/query.py", line 384, in get_aggregation
result = query.get_compiler(using).execute_sql(SINGLE)
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 818, in execute_sql
cursor.execute(sql, params)
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 119, in execute
return self.cursor.execute(query, args)
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/MySQL_python-1.2.4-py2.6-linux-x86_64.egg/MySQLdb/cursors.py", line 201, in execute
self.errorhandler(self, exc, value)
File "/home/sapegin/vulndb_mercurial/vulndb/HANA/PYTHON/Python/lib/python2.6/site-packages/MySQL_python-1.2.4-py2.6-linux-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
DatabaseError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0.6.15-2ubuntu9.7'' LIKE CONCAT(''%'',version,''%''))' at line 1")
ERROR 2014-07-21 11:33:55,517 django.request: Internal Server Error: /vulndb/inventory/
I bet the error is that you have extra quotes in your .extra call's parameters. Remove the quotes around %s and see if that fixes the problem.
In Django the DatabaseWrapper will automatically add quotes there. The 'icontains' operator is translated to 'icontains': 'LIKE %s', where %s will be replaced with the string which is concatenated with % signs before and after the search term.
Internally Django will use the quote_name function to do the quotation marks.
def quote_name(self, name):
if name.startswith("`") and name.endswith("`"):
return name # Quoting once is enough.
return "`%s`" % name
Maybe for your use-case the raw Query might be a better Solution:
Model.objects.raw('Select .... FROM .... WHERE ....', params=None, translations=None)
Have a look at: https://docs.djangoproject.com/en/dev/topics/db/sql/
I think alecxe is right the debug in .query is not sufficient.