I've got a PostgreSQL table with a column of type bytea. Porting that table from SQLite, I ran into an issue - I couldn't figure out how to pass raw binary data to an SQL query. The framework I use is PyGreSQL. I want to stick to the DB-API 2.0 interface to avoid a lot of conversion.
That interface, unlike the classic one (dollar-sign parameters) and SQLite (question-mark parameters), requires to specify the type (%-formatting like the old Python's).
The data I want to pass is a PNG file, binary-read using the 'rb' flag in the open() method.
The query code looks like this:
db = pgdb.connect(args)
c = db.cursor()
c.execute('INSERT INTO tbl VALUES (%b)', (b'test_bytes',))
This gives an error unsupported format character 'b' (0x62) at index 54 and doesn't allow the formatting to happen. What could be done to solve this issue?
Not really a solution, but a good-enough workaround. I decided to use psycopg2 instead of PyGreSQL as it is more supported and common, so it's easier to find info about any common problems.
There, the solution would be to use %s for every type, which I find much more Pythonic (no need to think about types in a dynamically-typed language)
So, my query would look like this:
c.execute('INSERT INTO tbl VALUES (%s)', (b'test_bytes',))
Related
Pretty new to sqlite3, so bear with me here..
I'd like to have a function to which I can pass the table name, and the values to update.
I initially started with something like this:
def add_to_table(table_name, string):
cursor.execute('INSERT INTO {table} VALUES ({var})'
.format(
table=table_name,
var=string)
)
Which works A-OK, but further reading about sqlite3 suggested that this was a terribly insecure way to go about things. However, using their ? syntax, I'm unable to pass in a name to specify the variable.
I tried adding in a ? in place of the table, but that throws a syntax error.
cursor.execute('INSERT INTO ? VALUES (?)', ('mytable','"Jello, world!"'))
>> >sqlite3.OperationalError: near "?": syntax error
Can the table in an sql statement be passed in safely and dynamically?
Its not the dynamic string substitution per-se thats the problem. Its dynamic string substitution with an user-supplied string thats the big problem because that opens you to SQL-injection attacks. If you are absolutely 100% sure that the tablename is a safe string that you control then splicing it into the SQL query will be safe.
if some_condition():
table_name = 'TABLE_A'
else:
table_name = 'TABLE_B'
cursor.execute('INSERT INTO '+ table_name + 'VALUES (?)', values)
That said, using dynamic SQL like that is certainly a code smell so you should double check to see if you can find a simpler alternative without the dynamically generated SQL strings. Additionally, if you really want dynamic SQL then something like SQLAlchemy might be useful to guarantee that the SQL you generate is well formed.
Composing SQL statements using string manipulation is odd not only because of security implications, but also because strings are "dumb" objects. Using sqlalchemy core (you don't even need the ORM part) is almost like using strings, but each fragment will be a lot smarter and allow for easier composition. Take a look at the sqlalchemy wiki to get a notion of what I'm talking about.
For example, using sqlsoup your code would look like this:
db = SQLSoup('sqlite://yourdatabase')
table = getattr(db, tablename)
table.insert(fieldname='value', otherfield=123)
db.commit()
Another advantage: code is database independent - want to move to oracle? Change the connection string and you are done.
I used MySQL Connector/Python API, NOT MySQLdb.
I need to dynamically insert values into a sparse table so I wrote the Python code like this:
cur.executemany("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
where
data=[('Depth', '17.5cm', Decimal('3003')), ('Input_Voltage', '110 V AC', Decimal('3004'))]
But it resulted an error:
TypeError: not enough arguments for format string
Is there any solution for this problem? Is it possible to use executemany when there is a
substitution of a field in query?
Thanks.
Let's start with the original method:
As the error message suggests you have a problem with your SQL syntax (not Python). If you insert your values you are effectively trying to execute
UPDATE myTABLE SET 'Depth'='17.5cm' WHERE id='3003'
You should notice that you are trying to assign a value to a string 'Depth', not a database field. The reason for this is that the %s substitution of the mysql module is only possible for values, not for tables/fields or other object identifiers.
In the second try you are not using the substitution anymore. Instead you use generic python string interpolation, which however looks similar. This does not work for you because you have a , and a pair of brackets too much in your code. It should read:
cur.execute("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
I also replaced executemany with execute because this method will work only for a single row. However your example only has one row, so there is no need to use executemany anyway.
The second method has some drawbacks however. The substitution is not guaranteed to be quoted or formatted in a correct manner for the SQL query, which might cause unexpected behaviour for certain inputs and may be a security concern.
I would rather ask, why it is necessary to provide the field name dynamically in the first place. This should not be necessary and might cause some trouble.
Pretty new to sqlite3, so bear with me here..
I'd like to have a function to which I can pass the table name, and the values to update.
I initially started with something like this:
def add_to_table(table_name, string):
cursor.execute('INSERT INTO {table} VALUES ({var})'
.format(
table=table_name,
var=string)
)
Which works A-OK, but further reading about sqlite3 suggested that this was a terribly insecure way to go about things. However, using their ? syntax, I'm unable to pass in a name to specify the variable.
I tried adding in a ? in place of the table, but that throws a syntax error.
cursor.execute('INSERT INTO ? VALUES (?)', ('mytable','"Jello, world!"'))
>> >sqlite3.OperationalError: near "?": syntax error
Can the table in an sql statement be passed in safely and dynamically?
Its not the dynamic string substitution per-se thats the problem. Its dynamic string substitution with an user-supplied string thats the big problem because that opens you to SQL-injection attacks. If you are absolutely 100% sure that the tablename is a safe string that you control then splicing it into the SQL query will be safe.
if some_condition():
table_name = 'TABLE_A'
else:
table_name = 'TABLE_B'
cursor.execute('INSERT INTO '+ table_name + 'VALUES (?)', values)
That said, using dynamic SQL like that is certainly a code smell so you should double check to see if you can find a simpler alternative without the dynamically generated SQL strings. Additionally, if you really want dynamic SQL then something like SQLAlchemy might be useful to guarantee that the SQL you generate is well formed.
Composing SQL statements using string manipulation is odd not only because of security implications, but also because strings are "dumb" objects. Using sqlalchemy core (you don't even need the ORM part) is almost like using strings, but each fragment will be a lot smarter and allow for easier composition. Take a look at the sqlalchemy wiki to get a notion of what I'm talking about.
For example, using sqlsoup your code would look like this:
db = SQLSoup('sqlite://yourdatabase')
table = getattr(db, tablename)
table.insert(fieldname='value', otherfield=123)
db.commit()
Another advantage: code is database independent - want to move to oracle? Change the connection string and you are done.
I'm using cx_Oracle in Python and can't get a variable be used as table name, like in this simple example:
query = "select * from some.:usertable.userinfo"
bindvars = {'usertable':usertable}
cursor.execute(query, bindvars)
What is the correct syntax? Variable substition works fine when I use WHERE… etc. but not with table names. I guess I have to separate ":usertable" somehow…
Database adapters rarely support using parameters for anything that isn't a 'value' (something that needs quoting). Either use string formatting (dodgy, you run the risk of a sql injection) or use a library like SQLAlchemy that let's you produce valid SQL using Python code.
If you are certain your usertable value is sane (checked against a list of existing table names, for example), the following would work:
query = 'select * from some.{usertable}.userinfo'.format(usertable=usertable)
You cannot bind an object name in Oracle, only a literal. Oracle does, however, have an inbuilt package dbms_assert, to help prevent SQL injection when using dynamic object names. The most useful function in your case is probably sql_object_name, which:
"... verifies that the input parameter string is a qualified SQL
identifier of an existing SQL object."
For instance you could do the following in cx_Oracle.
object_name = cursor.callfunc('sys.dbms_assert.sql_object_name'
, cx_Oracle.string, ['usertable'])
It raises ORA-44002, if the name is invalid, which you can capture in cx_Oracle, or if everything's fine continue as Martijn has suggested.
I would recommend reading Oracle's guide to guarding against SQL injection.
Perhaps it's a bit late to reply, but I was dealing with the same thing 2 days ago.
The solution is, as Martjin says, to format the query.
query = f'select * from {tableName}'
Hope it helps someone as it helped me.
Usually i use Django orm for making database related query in python but now i am using the python itself
I am trying to update a row of my mysql database
query ='UPDATE callerdetail SET upload="{0}" WHERE agent="{1}" AND custid="{2}"AND screenname="{3}" AND status="1"'.format(get.uploaded,get.agent,get.custid,get.screenname)
But i am getting the error
query ='UPDATE callerdetail SET upload="{0}" WHERE agent="{1}" AND custid="{2}"AND screenname="{3}" AND status="1"'.format(get.uploaded,get.agent,get.custid,get.screenname)
AttributeError: 'C' object has no attribute 'uploaded'
Please help me what is wrong with my query ?
Get is probably mapping to a c object. Try renaming your "get" object to something else.
Here is a list of reserved words. I don't see get in there, but it sound like it could be part of a c library that's being included. If you're including something with from x import *, you could be importing it without knowing.
In short - get probably isn't what you think it is.
However, before you go much further building SQL queries with string formatting, I strongly advise you not to! Search for "SQL injection" and you'll see why. Python DB API compliant libraries utilise "placeholders" which the library can use to insert the variables into a query for you providing any necessary escaping/quoting.
So instead of:
query ='UPDATE callerdetail SET upload="{0}" WHERE agent="{1}" AND custid="{2}"AND screenname="{3}" AND status="1"'.format(get.uploaded,get.agent,get.custid,get.screenname)
An example using SQLite3 (using ? as a placeholder - others use %s or :1 or %(name)s - or any/all of the above - but that'll be detailed in the docs of your library):
query = "update callerdetail set upload=? where agent=? and custid=? and screename=? and status=?"
Then when it comes to execute the query, you provide the values to be substituted as a separate argument:
cursor.execute(query, (get.uploaded, get.agent, get.custid, get.screenname))
If you really wanted, you could have a convenience function, and reduce this to:
from operator import attrgetter
get_fields = attrgetter('uploaded', 'agent', 'custid', 'screenname')
cursor.execute(query, get_fields(get))