I'm getting a weird error when inserting some data from a Python script to MySQL. It's basically related to a variable being blank that I am inserting. I take it that MySQL does not like blank variables but is there something else I can change it to so it works with my insert statement?
I can successfully use an IF statement to turn it to 0 if its blank but this may mess up some of the data analytics I plan to do in MySQL later. Is there a way to convert it to NULL or something so MySQL accepts it but doesn't add anything?
When using mysqldb and cursor.execute(), pass the value None, not "NULL":
value = None
cursor.execute("INSERT INTO table (`column1`) VALUES (%s)", (value,))
Found the answer here
if the col1 is char, col2 is int, a trick could be:
insert into table (col1, col2) values (%s, %s) % ("'{}'".format(val1) if val1 else "NULL", val2 if val2 else "NULL");
you do not need to add ' ' to %s, it could be processed before pass value to sql.
this method works when execute sql with session of sqlalchemy, for example session.execute(text(sql))
ps: sql is not tested yet
Quick note about using parameters in SQL statements with Python. See the RealPython article on this topic - Preventing SQL Injection Attacks With Python. Here's another good article from TowardsDataScience.com - A Simple Approach To Templated SQL Queries In Python. These helped me with same None/NULL issue.
Also, I found that if I put "NULL" (without quotes) directly into the INSERT query in VALUES, it was interpreted appropriately in the SQL Server DB. The translation problem only exists if needing to conditionally add NULL or a value via string interpolation.
Examples:
cursor.execute("SELECT admin FROM users WHERE username = %s'", (username, ));
cursor.execute("SELECT admin FROM users WHERE username = %(username)s", {'username': username});
UPDATE: This StackOverflow discussion is more in line with what I'm trying to do and may help someone else.
Example:
import pypyodbc
myData = [
(1, 'foo'),
(2, None),
(3, 'bar'),
]
connStr = """
DSN=myDb_SQLEXPRESS;
"""
cnxn = pypyodbc.connect(connStr)
crsr = cnxn.cursor()
sql = """
INSERT INTO myTable VALUES (?, ?)
"""
for dataRow in myData:
print(dataRow)
crsr.execute(sql, dataRow)
cnxn.commit()
crsr.close()
cnxn.close()
Based on above answers I wrote a wrapper function for my use case, you can try and change the function according to your need.
def sanitizeData(value):
if value in ('', None):
return "NULL"
# This case handles the case where value already has ' in it (ex: O'Brien). This is how SQL skils single quotes
if type(value) is str:
return "'{}'".format(value.replace("'", "''"))
return value
Now call the sql query like so,
"INSERT INTO %s (Name, Email) VALUES (%s, %s)"%(table_name, sanitizeData(actual_name), sanitizeData(actual_email))
Why not set the variable equal to some string like 'no price' and then filter this out later when you want to do math on the numbers?
filter(lambda x: x != 'no price',list_of_data_from_database)
Do a quick check for blank, and if it is, set it equal to NULL:
if(!variable_to_insert)
variable_to_insert = "NULL"
...then make sure that the inserted variable is not in quotes for the insert statement, like:
insert = "INSERT INTO table (var) VALUES (%s)" % (variable_to_insert)
...
not like:
insert = "INSERT INTO table (var) VALUES ('%s')" % (variable_to_insert)
...
Related
I am new in Python. I have written following piece of code to construct INSERT sql query:
sql = f"INSERT INTO my_table (col1, col2) VALUES (%s, %s)"
params = []
for k, v in my_dict.items():
params.append(str(k), v) ##AS COL1 is of type varchar I am typecasting first param k
for param in params:
curr_sql_query = sql % param
log.info(curr_sql_query)
db_obj.execute(sql)
But even after converting first param to string as above, sql string I am getting as follows:
INSERT INTO my_table (col1, col2) VALUES (1.2.3.4, 1) where value should have been ('1.2.3.4', 1)
The error I am getting is as follows while trying to run the query string using db_obj.execute(sql) :
error=(1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.3.4 at line 1')"}
str(k) does not add quotes('). You should do it yourself.
params.append((f"'{k}'", v))
If k is already str type, you can use repr that repesenting more detail for object,
rerp('test') returns "'test'" whereas str('test') returns"test".
params.append((repr(k), v))
But be careful with it, since if you already have ' in k, it repr wraps your string with ".
For example,
repr('\'Hello\' is string')
output:
"'Hello' is string"
You should use placeholders instead to avoid issues with quotes, escaped characters and possible code injection:
sql = "INSERT INTO my_table (col1, col2) VALUES (%s, %s)"
for k, v in my_dict.items():
db_obj.execute(sql, (str(k), v))
I want to define a function with arguments which adds data to a table in sqlite3.
I am following a tutorial on youtube, which is this one: https://www.youtube.com/watch?v=o-vsdfCBpsU
I did the same thing, and had no problems until I decided to put in some arguments:
import sqlite3
conn = sqlite3.connect('tutorial.db')
c = conn.cursor()
def create_table():
#c.execute('CREATE TABLE IF NOT EXISTS stuffToPlot(unix REAL, datestamp TEXT, keyword TEXT, value REAL)')
c.execute('CREATE TABLE IF NOT EXISTS info(names TEXT, floats REAL, ints INTEGER, personalities TEXT)')
def data_entry(name, float, int, personality):
#c.execute("INSERT INTO stuffToPlot VALUES(8010, '2016-01-02', 'Python', 8)")
c.execute("INSERT INTO info VALUES(name, float, int, personality)")
conn.commit()
c.close()
conn.close()
create_table()
ref = [25,35,45,55,65,78,89,90]
for i in range(5):
for j in ref:
data_entry(j, .888, i, 'kind')
In the data_entry function, the "name" inside the parentheses of VALUES() apparently causes this error:
OperationalError: no such column: name
I am doing this for an analysis of a simulation, so I'd be dealing with lots of lines of data in the database, which needs to be done using a function like this
you need to use bind variables to pass values. If you change your insert execute to be
c.execute("INSERT INTO info VALUES(?,?,?,?)", [name, float, int, personality])
it should work. This tells it to insert the values and and swaps the '?' for the values in the list passed to execute.
Also, as someone noted in the comments you need to decided if you want the column to be 'name' or 'names' and then change the queries accordingly.
And as a last note, you really shouldn't use float and int as variable names. They are data types so it can be confusing.
Edit this should work. made some improvements. I added a finally block to close the connection. This way even if there is an error the connection will be closed. There is def a cleaner way to write this, but this should work at least.
import sqlite3
conn = sqlite3.connect('tutorial.db')
c = conn.cursor()
def create_table():
#c.execute('CREATE TABLE IF NOT EXISTS stuffToPlot(unix REAL, datestamp TEXT, keyword TEXT, value REAL)')
c.execute('CREATE TABLE IF NOT EXISTS info(names TEXT, floats REAL, ints INTEGER, personalities TEXT)')
def data_entry(name, float, int, personality):
#c.execute("INSERT INTO stuffToPlot VALUES(8010, '2016-01-02', 'Python', 8)")
c.execute("INSERT INTO info VALUES(?,?,?,?)", [name, float, int, personality])
conn.commit()
try:
create_table()
ref = [25,35,45,55,65,78,89,90]
for i in range(5):
for j in ref:
data_entry(j, .888, i, 'kind')
except:
raise
finally:
conn.close()
Your SQL command in data entry won't work. You're not actually providing any values to insert, and you're not defining which columns to use.
A proper insertion with sqlite3 would look something like this:
to_sql_dict = {
"name": name,
"float": float,
"int": int,
"personality": personality
}
SQL = """INSERT INTO info (
names, floats, ints, personalities
) VALUES (
:name, :float, :int, :personality
)"""
c.execute(SQL, to_sql_dict)
This is the proper way to do it. A simpler way involves SQL-only, but is advised against as it is supposedly susceptible to SQL injection attacks:
SQL = """INSERT INTO info (
names, floats, ints, personalities
) VALUES (
{}, {}, {}, {}
)""".format(name, float, int, personality)
c.execute(SQL)
Either way should work, but as I mentioned you should try to get it to work the first way.
PS. Note on column names: it's generally not common practice to use plurals.
EDIT: Code.
I'm using psycopg2 to interact with a PostgreSQL database. I have a function whereby any number of columns (from a single column to all columns) in a table could be inserted into. My question is: how would one properly, dynamically, construct this query?
At the moment I am using string formatting and concatenation and I know this is the absolute worst way to do this. Consider the below code where, in this case, my unknown number of columns (i.e. keys from a dict is in fact 2):
dictOfUnknownLength = {'key1': 3, 'key2': 'myString'}
def createMyQuery(user_ids, dictOfUnknownLength):
fields, values = list(), list()
for key, val in dictOfUnknownLength.items():
fields.append(key)
values.append(val)
fields = str(fields).replace('[', '(').replace(']', ')').replace("'", "")
values = str(values).replace('[', '(').replace(']', ')')
query = f"INSERT INTO myTable {fields} VALUES {values} RETURNING someValue;"
query = INSERT INTO myTable (key1, key2) VALUES (3, 'myString') RETURNING someValue;
This provides a correctly formatted query but is of course prone to SQL injections and the like and, as such, is not an acceptable method of achieving my goal.
In other queries I am using the recommended methods of query construction when handling a known number of variables (%s and separate argument to .execute() containing variables) but I'm unsure how to adapt this to accommodate an unknown number of variables without using string formatting.
How can I elegantly and safely construct a query with an unknown number of specified insert columns?
To add to your worries, the current methodology using .replace() is prone to edge cases where fields or values contain [, ], or '. They will get replaced no matter what and may mess up your query.
You could always use .join() to join a variable number of values in your list. To top it up, format the query appropriately with %s after VALUES and pass your arguments into .execute().
Note: You may also want to consider the case where the number of fields is not equal to the number values.
import psycopg2
conn = psycopg2.connect("dbname=test user=postgres")
cur = conn.cursor()
dictOfUnknownLength = {'key1': 3, 'key2': 'myString'}
def createMyQuery(user_ids, dictOfUnknownLength):
# Directly assign keys/values.
fields, values = list(dictOfUnknownLength.keys()), list(dictOfUnknownLength.values())
if len(fields) != len(values):
# Raise an error? SQL won't work in this case anyways...
pass
# Stringify the fields and values.
fieldsParam = ','.join(fields) # "key1, key2"
valuesParam = ','.join(['%s']*len(values))) # "%s, %s"
# "INSERT ... (key1, key2) VALUES (%s, %s) ..."
query = 'INSERT INTO myTable ({}) VALUES ({}) RETURNING someValue;'.format(fieldsParam, valuesParam)
# .execute('INSERT ... (key1, key2) VALUES (%s, %s) ...', [3, 'myString'])
cur.execute(query, values) # Anti-SQL-injection: pass placeholder
# values as second argument.
lst = [{'Fruit':'Apple','HadToday':2},{'Fruit':'Banana','HadToday':8}]
I have a long list of dictionaries of the form above.
I have two fixed variables.
person = 'Sam'
date = datetime.datetime.now()
I wish to insert this information into a mysql table.
How I do it currently
for item in lst:
item['Person'] = person
item['Date'] = date
cursor.executemany("""
INSERT INTO myTable (Person,Date,Fruit,HadToday)
VALUES (%(Person)s, %(Date)s, %(Fruit)s, %(HadToday)s)""", lst)
conn.commit()
Is their a way to do it, that bypasses the loop as the person and date variables are constant. I have tried
lst = [{'Fruit':'Apple','HadToday':2},{'Fruit':'Banana','HadToday':8}]
cursor.executemany("""
INSERT INTO myTable (Person,Date,Fruit,HadToday)
VALUES (%s, %s, %(Fruit)s, %(HadToday)s)""", (person,date,lst))
conn.commit()
TypeError: not enough arguments for format string
Your problem here is, that it tries to apply all of lst into %(Fruit)s and nothing is left for %(HadToday)s).
You should not fix it by hardcoding the fixed values into the statement as you get into troubles if you have a name like "Tim O'Molligan" - its better to let the db handle the correct formatting.
Not mysql, but you get the gist: http://initd.org/psycopg/docs/usage.html#the-problem-with-the-query-parameters - learned this myself just a week ago ;o)
The probably cleanest way would be to use
cursor.execute("SET #myname = %s", (person,))
cursor.execute("SET #mydate = %s", (datetime.datetime.now(),))
and use
cursor.executemany("""
INSERT INTO myTable (Person,Date,Fruit,HadToday)
VALUES (#myname, #mydate, %(Fruit)s, %(HadToday)s)""", lst)
I am not 100% about the syntax, but I hope you get the idea. Comment/edit the answer if I have a misspell in it.
If I have a statement like so:
cursor.execute("INSERT INTO MYTABLE(name, age, hair_color) VALUES (?, ?, ?)" , ("Alice", 24, None))
cursor.execute("SELECT * FROM MYTABLE WHERE Id=?", (1,))
print cursor.fetchone()[2] is None # This is false! I want this to be true.
Retrieving the hair_color will be u'None' rather than a NoneType.
I know I can just do a simple check to see if it's equal to a unicode "None" but rather than specifying this condition, is there a way to tell python that I want fields that have "None" to be replaced with a NoneType when I do a fetchone() or fetchall()?
MYTABLE is (Id INTEGER PRIMARY KEY, name TEXT, age INTEGER, hair_color TEXT)
Sorry guys, I just figured it out.
As it turns out, I have to explicitly tell sqlite3 that I want python to automatically convert between types.
So the solution was to do this:
conn = sqlite3.connect(db, detect_types=sqlite3.PARSE_DECLTYPES)