How do I insert a python dictionary into a Postgresql2 table? I keep getting the following error, so my query is not formatted correctly:
Error syntax error at or near "To" LINE 1: INSERT INTO bill_summary VALUES(To designate the facility of...
import psycopg2
import json
import psycopg2.extras
import sys
with open('data.json', 'r') as f:
data = json.load(f)
con = None
try:
con = psycopg2.connect(database='sanctionsdb', user='dbuser')
cur = con.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute("CREATE TABLE bill_summary(title VARCHAR PRIMARY KEY, summary_text VARCHAR, action_date VARCHAR, action_desc VARCHAR)")
for d in data:
action_date = d['action-date']
title = d['title']
summary_text = d['summary-text']
action_date = d['action-date']
action_desc = d['action-desc']
q = "INSERT INTO bill_summary VALUES(" +str(title)+str(summary_text)+str(action_date)+str(action_desc)+")"
cur.execute(q)
con.commit()
except psycopg2.DatabaseError, e:
if con:
con.rollback()
print 'Error %s' % e
sys.exit(1)
finally:
if con:
con.close()
You should use the dictionary as the second parameter to cursor.execute(). See the example code after this statement in the documentation:
Named arguments are supported too using %(name)s placeholders in the query and specifying the values into a mapping.
So your code may be as simple as this:
with open('data.json', 'r') as f:
data = json.load(f)
print(data)
""" above prints something like this:
{'title': 'the first action', 'summary-text': 'some summary', 'action-date': '2018-08-08', 'action-desc': 'action description'}
use the json keys as named parameters:
"""
cur = con.cursor()
q = "INSERT INTO bill_summary VALUES(%(title)s, %(summary-text)s, %(action-date)s, %(action-desc)s)"
cur.execute(q, data)
con.commit()
Note also this warning (from the same page of the documentation):
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
q = "INSERT INTO bill_summary VALUES(" +str(title)+str(summary_text)+str(action_date)+str(action_desc)+")"
You're writing your query in a wrong way, by concatenating the values, they should rather be the comma-separated elements, like this:
q = "INSERT INTO bill_summary VALUES({0},{1},{2},{3})".format(str(title), str(summery_text), str(action_date),str(action_desc))
Since you're not specifying the columns names, I already suppose they are in the same orders as you have written the value in your insert query. There are basically two way of writing insert query in postgresql. One is by specifying the columns names and their corresponding values like this:
INSERT INTO TABLE_NAME (column1, column2, column3,...columnN)
VALUES (value1, value2, value3,...valueN);
Another way is, You may not need to specify the column(s) name in the SQL query if you are adding values for all the columns of the table. However, make sure the order of the values is in the same order as the columns in the table. Which you have used in your query, like this:
INSERT INTO TABLE_NAME VALUES (value1,value2,value3,...valueN);
Related
I am struggling with generating the delete query where parameters for the query is actually a set of values.
So I need to delete rows where parameters are a pair values for example:
delete from table where col1 = %s and col2 = %s
which can be executed in Python like:
cur = conn.cursor()
cur.execute(query, (col1_value, col2_value))
Now I would like to run a query:
delete from table where (col1, col2) in ( (col1_value1, col2_value1), (col1_value2, col2_value2) );
I can generate the queries and values and execute the exact SQL but I can't quite generate prepared statement.
I tried:
delete from table where (col1, col2) in %s
and
delete from table where (col1, col2) in (%s)
But when I try to execute:
cur.execute(query, list_of_col_value_tuples)
or
cur.execute(query, tuple_of_col_value_tuples)
I get an exception that indicates that psycopg2 cannot convert arguments to strings.
Is there any way to use psycopg2 to execute a query like this?
You could dynamically add %s placeholders to your query:
cur = con.cursor()
query = "delete from table where (role, username) in (%s)"
options = [('admin', 'foo'), ('user', 'bar')]
placeholders = '%s,' * len(options)
query = query % placeholders[:-1] # remove last comma
print(query)
print(cur.mogrify(query, options).decode('utf-8'))
Out:
delete from table where (role, user) in (%s,%s)
delete from table where (role, user) in (('admin', 'foo'),('user', 'bar'))
Alternatively, build the query using psycopg2.sql as answered there.
Actually the resolution is quite easy if carefully constructed.
In the miscellaneous goodies of psycopg2 there is a function execute_values.
While all the examples that are given by psycopg2 deal with inserts as the function basically converts the list of arguments into a VALUES list if the call to delete is formatted like so:
qry = "delete from table where (col1, col2) in (%s)"
The call:
execute_values(cur=cur, qry=qry, argslist=<list of value tuples>)
will make the delete perform exactly as required.
I have list of tuples with data something like this:
list1 = [(1100, 'abc', '{"1209": "Y", "1210": "Y"}'), (1100, 'abc', None)]
def insert_sample_data(col_val):
cur = self.con.cursor()
sql = """insert into sampletable values {}""".format(col_val)
cur.execute(sql)
self.con.commit()
cur.close()
values1 = ', '.join(map(str, list1)) #bulk insert
insert_sample_data(values1)
Table Structure:
ssid int, name varchar, rules jsonb
When I am trying to insert the data but it throws me an error saying "insert column "none" does not exist". How can we load the data into table with 'Null' or 'None'?
I looked at this solution but it does not help in this case How to insert 'NULL' values into PostgreSQL database using Python?
As #shmee states, you need to use something like executemany and parameterize your values instead of using format, which is vulnerable to SQL injection.
I would modify your code as follows:
def insert_sample_data(self, values): # added self since you are referencing it below
with self.con.cursor() as cur:
sql = "insert into sampletable values (%s, %s, %s)" # Use %s for parameters
cur.executemany(sql, values) # Pass the list of tuples directly
self.con.commit()
list1 = [(1100, 'abc', '{"1209": "Y", "1210": "Y"}'), (1100, 'abc', None)]
self.insert_sample_data(list1) # pass the list directly
Im trying to pull some XML from a URL, parse it and store the entries in an sqlite3 database, Im trying numerous things and all are failing. Codde so far:
#!/usr/bin/env python
from urllib2 import urlopen
import gc
import xml.etree.ElementTree as ET
import sqlite3
rosetta_url = ("https://boinc.bakerlab.org/rosetta/team_email_list.php?teamid=12575&account_key=Y&xml=1")
root = ET.parse(urlopen(rosetta_url)).getroot()
cpids = [el.text for el in root.findall('.//user/cpid')]
print cpids
conn = sqlite3.connect("GridcoinTeam.db")
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS GRIDCOINTEAM (cpid TEXT)''')
c.executemany("INSERT INTO GRIDCOINTEAM VALUES (?);", cpids)
conn.commit()
conn.close()
conn = sqlite3.connect("GridcoinTeam.db")
c = conn.cursor()
cpids = c.execute('select cpid from GRIDCOINTEAM').fetchall()
conn.close()
print cpids
gc.collect()
Im getting the error:
Incorrect number of bindings supplied. The current statement uses 1, and there are 32 supplied.
I tried making the insertion tuples by changing to
c.executemany("INSERT INTO GRIDCOINTEAM VALUES (?);", (cpids, ))
but that just gives:
Incorrect number of bindings supplied. The current statement uses 1, and there are 3289 supplied.
The XML extract is in the form ['5da243d1f47b7852d372c51d6ee660d7', '5a6d18b942518aca60833401e70b75b1', '527ab53f75164864b74a89f3db6986b8'], but there are several thousand entries.
Thanks.
You need to insert this as multiple rows instead of multiple columns into one row
cpids = [el.text for el in root.findall('.//user/cpid')]
cpids = zip(*[iter(cpids)]*1)
print cpids
The problem lies in
c.executemany("INSERT INTO GRIDCOINTEAM VALUES (?);", cpids)
This executemany expects a list of tuples, but you pass a list of strings. What the code does effectively is:
for entry in cpids:
c.execute("INSERT INTO GRIDCOINTEAM VALUES (?);", *entry)
Note the star before entry, which unloads the string, and which gives you 32+ params whereas you only want one.
In order to fix that we'd need to know your GRIDCOINTEAM table schema. If you have a table with only one column and you want to insert that, you could probably do this:
for entry in cpids:
c.execute("INSERT INTO GRIDCOINTEAM VALUES (?)", entry)
In contrast to executemany, execute takes each parameter as one param - no tuples and lists unloading here.
Alternatively you can resort to using executemany, but you'd then need to wrap every one of your strings in a tuple or generator:
c.executemany("INSERT INTO GRIDCOINTEAM VALUES (?);", [(i,) for i in cpids])
I'm trying to make a function which inserts a row into the SQLite3 database using dictionary.
I found here, on SO a way to do that, but it unfortunately does not work. There is some problem I can't figure out.
def insert_into_table(self,data):
for key in data.keys(): # ADDING COLUMNS IF NECESSARY
columns = self.get_column_names()
column = key.replace(' ','_')
if column not in columns:
self.cur.execute("""ALTER TABLE vsetkyfirmy ADD COLUMN {} TEXT""".format(column.encode('utf-8')))
self.conn.commit()
new_data={}
for v,k in data.iteritems(): # new dictionary with remaden names (*column = key.replace(' ','_'))
new_data[self.remake_name(v)]=k
columns = ', '.join(new_data.keys())
placeholders = ':'+', :'.join(new_data.keys())
query = 'INSERT INTO vsetkyfirmy (%s) VALUES (%s)' % (columns, placeholders)
self.cur.execute(query, new_data)
self.conn.commit()
EXCEPTION:
self.cur.execute(query, new_data)
sqlite3.ProgrammingError: You did not supply a value for binding 1.
When I print query and new_data everything seems correct:
INSERT INTO vsetkyfirmy (Obchodné_meno, IČ_DPH, Sídlo, PSČ, Spoločník, IČO, Základné_imanie, Konateľ, Ročný_obrat, Dátum_vzniku, Právna_forma) VALUES (:Obchodné_meno, :IČ_DPH, :Sídlo, :PSČ, :Spoločník, :IČO, :Základné_imanie, :Konateľ, :Ročný_obrat, :Dátum_vzniku, :Právna_forma)
{u'Obchodn\xe9_meno': 'PRspol. s r.o.', u'I\u010c_DPH': 'S9540', u'S\xeddlo': u'Bansk\xe1 Bystrica, Orembursk\xe1 2', u'PS\u010c': '97401', u'Spolo\u010dn\xedk': u'Dana Dzurianikov\xe1', u'I\u010cO': '3067', u'Z\xe1kladn\xe9_imanie': u'142899 \u20ac', u'Konate\u013e': 'Miroslav Dz', u'Ro\u010dn\xfd_obrat': '2014: 482 EUR', u'D\xe1tum_vzniku': '01.12.1991 ', u'Pr\xe1vna_forma': u'Spolo\u010dnos\u0165 s ru\u010den\xedm obmedzen\xfdm'}
EDIT: So I've tried to remove ':' from query so it looks like:
INSERT INTO vsetkyfirmy (Obchodné_meno, IČ_DPH, Sídlo, PSČ, Spoločník, IČO, Základné_imanie, Konateľ, Ročný_obrat, Dátum_vzniku, Právna_forma) VALUES (Obchodné_meno, IČ_DPH, Sídlo, PSČ, Spoločník, IČO, Základné_imanie, Konateľ, Ročný_obrat, Dátum_vzniku, Právna_forma)
And it returns that sqlite3.OperationalError: no such column: Obchodné_meno
I don't know where is the problem, could it be in encoding?
You are calling encode('utf-8') when creating the table, but not when inserting.
SQLite indeed uses UTF-8, but the sqlite3 module automatically handles conversion from/to Python's internal Unicode string encoding. Don't try to reencode manually.
I am trying to access PostgreSQL using psycopg2:
sql = """
SELECT
%s
FROM
table;
"""
cur = con.cursor()
input = (['id', 'name'], )
cur.execute(sql, input)
data = pd.DataFrame.from_records(cur.fetchall())
However, the returned result is:
0
0 [id, name]
1 [id, name]
2 [id, name]
3 [id, name]
4 [id, name]
If I try to access single column, it looks like:
0
0 id
1 id
2 id
3 id
4 id
It looks like something is wrong with the quoting around column name (single quote which should not be there):
In [49]: print cur.mogrify(sql, input)
SELECT
'id'
FROM
table;
but I am following doc: http://initd.org/psycopg/docs/usage.html#
Anyone can tell me what is going on here? Thanks a lot!!!
Use the AsIs extension
import psycopg2
from psycopg2.extensions import AsIs
column_list = ['id','name']
columns = ', '.join(column_list)
cursor.execute("SELECT %s FROM table", (AsIs(columns),))
And mogrify will show that it is not quoting the column names and passing them in as is.
Nowadays, you can use sql.Identifier to do this in a clean and secure way :
from psycopg2 import sql
statement = """
SELECT
{id}, {name}
FROM
table;
"""
with con.cursor() as cur:
cur.execute(sql.SQL(statement).format(
id=sql.SQL.Identifier("id"),
name=sql.SQL.Identifier("name")
))
data = pd.DataFrame.from_records(cur.fetchall())
More information on query composition here : https://www.psycopg.org/docs/sql.html
The reason was that you were passing the string representation of the array ['id', 'name'] as SQL query parameter but not as the column names. So the resulting query was similar to
SELECT 'id, name' FROM table
Looks your table had 5 rows so the returned result was just this literal for each row.
Column names cannot be the SQL query parameters but can be just the usual string parameters which you can prepare before executing the query-
sql = """
SELECT
%s
FROM
table;
"""
input = 'id, name'
sql = sql % input
print(sql)
cur = con.cursor()
cur.execute(sql)
data = pd.DataFrame.from_records(cur.fetchall())
In this case the resulting query is
SELECT
id, name
FROM
table;