Develop Reference

Postgres connection in python

I am struggling to establish a connection inside data iteration. Means I am running a select query to postgres and iterating the return data. after some transformation I am writing it to another table. But it is not working. Sample python code is below.
conn = pgconn(------)
cursor = pgconn.Cursor()
query1 = "select * from table"
query2 = "select * from table2 where Id=(%s);"
cursor.execute(query1)
result = query1.fetchall()
for row in result:
If row.a == 2:
cursor.execute(query2, [row.time])
In the above python code I can't able to extract the data by running query2 and passing query1 result as a parameter. It seems cursor is blocked by the query1 so query2 execution is not happening. Please some one help in this issue.

First of all you can write a join statement to do this and can get the data easily
select * from table join table2 where table2.id == table.time
Also why this is not working maybe because the cursor object is getting override inside the for loop and thus the query results get changed.

Use RealDictCursor, and correct the syntax on your inside call to execute():
import psycopg2
import psycopg2.extras
conn = pgconn(------)
cursor = conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
query1 = "select * from table"
query2 = "select * from table2 where Id=(%s);"
cursor.execute(query1)
result = query1.fetchall()
for row in result:
If row.a == 2:
cursor.execute(query2, (row['time'],))

1. install psycopg2 and psycopg2.extras. ( pip install)
Then set up your Postgres Connection like:
def Postgres_init(self):
try:
conn = psycopg2.connect(host=os.environ['SD_POSTGRES_SERVER'],
user=os.environ['SD_POSTGRES_USER'],
password=os.environ['SD_POSTGRES_PASSWORD'],
port=os.environ['SD_POSTGRES_PORT'],
database=os.environ['SD_POSTGRES_DATABASE'])
logging.info("Connected to PostgreSQL")
except (Exception, psycopg2.Error) as error:
logging.info(error)
2. Connect your Cursor with the defined connection
cursor = conn.cursor()
3. Execute your query:
cursor.execute("""SELECT COUNT (column1) from tablename WHERE column2 =%s""", (
Value,)) # Check if already exists
result = cursor.fetchone()
Now the value is stored in the "result" variable. Now you can execute the next query like:
cursor.execute("""
INSERT INTO tablename2
(column1, column2, column3)
VALUES
(%s, %s, %s)
ON CONFLICT(column1) DO UPDATE
SET
column2=excluded.column2,
column3=excluded.column3;
""", (result, column2, column3)
)
Now the result of query 1 is stored in the second table in the first column.
Now you can close your connection:
conn.close()

Python variables in mysql request

I'm quite new in Python (Python 3.4.6) :)
I'm trying to insert into a mysql db some lines but with variables.
At the beginning, I've a dictionary list_hosts.
Here is my code :
import mysql.connector
import time
db = mysql.connector.connect(host='localhost', user='xxxxx', passwd='xxxxx', database='xxxxx')
cursor = db.cursor()
now_db = time.strftime('%Y-%m-%d %H:%M:%S')
for key, value in list_hosts
key_db += key+", "
value_ex += "%s, "
value_db += "\""+value+"\", "
key_db = key_db.strip(" ")
key_db = key_db.strip(",")
value_ex = value_ex.strip(" ")
value_ex = value_ex.strip(",")
value_db = value_db.strip(" ")
value_db = value_db.strip(",")
add_host = ("INSERT INTO nagios_hosts (date_created, date_modified, "+key_db+") VALUES ("+value_ex+")")
data_host = ("\""+now_db+"\", \""+now_db+"\", "+value_db)
cursor.execute(add_host, data_host)
db.commit()
db.close()
Example of list_hosts:
OrderedDict([('xxxx1', 'data1'), ('xxxx2', 'data2'), ('xxxx3', 'data3'), ('xxxx4', 'data4'), ('xxxx5', 'data5'), ('xxxx6', 'data6')])
I've simplified the code of course.
I did it like this as I've never have the same amount of items in the dictionnary.
I'm trying to create something like this :
add_host - INSERT INTO TABLE (date_created, date_modified, xxxx1, xxxx2, xxxx3, xxxx4, xxxx5, xxxx6) VALUES (%s, %s, %s, %s, %s, %s)
data_host - now, now, data1, data2, data3, data4, data5, data6
Where there are never the same number of xxxx...
They all exist in the DB, but I don't need to fill each column for each item in the dictionnary.
When I execute I get this error :
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'xxxxxxxxxxxxxxxxxxxxx' at line 1
As I'm beginning with Python, I think there are a lot of things we can clean too... don't hesitate :)

Here's a canonical python3 (python2 compatible) solution:
import time
from collections import OrderedDict
list_hosts = OrderedDict([("field1", "value1"), ("field2", "value2"), ("fieldN", "valueN")])
# builds a comma-separated string of db placeholders for the values:
placeholders = ", ".join(["%s"] * (len(list_hosts) + 2))
# builds a comma-separated string of field names
fields = ", ".join(("date_created","date_modified") + tuple(list_hosts.keys()))
# builds a tuple of values including the dates
now_db = time.strftime('%Y-%m-%d %H:%M:%S')
values = (now_db, now_db) + tuple(list_hosts.values())
# build the SQL query:
sql = "INSERT INTO nagio_hosts({}) VALUES({})".format(fields, placeholders)
# and safely execute it
cursor.execute(sql, values)
db.commit()

As #khelwood mentioned in the comments, you should use parameterized queries.
If the number of columns you're inserting varies, you might prefer to generate a tuple and use it in a parameterized query then.
cursor.execute() accepts two parameters:
a query as a string;
parameters as a tuple.
The idea is to generate the string and the tuple and pass those to cursor.execute().
You'll need something like this:
list_hosts = {'xxxx1': 'data1', 'xxxx2': 'data2', 'xxxx3': 'data3', 'xxxx4': 'data4'}
keys = [] # creating a list for keys
values = () # creating a tuple for values
for key, value in list_hosts.items():
keys.append(key)
values = values + (value,)
keys_str = ', '.join(keys)
ps = ', '.join(['%s'] * len(list_hosts))
query = "INSERT INTO tbl (%s) VALUES (%s)" % (keys_str, ps)
print(query)
# INSERT INTO tbl (data_created, data_modified, xxxx1, xxxx2, xxxx3, xxxx4) VALUES (%s, %s, %s, %s)
cursor.execute(query, values)
Just tried it on a sample data, works fine!

Use a python dictionary to insert into mysql

I am trying to take the data from a dictionary (the example is simplified for readability) and insert it into a mysql database.
I have the following piece of code.
import pymysql
conn = pymysql.connect(server, user , password, "db")
cur = conn.cursor()
ORFs={'E7': '562', 'E6': '83', 'E1': '865', 'E2': '2756 '}
table="genome"
cols = ORFs.keys()
vals = ORFs.values()
sql = "INSERT INTO %s (%s) VALUES(%s)" % (
table, ",".join(cols), ",".join(vals))
print sql
print ORFs.values()
cur.execute(sql, ORFs.values())
cur.close()
conn.close()
the print sql statement returns
INSERT INTO genome (E7,E6,E1,E2) VALUES(562,83,865,2756 )
when I type this directly into the mysql command line, the mysql command works. But when I run the python script I get an error:
<type 'exceptions.TypeError'>: not all arguments converted during string formatting
args = ('not all arguments converted during string formatting',)
message = 'not all arguments converted during string formatting'
As always, any suggestions would be highly appreciated.

The previous answer doesn't work for non string dictionary value. This one is a revised version.
format_string = ','.join(['%s'] * len(dict))
self.db.set("""INSERT IGNORE INTO listings ({0}) VALUES ({1})""".format(", ".join(dict.keys()),format_string),
(dict.values()))

sql = "INSERT INTO %s (%s) VALUES(%s)" % (
table, ",".join(cols), ",".join(vals))
This SQL includes values and cur.execute(sql, ORFs.values()) has values, too.
So, it should be cur.execute(sql).

In my case, I will skip null columns.
data = {'k': 'v'}
fs = ','.join(list(map(lambda x: '`' + x + '`', [*data.keys()])))
vs = ','.join(list(map(lambda x: '%(' + x + ')s', [*data.keys()])))
sql = "INSERT INTO `%s` (%s) VALUES (%s)" % (table, fs, vs)
count = cursor.execute(sql, data)

Python insert into SQL unicode error

I have the following error :
db = MySQLdb.connect(host="localhost", # host
user="root", # username
passwd="", # password
db="test",charset='utf8') #
cur = db.cursor()
x = "испытание" # random unicode characters
sql = "INSERT INTO links(test) VALUES(N'%s');"
lst = ( x ) #x is unicode data
cur.execute(sql,lst)
The error I get is : MySQL Error [1064]: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax ...

x = "испытание" # random unicode characters
(What Python version are you using? If 2.x, those are not Unicode characters, they're bytes.)
sql = "INSERT INTO links(test) VALUES(N'%s');"
When you use parameterised queries, you don't include the string literal delimiters ''. For MySQLdb where the parameter marker is %s, it should just be:
sql = "INSERT INTO links(test) VALUES(%s);"
(Note also NVARCHAR is unnecessary in MySQL.)
lst = ( x ) #x is unicode data
Here lst is the same value as x, you haven't got a tuple. If you really want a tuple-of-one then say (x,), but probably using an actual list [x] is clearer.

what value are you trying to insert? nothing. what does N'%s' represents. I guess you are trying to insert from a form so I have work on this code, tested it and is running.
First create database with 2 column employee_id and lastname. Then test this code. it will work for you. If you get it right please tick this as answer. Sectona...
#!c:/python25/python
import MySQLdb
# Import modules for CGI handling
import cgi, cgitb
# Create instance of FieldStorage
form = cgi.FieldStorage()
# get form data from form field
employee_id = form.getvalue('employee_id')
lastname = form.getvalue('lastname')
# Open database connection
db = MySQLdb.connect("localhost","sectona","sectona90","sectona_db" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
sql = "INSERT INTO EMPLOYEE(employee_id, \
lastname) \
VALUES ('%s', '%s')" % \
(employee_id, lastname)
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
print "Content-type:text/html\r\n\r\n"
print '<html>'
print '<body>'
print '<h2>Data Submitted Successfully</h2>'
print "<b>Employee Identity:</b> %s<br>" % (employee_id)
print "<b>Last Name:</b> %s<br>" % (lastname)
print '</body>'
print '</html>'

sqlite3.Warning: You can only execute one statement at a time

I get the error when running this code:
import sqlite3
user_name = raw_input("Please enter the name: ")
user_email = raw_input("Please enter the email: ")
db = sqlite3.connect("customer")
cursor=db.cursor()
sql = """INSERT INTO customer
(name, email) VALUES (?,?);,
(user_name, user_email)"""
cursor.execute(sql)
Why is this happening?

While the other posters are correct about your statement formatting you are receiving this particular error because you are attempting to perform multiple statements in one query (notice the ; in your query which separates statements).
From Python sqlite3 docs:
"execute() will only execute a single SQL statement. If you try to execute more than one
statement with it, it will raise a Warning. Use executescript() if you want to execute
multiple SQL statements with one call."
https://docs.python.org/2/library/sqlite3.html
Now your statement will not execute properly even if you use executescript() because there are other issues with the way it is formatted (see other posted answers). But the error you are receiving is specifically because of your multiple statements. I am posting this answer for others that may have wandered here after searching for that error.

Use executescript instead of execute
execute() will only execute a single SQL statement. If you try to execute more than one statement with it, it will raise a Warning. Use executescript() if you want to execute multiple SQL statements with one call.
https://docs.python.org/2/library/sqlite3.html#sqlite3.Cursor.execute

You have a ;, in the middle of the query string - that is an invalid syntax. Pass a dictionary as a second argument to execute if you want to use a named parameter binding.
sql = "INSERT INTO customer (name, email) VALUES (:name, :email)"
cursor.execute(sql, {'name':user_name, 'email':user_email})

Try this:
sql = """INSERT INTO customer
(name, email) VALUES (?,?)"""
cursor.execute(sql, (user_name, user_email))

import sqlite3
def DB():
List = {"Name":"Omar", "Age":"33"}
columns = ', '.join("" + str(x).replace('/', '_') + "" for x in List.keys())
values = ', '.join("'" + str(x).replace('/', '_') + "'" for x in List.values())
sql_qry = "INSERT INTO %s ( %s ) values (?,?) ; ( %s )" % ('Table Name', columns, values)
conn = sqlite3.connect("DBname.db")
curr = conn.cursor()
# curr.execute("""create table if not exists TestTable(
# Name text,
# Age text
# )""")
# print columns
# print values
# print sql
# sql = 'INSERT INTO yell (Name , Age) values (%s, %s)'
curr.execute(sql_qry)
DB()