I am struggling to establish a connection inside data iteration. Means I am running a select query to postgres and iterating the return data. after some transformation I am writing it to another table. But it is not working. Sample python code is below.
conn = pgconn(------)
cursor = pgconn.Cursor()
query1 = "select * from table"
query2 = "select * from table2 where Id=(%s);"
cursor.execute(query1)
result = query1.fetchall()
for row in result:
If row.a == 2:
cursor.execute(query2, [row.time])
In the above python code I can't able to extract the data by running query2 and passing query1 result as a parameter. It seems cursor is blocked by the query1 so query2 execution is not happening. Please some one help in this issue.
First of all you can write a join statement to do this and can get the data easily
select * from table join table2 where table2.id == table.time
Also why this is not working maybe because the cursor object is getting override inside the for loop and thus the query results get changed.
Use RealDictCursor, and correct the syntax on your inside call to execute():
import psycopg2
import psycopg2.extras
conn = pgconn(------)
cursor = conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
query1 = "select * from table"
query2 = "select * from table2 where Id=(%s);"
cursor.execute(query1)
result = query1.fetchall()
for row in result:
If row.a == 2:
cursor.execute(query2, (row['time'],))
1. install psycopg2 and psycopg2.extras. ( pip install)
Then set up your Postgres Connection like:
def Postgres_init(self):
try:
conn = psycopg2.connect(host=os.environ['SD_POSTGRES_SERVER'],
user=os.environ['SD_POSTGRES_USER'],
password=os.environ['SD_POSTGRES_PASSWORD'],
port=os.environ['SD_POSTGRES_PORT'],
database=os.environ['SD_POSTGRES_DATABASE'])
logging.info("Connected to PostgreSQL")
except (Exception, psycopg2.Error) as error:
logging.info(error)
2. Connect your Cursor with the defined connection
cursor = conn.cursor()
3. Execute your query:
cursor.execute("""SELECT COUNT (column1) from tablename WHERE column2 =%s""", (
Value,)) # Check if already exists
result = cursor.fetchone()
Now the value is stored in the "result" variable. Now you can execute the next query like:
cursor.execute("""
INSERT INTO tablename2
(column1, column2, column3)
VALUES
(%s, %s, %s)
ON CONFLICT(column1) DO UPDATE
SET
column2=excluded.column2,
column3=excluded.column3;
""", (result, column2, column3)
)
Now the result of query 1 is stored in the second table in the first column.
Now you can close your connection:
conn.close()
Related
I wrote a fairly simple SQL while loop and tried to submit it via pyodbc cursor. But it didn't work, while working perfectly fine in SQL Server Management Studio.
My understanding is that one cannot pass more than one statement with the cursor. But then how does one execute a SQL while loop? I know I can do the below query with the while loop inside the python by cursor.rowcount, but my question is about generic queries with various SQL functions (like while here).
conn = get_output_conn(env=ENVIRONMENT)
conn.autocommit=True
cursor = conn.cursor()
query = """WHILE 1 = 1
BEGIN
BEGIN TRANSACTION;
DELETE TOP(2000)
FROM table with(holdlock)
WHERE ReportDate = '2020-08-23';
IF ##ROWCOUNT < 1 BREAK;
COMMIT TRANSACTION;
END"""
cursor.execute(query)
cursor.commit()
Try testing your rowcount condition after the commit transaction; statement. The following works for me...
import pyodbc
conn = pyodbc.connect(
autoCommit=False,
driver="/usr/local/lib/libtdsodbc.so",
tds_version="7.4",
database="StackOverflow",
port=...,
server="...",
user="...",
password="..."
)
query1 = """drop table if exists dbo.DeleteExample;"""
cursor1 = conn.cursor()
cursor1.execute(query1)
cursor1.commit()
cursor1.close()
query2 = """
select cast('2020-08-23' as date) as ReportDate
into dbo.DeleteExample
from sys.objects a, sys.objects b"""
cursor2 = conn.cursor()
cursor2.execute(query2)
# About 10,000 rows depending on your database
print(cursor2.rowcount, "rows inserted")
cursor2.commit()
cursor2.close()
query3 = """
declare #RowCount int;
while 1=1
begin
begin transaction t1;
delete top (2000)
from dbo.DeleteExample
where ReportDate = '2020-08-23';
set #RowCount = ##RowCount;
commit transaction t1;
if #RowCount < 1 break;
end"""
cursor3 = conn.cursor()
cursor3.execute(query3)
# "2000" which only is the first rowcount...
print(cursor3.rowcount, "rows deleted")
cursor3.commit()
cursor3.close()
Which outputs...
% python ./example.py
(10609, 'rows inserted')
(2000, 'rows deleted')
Executing select count(1) from StackOverflow.dbo.DeleteExample in SSMS returns a count of 0.
I have a problem getting the query results from my Python-Code. The connection to the database seems to work, but i always get the error:
"InterfaceError: No result set to fetch from."
Can somebody help me with my problem? Thank you!!!
cnx = mysql.connector.connect(
host="127.0.0.1" ,
user="root" ,
passwd="*****",
db="testdb"
)
cursor = cnx.cursor()
query = ("Select * from employee ;")
cursor.execute(query)
row = cursor.fetchall()
If your problem is still not solved, you can consider replacing the python mysql driver package and use pymysql.
You can write code like this
#!/usr/bin/python
import pymysql
db = pymysql.connect(host="localhost", # your host, usually localhost
user="test", # your username
passwd="test", # your password
db="test") # name of the data base
# you must create a Cursor object. It will let
# you execute all the queries you need
cur = db.cursor()
query = ("SELECT * FROM employee")
# Use all the SQL you like
cur.execute(query)
# print all the first cell of all the rows
for row in cur.fetchall():
print(row[0])
db.close()
This should be able to find the result you want
add this to your code
for i in row:
print(i)
you did not print anything which is why that's not working
this will print each row in separate line
first try to print(row),if it fails try to execute using the for the loop,remove the semicolon in the select query statement
cursor = connection.cursor()
rows = cursor.execute('SELECT * FROM [DBname].[dbo].TableName where update_status is null ').fetchall()
for row in rows:
ds = row[0]
state = row[1]
here row[0] represent the first columnname in the database
& row[1] represent the second columnname in the database & so on
When running a query to a mysql database using MySqlHook, cursor.execute(query) returns int 1
My code is
import logging
from airflow.hooks.mysql_hook import MySqlHook
query = "SELECT col1, col2 FROM myschema.mytable LIMIT 1"
mysql = MySqlHook(mysql_conn_id=conn_id)
conn = mysql.get_conn()
cursor = conn.cursor()
result_cursor = cursor.execute(query)
logging.info(result_cursor) # this prints out "INFO - 1" in the log
df = pd.DataFrame(result_cursor.fetchall(), columns=result_cursor.keys()) # this triggers error "ERROR - 'int' object has no attribute 'fetchall'"
I would have expected result_cursor to return a "fetchable" result, since the query is working fine.
Cursor.execute() return value is not defined by the db-api spec, but for most implementations it returns the number of rows affected by the query.
To retrieve data, you have to either iterate over the cursor or call .fetchall().
It seems I cannot save cursor.execute(query) into variable result_cursor.
To make the code work, I simply needed to define the data for the data-frame as cursor.fetchall()
cursor.execute(query)
df = pd.DataFrame(list(cursor.fetchall()), column=[col[0] for col in cursor.description])
I query a table then loop through it to Update another table.
The console Prints shows correct data.
Not sure how to debug the cursor.execute for the UPDATE query.
It is not updating on the table. It's not a permission issue. If I run update command on my SQL workbench it works fine.
cursor = conn.cursor()
cursor.execute("Select Account_Name FROM dsf_CS_WebAppView")
for row in cursor.fetchall():
try:
cursor.execute("Select fullpath FROM customerdesignmap WHERE
fullpath LIKE '%{}%'".format(row.Account_Name))
rows = cursor.fetchall()
print(len(cursor.fetchall()))
if len(rows) > 0:
for rowb in rows:
print(rowb.fullpath)
print(row.Account_Name)
if len(row.Account_Name) > 2:
cursor.execute("UPDATE customerdesignmap SET householdname = {}, msid = {} WHERE fullpath LIKE '{}'".format(row.Account_Name, row.UniqueProjectNumber, rowb.fullpath))
conn.commit()
except:
pass
Consider a pure SQL solution as SQL Server supports UPDATE and JOIN across multiple tables. This avoids the nested loops, cursor calls, and string formatting of SQL commands.
UPDATE m
SET m.householdname = v.Account_Name,
m.msid = v.UniqueProjectNumber
FROM customerdesignmap m
JOIN dsf_CS_WebAppView v
ON m.fullpath LIKE CONCAT('%', v.Account_Name, '%')
In Python, run above in a single cursor.execute() with commit() call.
cursor.execute('''my SQL Query''')
conn.commit()
How can I determine if a table exists using the Psycopg2 Python library? I want a true or false boolean.
How about:
>>> import psycopg2
>>> conn = psycopg2.connect("dbname='mydb' user='username' host='localhost' password='foobar'")
>>> cur = conn.cursor()
>>> cur.execute("select * from information_schema.tables where table_name=%s", ('mytable',))
>>> bool(cur.rowcount)
True
An alternative using EXISTS is better in that it doesn't require that all rows be retrieved, but merely that at least one such row exists:
>>> cur.execute("select exists(select * from information_schema.tables where table_name=%s)", ('mytable',))
>>> cur.fetchone()[0]
True
I don't know the psycopg2 lib specifically, but the following query can be used to check for existence of a table:
SELECT EXISTS(SELECT 1 FROM information_schema.tables
WHERE table_catalog='DB_NAME' AND
table_schema='public' AND
table_name='TABLE_NAME');
The advantage of using information_schema over selecting directly from the pg_* tables is some degree of portability of the query.
select exists(select relname from pg_class
where relname = 'mytablename' and relkind='r');
The first answer did not work for me. I found success checking for the relation in pg_class:
def table_exists(con, table_str):
exists = False
try:
cur = con.cursor()
cur.execute("select exists(select relname from pg_class where relname='" + table_str + "')")
exists = cur.fetchone()[0]
print exists
cur.close()
except psycopg2.Error as e:
print e
return exists
#!/usr/bin/python
# -*- coding: utf-8 -*-
import psycopg2
import sys
con = None
try:
con = psycopg2.connect(database='testdb', user='janbodnar')
cur = con.cursor()
cur.execute('SELECT 1 from mytable')
ver = cur.fetchone()
print ver //здесь наш код при успехе
except psycopg2.DatabaseError, e:
print 'Error %s' % e
sys.exit(1)
finally:
if con:
con.close()
I know you asked for psycopg2 answers, but I thought I'd add a utility function based on pandas (which uses psycopg2 under the hood), just because pd.read_sql_query() makes things so convenient, e.g. avoiding creating/closing cursors.
import pandas as pd
def db_table_exists(conn, tablename):
# thanks to Peter Hansen's answer for this sql
sql = f"select * from information_schema.tables where table_name='{tablename}'"
# return results of sql query from conn as a pandas dataframe
results_df = pd.read_sql_query(sql, conn)
# True if we got any results back, False if we didn't
return bool(len(results_df))
I still use psycopg2 to create the db-connection object conn similarly to the other answers here.
The following solution is handling the schema too:
import psycopg2
with psycopg2.connect("dbname='dbname' user='user' host='host' port='port' password='password'") as conn:
cur = conn.cursor()
query = "select to_regclass(%s)"
cur.execute(query, ['{}.{}'.format('schema', 'table')])
exists = bool(cur.fetchone()[0])
Expanding on the above use of EXISTS, I needed something to test table existence generally. I found that testing for results using fetch on a select statement yielded the result "None" on an empty existing table -- not ideal.
Here's what I came up with:
import psycopg2
def exist_test(tabletotest):
schema=tabletotest.split('.')[0]
table=tabletotest.split('.')[1]
existtest="SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema = '"+schema+"' AND table_name = '"+table+"' );"
print('existtest',existtest)
cur.execute(existtest) # assumes youve already got your connection and cursor established
# print('exists',cur.fetchall()[0])
return ur.fetchall()[0] # returns true/false depending on whether table exists
exist_test('someschema.sometable')
You can look into pg_class catalog:
The catalog pg_class catalogs tables and most everything else that has
columns or is otherwise similar to a table. This includes indexes (but
see also pg_index), sequences (but see also pg_sequence), views,
materialized views, composite types, and TOAST tables; see relkind.
Below, when we mean all of these kinds of objects we speak of
“relations”. Not all columns are meaningful for all relation types.
Assuming an open connection with cur as cursor,
# python 3.6+
table = 'mytable'
cur.execute(f"SELECT EXISTS(SELECT relname FROM pg_class WHERE relname = {table});")
if cur.fetchone()[0]:
# if table exists, do something here
return True
cur.fetchone() will resolve to either True or False because of the EXISTS() function.