Develop Reference

psycopg2 AttributeError: 'Cursor' object has no attribute 'mogrify'

I have a class that helps me with SQL queries and inserts in Postgres. I'm using psycopg2==2.7.5 right now. One of the methods I'm using looks like this:
import pandas as pd
import psycopg2.extensions as ps_ext
from typing import List
def insert_with_open_connection(self, df: pd.DataFrame, table_name: str, cursor: ps_ext.cursor,
conn: ps_ext.connection,
success_msg: str = 'Success',
conflict_cols: List[str] = None):
try:
# Format the INSERT SQL query
cols = str(tuple(df.columns)).replace("'", '')
nc = df.shape[1]
ss = "(" + ''.join('%s,' for _ in range(nc))[:-1] + ")"
try:
args_str = str(b','.join(cursor.mogrify(ss, x) for x in df.values), 'utf-8')
except psycopg2.ProgrammingError:
args_str = str(b','.join(cursor.mogrify(ss, x) for x in self.clean_numpy_int_for_mogrify(df.values)),
'utf-8')
args_str = args_str.replace("\'NaN\'::float", 'NULL')
insert_sql = f'INSERT INTO {table_name} {cols} VALUES {args_str}'
if conflict_cols is not None:
conf_cols = str(tuple(conflict_cols)).replace("'", '').replace(',)', ')')
insert_sql += f"\nON CONFLICT {conf_cols} DO NOTHING"
insert_sql += ';'
cursor.execute(insert_sql)
conn.commit()
return success_msg, 200
except Exception:
return traceback.format_exc(), 400
The conn and cursor parameters are generated from a SqlAlchemy Engine with code like this:
def create_pool(self, **db_config):
db_user = self.user
db_pass = self.password
db_name = self.database
# e.g. "/cloudsql"
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
# i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
cloud_sql_connection_name = os.environ.get("CLOUD_SQL_CONNECTION_NAME",
'<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>')
self.pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL.create(drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={"unix_sock":
f"{db_socket_dir}/{cloud_sql_connection_name}/.s.PGSQL.5432"}),
**db_config
)
def get_db_connection(self) -> Connection:
if self.pool is None:
self.create_pool()
assert isinstance(self.pool, Engine)
try:
return self.pool.raw_connection()
except psycopg2.OperationalError:
self.create_pool()
return self.pool.raw_connection()
#contextlib.contextmanager
def db_connect(self):
db = self.get_db_connection()
cur = db.cursor()
try:
yield db, cur
finally:
db.close()
I'm trying to use this code inside a Google Cloud Function (Linux) and I get the following error/traceback when I run the insert_with_open_connection method there:
Traceback (most recent call last):
File "/workspace/db/sql_helper.py", line 221, in insert_with_open_connection
args_str = str(b','.join(cursor.mogrify(ss, x) for x in df.values), 'utf-8')
File "/workspace/db/sql_helper.py", line 221, in <genexpr>
args_str = str(b','.join(cursor.mogrify(ss, x) for x in df.values), 'utf-8')
AttributeError: 'Cursor' object has no attribute 'mogrify'
It's obvious that the cursor in the code doesn't seem to have the attribute mogrify, but based on the docs here, the mogrify method should exist.

I took a look at the code and noticed that you were using import psycopg2.extensions as ps_ext; and clearly that had mogrify in
the docs.
Then I came across this line:
self.pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL.create(drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={"unix_sock":
f"{db_socket_dir}/{cloud_sql_connection_name}/.s.PGSQL.5432"}),
**db_config
)
You aren't using the psycopg2 driver; but the pg8000 one and tracing
the way things are generated, the cursor as returned by the db.cursor() which in turn was created by the self.pool.raw_connection(),
I came to the conclusion that the cursor wasn't a ps_ext cursor but
a pg8000 cursor, which doesn't have the mogrify method as
shown in: https://github.com/tlocke/pg8000/blob/main/pg8000/dbapi.py
This is the likelihood of why you're having this error. I think
the solution is to change to using psycopg2 driver instead.
That said, this answer could be wrong and I'm barking up the wrong tree.

python question :sqlite3.ProgrammingError: Cannot operate on a closed cursor

I wrote a database-select function
def select_data():
conn = sqlite3.connect(dbpath)
cur = conn.cursor()
sql = "select VC_FUNDCODE from data_ds limit 100;"
fund_list = cur.execute(sql)
cur.close()
conn.close()
return fund_list
fund_list = select_data()
datalist = []
for item in fund_list:
datalist.append(item)
print(datalist)
Interpreter returns a traceback when i try to call it:
Traceback (most recent call last):
File "/Users/chinalife/Desktop/source/flaskTiantian/database.py", line 70, in
for item in fund_list:
sqlite3.ProgrammingError: Cannot operate on a closed cursor.

Thank you guys, maybe this a simple problem, but I am so happy I could figure it out, here is my corrected code:
#
def select_data():
conn = sqlite3.connect(dbpath)
cur = conn.cursor()
sql = "select VC_FUNDCODE from data_ds limit 100;"
fund_list = cur.execute(sql)
datalist = []
for item in fund_list:
datalist.append(item)
print(datalist)
cur.close()
conn.close()
return datalist
datalist = select_data()
print(datalist)

Why is this query not working from python to SQL?

I am new to coding and databases, I can not get the query to work if I write it long hand but I have a lot to carry out and want it in a function but cannot get it to work, it returns a parameters error
import mysql.connector
def connection_check_1(query, value):
mydb = mysql.connector.connect(
host="******",
user="*****",
passwd="*****",
database="****"
)
mycursor = mydb.cursor()
mycursor.execute(query, (value))
myresult = mycursor.fetchall()
mydb.close()
return myresult
value = "sheep"
query = 'select inlicence from licence where animal = %s'
myresult = connection_check_1(query, value)
print(myresult)
Here is the SQL table I have
create table licence
(
animal varchar (20) primary key,
inlicence int (1)
);
This is the error I get
Traceback (most recent call last):
File "*******************", line 20, in
myresult = connection_check_1(query, value)
File "********************", line 13, in connection_check_1
mycursor.execute(query, (value))
File "********************************************88", line 246, in execute
prepared = self._cnx.prepare_for_mysql(params)
File "/home/kev/PycharmProjects/test bed/venv/lib/python3.5/site-packages/mysql/connector/connection_cext.py", line 535, in prepare_for_mysql
raise ValueError("Could not process parameters")
ValueError: Could not process parameters
I have tried changing the way the query is written, changing it to fetchall().

Wrapping a value with () doesn't turn it in to a tuple. You probably meant to add a comma there:
mycursor.execute(query, (value,))
# Creates a one-element tuple-^

assigning variable to integer value retrieve from mysql table

I have used multiprocessing to write code for sensing data from three sensors. Actually I have retrive data from mysql table and assigned it to the variable. It is worked for first two sensors but for the third sensor it gives error. The small part of code is as follows,
def humidity_sensor():
conn = MySQLdb.connect(host = 'localhost', user = 'root',passwd= 'mumbai',db = 'Sensordata')
cursor = conn.cursor()
cursor.execute("SELECT `DVCPORTmc1_ID`, `DVCPORTmc1_DVCm_ID`,
`DVCPORTmc1_Port_No`, `DVCPORTmc1_SENSRm_ID`, `DVCPORTmc1_Start_Dt`,\
`DVCPORTmc1_Start_Time`, `DVCPORTmc1_End_Dt`, `DVCPORTmc1_End_Time`,
`DVCPORTmc1_Read_Interval`,`DVCPORTmc1_Added_by`)
FROM `M_Device_Port` WHERE DVCPORTmc1_data_del='0' and
DVCPORTmc1_Is_Active='0' and DVCPORTmc1_DVCm_ID = '16' and
DVCPORTmc1_Description='humidity sensor'")
row=cursor.fetchone()
interval3=int(row[8])
while True:
try:
date = time.strftime("%d/%m/%Y")
clock = time.strftime("%H:%M")
temp,humidity,light = main.reading()
cursor.execute("INSERT INTO sensor(date, clock, port,value)
VALUES(%s, %s, %s,%s)",(date, clock,2,humidity))
conn.commit()
time.sleep(interval3)
except:
conn.rollback()
p1=Process(target=light_sensor)
p2=Process(target=temp_sensor)
p3=Process(target=humidity_sensor)
p1.start()
p2.start()
p3.start()
which gives error as:
Traceback (most recent call last):
interval3=int(row[8])
Typeerror: 'NoneType'object has no attribute '__getitem__'

The error is here:
interval3=int(row[8])
[x] is the getitem method. But as I see here, row is None.
That's why you can't acces the element at index 8, because row is None
and you can't call getitem on None.
This means cursor.fetchone() returns None.

There is a problem with query to retrieve data from mysql table.It gives the output none i.e. when using cursor.fetchone () it gives NONE. So the integer value which is assigned for interval3 is NONE.

Rows as dictionaries - pymssql

I want to receive rows as dictionaries in pymssql. in python-idle i ran:
>>> conn = pymssql.connect(host='192.168.1.3', user='majid', password='123456789', database='GeneralTrafficMonitor', as_dict=True)
>>> cur = conn.cursor()
>>> cur.execute('SELECT TOP 10 * FROM dbo.tblTrafficCounterData')
>>> cur.as_dict
True
>>> for row in cur:
print row['ID']
But it gives:
Traceback (most recent call last):
File "<pyshell#83>", line 2, in <module>
print row['ID']
TypeError: tuple indices must be integers, not str
Could someone help?

You need to add the parameter as_dict=True like so:
cursor = conn.cursor(as_dict=True)
You will then be able to access row['id'] if it is a field name in the result set.
As per documentation below:
https://pythonhosted.org/pymssql/pymssql_examples.html#rows-as-dictionaries

Look at the version of pymssql that you are using. Only since 1.0.2 does it return a dict, earlier versions seem to return tuple.

It is possible to set as_dict=True while creating the connection itself.
pymssql.connect(server='.', user='', password='', database='', as_dict=True,)
https://pythonhosted.org/pymssql/ref/pymssql.html?highlight=connect#pymssql.connect

Specifying results as dictionaries can be done on a cursor by cursor basis:
import pymysql
from pymysql.cursors import DictCursor
# create database connection
# connect to database
mydb = pymysql.connect( ... )
mycursor = mydb.cursor(cursor=DictCursor)