I am trying to insert a small set of rows into sqlite using python and getting an error "Cannot operate on a closed database"
This is my code snippet:
import sqlite3
from sqlite3 import Error
db_file = "/home/sosuser/mediaserver/camera.db"
def create_connection(db_file):
conn = None
try:
conn = sqlite3.connect(db_file)
print(sqlite3.version)
except Error as e:
print(e)
finally:
if conn:
conn.close()
return conn
def create_task(conn, task):
sql = ''' INSERT INTO camerainfo(id, cameraid, maplat, maplong, name)
VALUES(?,?,?,?,?) '''
cur = conn.cursor()
cur.execute(sql, task)
def prepare_data(conn):
for cam in range(len(camID)):
print(camID[cam])
task = (cam, camID[cam], '12.972442','77.580643','testCAM')
create_task(conn, task)
conn.commit()
conn.close()
conn = create_connection(db_file)
prepare_data(conn)
Get the following error -
Traceback (most recent call last):
File "dumpCamera1.py", line 92, in <module>
prepare_data(conn)
File "dumpCamera1.py", line 86, in prepare_data
create_task(conn, task)
File "dumpCamera1.py", line 79, in create_task
cur = conn.cursor()
sqlite3.ProgrammingError: Cannot operate on a closed database.
Not sure where is my connection being closed. Might have done something very silly but would appreciate for pointers?
Thanks.
The finally clause in the create_connection function closes the connection before it's returned.
It looks as if you are trying to create a kind of context manager for the connection, but an sqlite3 Connection is already a context manager, so this is unnecessary.
You can do
with sqlite3.connect(dbfile) as conn:
print(sqlite3.version)
prepare_data(conn)
The connection will be closed automatically on exiting the context manager. You can trap errors raised inside the context manager by wrapping it in a try / except block.
Related
I'm getting following error
Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 473, in dumps
return cloudpickle.dumps(obj, pickle_protocol)
File "/databricks/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/databricks/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle 'psycopg2.extensions.cursor' object
PicklingError: Could not serialize object: TypeError: cannot pickle 'psycopg2.extensions.cursor' object
while running the below script
def get_connection():
conn_props = brConnect.value
print(conn_props)
#extract value from broadcast variables
database = conn_props.get("database")
user = conn_props.get("user")
pwd = conn_props.get("password")
host = conn_props.get("host")
db_conn = psycopg2.connect(
host = host,
user = user,
password = pwd,
database = database,
port = 5432
)
return db_conn
def process_partition_up(partition, db_cur):
updated_rows = 0
try:
for row in partition:
process_row(row, myq, db_cur)
except Exception as e:
print("Not connected")
return updated_rows
def update_final(df, db_cur):
df.rdd.coalesce(2).foreachPartition(lambda x: process_partition_up(x, db_cur))
def etl_process():
for id in ['003']:
conn = get_connection()
for t in ['email_table']:
query = f'''(select * from public.{t} where id= '{id}') as tab'''
df_updated = load_data(query)
if df_updated.count() > 0:
q1 = insert_ops(df_updated, t) #assume this function returns a insert query
query_props = q1
sc = spark.sparkContext
brConnectQ = sc.broadcast(query_props)
db_conn = get_connection()
db_cur = db_conn.cursor()
update_final(df_updated, db_cur)
conn.commit()
conn.close()
Explanation:
Here etl_process() internally calling get_connection() which returns a psycopg2 connection object. After that it's calling a update_final() which takes dataframe and psycopg2 cursor object as an arguments.
Now update_final() is calling process_partition_up() on each partition(df.rdd.coalesce(2).foreachPartition) which takes dataframe and psycopg2 cursor object as an arguments.
Here after passing psycopg2 cursor object to the process_partition_up(), I'm not getting cursor object rather I'm getting above error.
Can anyone help me out to resolve this error?
Thank you.
I think that you don't understand what's happening here.
You are creating a database connection in your driver(etl_process), and then trying to ship that live connection from the driver, across your network to executor to do the work.(your lambda in foreachPartitions is executed on the executor.)
That is what spark is telling you "cannot pickle 'psycopg2.extensions.cursor'". (It can't serialize your live connection to the database to ship it to an executor.)
You need to call conn = get_connection() from inside process_partition_up this will initialize the connection to the database from inside the executor.(And any other book keeping you need to do.)
FYI: The worst part that I want to call out is that this code will work on your local machine. This is because it's both the executor and the driver.
i have this psudo code which i want to close the mysql connection after the for loop
but i am getting the below error:
Traceback (most recent call last):
File "./myscript.py", line 201, in <module>
db.close
NameError: name 'db' is not defined
the code looks like the below:
def get_content(id):
db = mysql.connector.connect(host='localhost',user='user',password='password',database='dcname')
#get cursor
cursor = db.cursor()
cursor.execute("select id,num from table where job_db_inx={0}".format(index))
result = cursor.fetchall()
for job in list
id = get_content(id)
print(id)
db.close()
where should i place the db.close to close all the db connections
Consider using a context manager here:
import contextlib
def get_content(id, cursor):
cursor.execute("select id,num from table where job_db_inx={0}".format(index))
result = cursor.fetchall()
with contextlib.closing(mysql.connector.connect(...)) as conn:
cursor = conn.cursor()
for job in list
id = get_content(id, cursor)
print(id)
I used contextlib.closing here, but there's a very good chance that any given db api is already implemented as its own context manager. Python has a standard dbapi that is worth reading.
The actual code should look like: (make sure conn is closed)
def get_content(id,conn):
cursor = conn.cursor()
cursor.execute("select id,num from table where job_db_inx={0}".format(id))
result = cursor.fetchall()
return result
try:
conn = mysql.connector.connect(host='localhost',user='user',password='password',database='dcname')
for job in list
id = get_content(id,conn)
print(id)
finally:
conn.close()
With psycopg2, connection and querying the database works like so
conn = psycopg2.connect('connection string')
with conn:
cur=conn.cursor()
cur.execute("SELECT * FROM pg_stat_activity") #simple query
rows = cur.fetchall()
for row in rows:
print (row)
After trial and error, I found out that with conn is absolutely necessary or you will get many unexplained locks.
My question is: is there a way to setup the connection to avoid the need to use it?
From https://www.psycopg.org/docs/usage.html,
Warning
Unlike file objects or other resources, exiting the connection’s with
block doesn’t close the connection, but only the transaction
associated to it. If you want to make sure the connection is closed
after a certain point, you should still use a try-catch block:
conn = psycopg2.connect(DSN)
try:
# connection usage
finally:
conn.close()
In psycopg, the Context manager has been implemented in such a way that the with statement will only terminate the transaction and not close the connection for you. The connection needs to be closed by you separately.
In case of an error with your transaction, you have the option to rollback and raise the error.
One way to do this is to write the connection closing logic yourself.
def with_connection(func):
"""
Function decorator for passing connections
"""
def connection(*args, **kwargs):
# Here, you may even use a connection pool
conn = psycopg.connect(DSN)
try:
rv = func(conn, *args, **kwargs)
except Exception as e:
conn.rollback()
raise e
else:
# Can decide to see if you need to commit the transaction or not
conn.commit()
finally:
conn.close()
return rv
return connection
#with_connection
def run_sql(conn, arg1):
cur = conn.cursor()
cur.execute(SQL, (arg1))
Since Version 2.5, psycopg2 should support the with statement like you expect it to behave.
Docs
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
I am trying to update a mysql database using class system but cannot get the update part to work. It was all ok the old way but I wanted to use the class system with exception error control. Can someone please let me know what I am doing wrong. At the moment for this script I am just trying to send the variable Boileron to the database column office.
import MySQLdb
class DBSolar:
conn = None
def connect(self):
try:
self.conn = MySQLdb.connect("192.xxx.x.x", "exxxxx", "Oxxxx", "hxxxx")
except (MySQLdb.Error, MySQLdb.Warning) as e:
print (e)
self.conn = None
return self.conn
def query(self, sql):
try:
cursor = self.conn.cursor()
cursor.execute(sql)
except (AttributeError, MySQLdb.OperationalError):
self.connect()
cursor = self.conn.cursor()
cursor.execute(sql)
return cursor
def update(self, task):
boilerState = task
try:
sql = "UPDATE dashboard SET office = ? WHERE id = 1", (boilerState)
cursor = self.conn.cursor()
cursor.execute(sql)
except (AttributeError, MySQLdb.OperationalError):
self.connect()
cursor = self.conn.cursor()
cursor.execute(sql)
return
while 1:
BoilerOn = 1
print BoilerOn
dbSolar = DBSolar()
connSolar = dbSolar.connect()
if connSolar:
dbSolar.update(BoilerOn)
Below is the error report from putty console
Traceback (most recent call last):
File "test2.py", line 47, in <module>
dbSolar.update(BoilerOn)
File "test2.py", line 29, in update
cursor.execute(sql)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 223, in execute
self.errorhandler(self, TypeError, m)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorvalue
TypeError: query() argument 1 must be string or read-only buffer, not tuple
Got this working changed the update to the following
def update(self, task):
try:
cursor = self.conn.cursor()
cursor.execute("UPDATE dashboard SET office = %s WHERE id = 1", [task])
self.conn.commit()
except (MySQLdb.Error, MySQLdb.Warning) as e:
print (e)
self.connect()
#cursor = self.conn.cursor()
#cursor.execute(sql)
#self.connect.commit()
return
In the MySQL Developer document
Since by default Connector/Python turns autocommit off, and MySQL 5.5 and higher uses transactional InnoDB tables by default, it is necessary to commit your changes using the connection's commit() method. You could also roll back using the rollback() method.
You need to add self.conn.commit() after cursor.execute(sql) to commit all the changes.
Or turn on the autocommit which is described in Python mySQL Update, Working but not updating table
Your sql creata a tuple, but cursor.execute(statement, params) is expecting a string for the statement and either a tuple or a dictionary for params.
For MySQL UPDATE, you also need to commit the execution.
Therefore try:
self.conn.cursor.execute("UPDATE dashboard SET office = %s WHERE id = 1", (str(task), ))
self.conn.commit()
I would suggest you read the 'MySQL Connector/Python Development Guide better understating of using MySQL in python.
The following cx_Oracle code works fine when the database is up:
#!C:\Python27
import cx_Oracle
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
try:
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
finally:
curs.close()
finally:
conn.close()
But if the database happens to be down when I run this script, a NameError is raised:
Traceback (most recent call last):
File "C:\Users\ArtMetzer\Documents\Code\Python\db_conn_test.py", line 14, in <module>
conn.close()
NameError: name 'conn' is not defined
This makes sense to me: cx_Oracle wasn't able to instantiate a connection, so the variable conn never got set, and hence has no close() method.
In Python, what's the best way to ensure your database connection closes, while still gracefully handling the condition of a down database?
Doing something like the following seems like a massive kludge to me:
finally:
try:
conn.close()
except NameError:
pass
You can try initializing conn to something like None before-hand and testing that in the finally block. This works because the only place the connection is set to something else is when it is opened. So opened implies non-None and None implies not-opened:
#!C:\Python27
import cx_Oracle
conn = None
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
try:
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
finally:
curs.close()
finally:
if conn is not None:
conn.close()
According to the docs, you don't really need to bother closing the connection:
https://cx-oracle.readthedocs.io/en/latest/user_guide/connection_handling.html#closing-connections
The important line being:
Alternatively, you may prefer to let connections be automatically cleaned up when references to them go out of scope. This lets cx_Oracle close dependent resources in the correct order.
(not exactly an answer, but comments don't have nice formatting)
Try this:
#!C:\Python27
import cx_Oracle
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
try:
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
finally:
curs.close()
conn.close()
except Exception as e:
print e
Not ideal, but should work better. I'm also wondering why so much nesting. Why not do this:
#!C:\Python27
import cx_Oracle
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
curs.close()
conn.close()
except Exception as e:
print e
BTW, I have this assumption that the connection and the cursor will close automatically on exit, removing the need to close them explicitly.