I am writing a Python application (console based) that makes use of a PostgreSQL database (via psycopg2) and R (via rpy). It is a large procedure-based application and involves several steps and sometimes repeating of steps and do not always involve all steps.
I have is the following:
main_file.py
modules/__init__.py
modules/module1.py
modules/module2.py
functions/__init__.py
functions/function1.py
functions/function2.py
The init files just states import module1, module2 or function1, function2 depending which init file it is.
The content of the main_file.py looks something like this:
import modules
from functions import function1
class myClass():
def my_function(self):
scripts = [
# modules.module1.function,
modules.module2.function,
]
print "Welcome to the program."
function1.connect()
for i in scripts:
i
cur.close()
print "End of program"
if __name__ == '__main__':
myClass().my_function()
The reason for the loop is to comment out certain steps if I don't need them. The connect() function I'm trying to call is the psycopg2 connection. It looks like this (inside function1.py file):
import sys
import psycopg2
def connect():
try:
con = psycopg2.connect(database=dbname, user=dbuser)
cur = con.cursor()
db = cur.execute
except psycopg2.DatabaseError, e:
if con:
con.rollback()
print e
sys.exit
In the main_file.py example I'm trying to run module2, which needs to connect to the database, using something like the following:
def function:
db("SELECT * INTO new_table FROM old_table")
con.commit()
How do I get Python (2.7) to recognise the global names db, cur and con? Thus connecting once-off to the database and keeping the active connection through all steps in the program?
You should add a function to the module that initialize the DB that will return the created DB objects, and then have every module that wants to use the DB call that function:
function1.py
import sys
import psycopg2
con = cur = db = None
def connect():
global con, cur, db
try:
con = psycopg2.connect(database=dbname, user=dbuser)
cur = con.cursor()
db = cur.execute
except psycopg2.DatabaseError, e:
if con:
con.rollback()
print e
sys.exit
def get_db():
if not (con and cur and db):
connect()
return (con, cur, db)
function2.py
import function1
con, cur, db = function1.get_db()
def function:
db("SELECT * INTO new_table FROM old_table")
con.commit()
There's no way to make certain variables global to every single module in package. You have explicitly import them from whatever module they live in, or return them from a function call.
Related
I am very new to python and I just can't seem to find an answer to this error. When I run the code below I get the error
AttributeError: module 'odbc' has no attribute 'connect'
However, the error only shows in eclipse. There's no problem if I run it via command line. I am running python 3.5. What am I doing wrong?
try:
import pyodbc
except ImportError:
import odbc as pyodbc
# Specifying the ODBC driver, server name, database, etc. directly
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=PXLstr,17;DATABASE=Dept_MR;UID=guest;PWD=password')
The suggestion to remove the try...except block did not work for me. Now the actual import is throwing the error as below:
Traceback (most recent call last):
File "C:\Users\a\workspace\TestPyProject\src\helloworld.py", line 2, in <module>
import pyodbc
File "C:\Users\a\AppData\Local\Continuum\Anaconda3\Lib\site-packages\sqlalchemy\dialects\mssql\pyodbc.py", line 105, in <module>
from .base import MSExecutionContext, MSDialect, VARBINARY
I do have pyodbc installed and the import and connect works fine with the command line on windows.
thank you
The problem here is that the pyodbc module is not importing in your try / except block. I would highly recommend not putting import statements in try blocks. First, you would want to make sure you have pyodbc installed (pip install pyodbc), preferably in a virtualenv, then you can do something like this:
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=PXLstr,17;DATABASE=Dept_MR;UID=guest;PWD=password')
cursor = cnxn.cursor()
cursor.execute('SELECT 1')
for row in cursor.fetchall():
print(row)
If you're running on Windows (it appears so, given the DRIVER= parameter), take a look at virtualenvwrapper-win for managing Windows Python virtual environments: https://pypi.python.org/pypi/virtualenvwrapper-win
Good luck!
Flipper's answer helped to establish that the problem was with referencing an incorrect library in External Libraries list in eclipse. After fixing it, the issue was resolved.
What is the name of your python file? If you inadvertently name it as 'pyodbc.py', you got that error. Because it tries to import itself instead of the intended pyodbc module.
here is the solution!
simply install and use 'pypyodbc' instead of 'pyodbc'!
I have my tested example as below. change your data for SERVER_NAME and DATA_NAME and DRIVER. also put your own records.good luck!
import sys
import pypyodbc as odbc
records = [
['x', 'Movie', '2020-01-09', 2020],
['y', 'TV Show', None, 2019]
]
DRIVER = 'ODBC Driver 11 for SQL Server'
SERVER_NAME = '(LocalDB)\MSSQLLocalDB'
DATABASE_NAME = 'D:\ASPNET\SHOJA.IR\SHOJA.IR\APP_DATA\DATABASE3.MDF'
conn_string = f"""
Driver={{{DRIVER}}};
Server={SERVER_NAME};
Database={DATABASE_NAME};
Trust_Connection=yes;
"""
try:
conn = odbc.connect(conn_string)
except Exception as e:
print(e)
print('task is terminated')
sys.exit()
else:
cursor = conn.cursor()
insert_statement = """
INSERT INTO NetflixMovies
VALUES (?, ?, ?, ?)
"""
try:
for record in records:
print(record)
cursor.execute(insert_statement, record)
except Exception as e:
cursor.rollback()
print(e.value)
print('transaction rolled back')
else:
print('records inserted successfully')
cursor.commit()
cursor.close()
finally:
if conn.connected == 1:
print('connection closed')
conn.close()
I want to use a variable in my SQL script. Variable's value is 100 (just a number). I have stored it as a csv. file in this directory: C:\Users\Dino\Desktop\my_file.csv.
I want in the sql script to run this:
import os
from ask_db import ask_db_params #this script creates the connection to the database
import sys, os
def my_function(cur, conn):
sql="""
\set outputdir'C:\\Users\\Dino\\Desktop'
\set new_var :outputdir '\\my_file.csv'
copy my_file to :'new_var';"""
cur.execute(sql)
conn.commit()
if __name__ == '__main__':
try:
conn = ask_db_params()
cur = conn.cursor()
analysis_data(cur,conn)
logging.info('Data analysed.')
except Exception as e:
logging.error('Failure', exc_info=True)
exit(1)
I have the error:
syntax error at or near "\" ....
It refers to the first line.
Any help regarding the syntax?
P.S. I m running python to call the sql script. Windows OS
That won't work. \set is a psql command, not an SQL command.
You will have to use string manipulation in Python to construct an SQL string that looks like this:
COPY my_file TO 'C:\Users\Dino\Desktop\my_file.csv'
Then use execute() with that SQL string.
I have a SQLite3 DB. I need to parse 10000 files. I read some data from each file, and then query the DB with this data to get a result. My code works fine in a single process environment. But I get an error when trying to use the mulitprocessing Pool.
My approach without multiprocessing (works OK):
1. Open DB connection object
2. for f in files:
foo(f, x1=x1, x2=x2, ..., db=DB)
3. Close DB
My approach with multiprocessing (does NOT work):
1. Open DB
2. pool = multiprocessing.Pool(processes=4)
3. pool.map(functools.partial(foo, x1=x1, x2=x2, ..., db=DB), [files])
4. pool.close()
5. Close DB
I get the following error: sqlite3.ProgrammingError: Base Cursor.__init__ not called.
My DB class is implemented as follows:
def open_db(sqlite_file):
"""Open SQLite database connection.
Args:
sqlite_file -- File path
Return:
Connection
"""
log.info('Open SQLite database %s', sqlite_file)
try:
conn = sqlite3.connect(sqlite_file)
except sqlite3.Error, e:
log.error('Unable to open SQLite database %s', e.args[0])
sys.exit(1)
return conn
def close_db(conn, sqlite_file):
"""Close SQLite database connection.
Args:
conn -- Connection
"""
if conn:
log.info('Close SQLite database %s', sqlite_file)
conn.close()
class MapDB:
def __init__(self, sqlite_file):
"""Initialize.
Args:
sqlite_file -- File path
"""
# 1. Open database.
# 2. Setup to receive data as dict().
# 3. Get cursor to execute queries.
self._sqlite_file = sqlite_file
self._conn = open_db(sqlite_file)
self._conn.row_factory = sqlite3.Row
self._cursor = self._conn.cursor()
def close(self):
"""Close DB connection."""
if self._cursor:
self._cursor.close()
close_db(self._conn, self._sqlite_file)
def check(self):
...
def get_driver_net(self, net):
...
def get_cell_id(self, net):
...
Function foo() looks like this:
def foo(f, x1, x2, db):
extract some data from file f
r1 = db.get_driver_net(...)
r2 = db.get_cell_id(...)
The overall not working implementation is as follows:
mapdb = MapDB(sqlite_file)
log.info('Create NetInfo objects')
pool = multiprocessing.Pool(processes=4)
files = [get list of files to process]
pool.map(functools.partial(foo, x1=x1, x2=x2, db=mapdb), files)
pool.close()
mapdb.close()
To fix this, I think I need to create the MapDB() object inside each pool worker (so have 4 parallel/independent connections). But I'm not sure how to do this. Can someone show me an example of how to accomplish this with Pool?
What about defining foo like this:
def foo(f, x1, x2, db_path):
mapdb = MapDB(db_path)
... open mapdb
... process data ...
... close mapdb
and then change your pool.map call to:
pool.map(functools.partial(foo, x1=x1, x2=x2, db_path="path-to-sqlite3-db"), files)
Update
Another option is to handle the worker threads yourself and distribute work via a Queue.
from Queue import Queue
from threading import Thread
q = Queue()
def worker():
mapdb = ...open the sqlite database
while True:
item = q.get()
if item[0] == "file":
file = item[1]
... process file ...
q.task_done()
else:
q.task_done()
break
...close sqlite connection...
# Start up the workers
nworkers = 4
for i in range(nworkers):
worker = Thread(target=worker)
worker.daemon = True
worker.start()
# Place work on the Queue
for x in ...list of files...:
q.put(("file",x))
# Place termination tokens onto the Queue
for i in range(nworkers):
q.put(("end",))
# Wait for all work to be done.
q.join()
The termination tokens are used to ensure that the sqlite connections are closed - in case that matters.
I'm trying to execute this simple python script but it seems to do nothing: I don't get any error, I try to execute the query directly on sqlite3 and it works....I don't have any idea why isn't working, can anyone help me?
import sqlite3 as lite
import sys
con = None
try:
con = lite.connect('/home/pi/Moranberries/web/moranberries.db')
cur = con.cursor()
cur.execute("INSERT INTO sensor_interior (temperatura,humedad) VALUES (111,222)")
except lite.Error, e:
print "Error %s:" % e.args[0]
sys.exit(1)
finally:
if con:
con.close()
To execute this script I named it prueba.py an call it from terminal as this:
python prueba.py
There is no error message.
You're not committing your changes to the DB. If you call con.commit() after cur.execute, it should write the changes.
I have a raspberry pi that records temperature and stores them in a MySQL database on my website. I often toy with the script, so I am hitting ctrl+c on the running script and re executing it. I would like to properly issue close() on the database connection. How can I run a line of code when the script is exited in python?
import MySQLdb
con = MySQLdb.connect(...)
cursor = con.cursor()
# ...
#if script closes:
#con.close()
import MySQLdb
con = MySQLdb.connect(...)
cursor = con.cursor()
try:
# do stuff with your DB
finally:
con.close()
The finally clause is executed on success as well as on error (exception).
If you hit Ctrl-C, you get a KeyboardInterrupt exception.
or:
import atexit
def close_db(con):
con.close()
# any other stuff you need to run on exiting
import MySQLdb
con = MySQLdb.connect(...)
# ...
atexit.register(close_db, con=con)
See here for more info.