Develop Reference

Python Package and Methods not Importing

I built a simple class with a couple methods to make my life a little easier when loading data into Postgres with Python. I also attempted to package it so I could pip install it (just to experiment, never done that before).
import psycopg2
from sqlalchemy import create_engine
import io
class py_psql:
engine = None
def engine(self, username, password, hostname, port, database):
connection = 'postgresql+psycopg2://{}:{}#{}:{}/{}'.format(ntid.lower(), pw, hostname, port, database)
self.engine = create_engine(connection)
def query(self, query):
pg_eng = self.engine
return pd.read_sql_query(query, pg_eng)
def write(self, write_name, df, if_exists='replace', index=False):
mem_size = df.memory_usage().sum()/1024**2
pg_eng = self.engine
def write_data():
df.head(0).to_sql(write_name, pg_eng, if_exists=if_exists,index=index)
conn = pg_eng.raw_connection()
cur = conn.cursor()
output = io.StringIO()
df.to_csv(output, sep='\t', header=False, index=False)
output.seek(0)
contents = output.getvalue()
cur.copy_from(output, write_name, null="")
conn.commit()
if mem_size > 100:
validate_size = input('DataFrame is {}mb, proceed anyway? (y/n): '.format(mem_size))
if validate_size == 'y':
write_data()
else:
print("Canceling write to database")
else:
write_data()
My package directory looks like this:
py_psql
py_psql.py
__init__.py
setup.py
My init.py is empty since I read elsewhere that I was able to do that. I'm not remotely an expert here...
I was able to pip install that package and import it, and if I were to paste this class into a python shell, I would be able to do something like
test = py_psql()
test.engine(ntid, pw, hostname, port, database)
and have it create the sqlalchemy engine. However, when I import it after the pip install I can't even initialize a py_psql object:
>>> test = py_psql()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
>>> py_psql.engine(ntid, pw, hostname, port, database)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'py_psql' has no attribute 'engine'
I'm sure I'm messing up something obvious here, but I found the process of packaging fairly confusing while researching this. What am I doing incorrectly?

Are you sure you imported your package correctly after pip install?
For example:
from py_psql.py_psql import py_psql
test = py_psql()
test.engine(ntid, pw, hostname, port, database)

I get NotImplementedError when trying to do a prepared statement with mysql python connector

I want to use prepared statements to insert data into a MySQL DB (version 5.7) using python, but I keep getting a NotImplementedError.
I'm following the documentation here: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html
Using Python 2.7 and version 8.0.11 of mysql-connector-python library:
pip show mysql-connector-python
---
Metadata-Version: 2.1
Name: mysql-connector-python
Version: 8.0.11
Summary: MySQL driver written in Python
Home-page: http://dev.mysql.com/doc/connector-python/en/index.html
This is a cleaned version (no specific hostname, username, password, columns, or tables) of the python script I'm running:
import mysql.connector
from mysql.connector.cursor import MySQLCursorPrepared
connection = mysql.connector.connect(user=username, password=password,
host='sql_server_host',
database='dbname')
print('Connected! getting cursor')
cursor = connection.cursor(cursor_class=MySQLCursorPrepared)
select = "SELECT * FROM table_name WHERE column1 = ?"
param = 'param1'
print('Executing statement')
cursor.execute(select, (param,))
rows = cursor.fetchall()
for row in rows:
value = row.column1
print('value: '+ value)
I get this error when I run this:
Traceback (most recent call last):
File "test.py", line 18, in <module>
cursor.execute(select, (param,))
File "/home/user/.local/lib/python2.7/site-packages/mysql/connector/cursor.py", line 1186, in execute
self._prepared = self._connection.cmd_stmt_prepare(operation)
File "/home/user/.local/lib/python2.7/site-packages/mysql/connector/abstracts.py", line 969, in cmd_stmt_prepare
raise NotImplementedError
NotImplementedError

CEXT will be enabled by default if you have it, and prepared statements are not supported in CEXT at the time of writing.
You can disable the use of CEXT when you connect by adding the keyword argument use_pure=True as follows:
connection = mysql.connector.connect(user=username, password=password,
host='sql_server_host',
database='dbname',
use_pure=True)
Support for prepared statements in CEXT will be included in the upcoming mysql-connector-python 8.0.17 release (according to the MySQL bug report). So once that is available, upgrade to at least 8.0.17 to solve this without needing use_pure=True.

Python - invalid syntax in both 2.7 and 3.6, but at different places - revised code [closed]

Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 4 years ago.
Improve this question
My code below is for inserting data into a PostgreSQL database - I've commented out the db code (just trying to print the SQL string instead). However, for some reason, I just can't get past the syntax errors.
I'm using Fedora 28 64 bit.
> python2.7 filename.py
or
> python3.6 filename.py
Following insightful comments, I now have a problem with psycopg2 with both 2.7 and 3.6.
2.7
[pol#polhost db]$ python2.7 ./ace_2_pg.py
I am unable to connect to the database
Traceback (most recent call last):
File "./ace_2_pg.py", line 12, in <module>
cur = conn.cursor()
NameError: name 'conn' is not defined
3.6
[pol#polhost db]$ python3.6 ./ace_2_pg.py
Traceback (most recent call last):
File "./ace_2_pg.py", line 4, in <module>
import psycopg2
ModuleNotFoundError: No module named 'psycopg2'
[pol#polhost db]$
I just can't understand it!
Try to install psycopg2, but it's already installed. Has to be, since I've successfully inserted data before now!
python2.7
[pol#polhost db]$ pip install psycopg2
Requirement already satisfied: psycopg2 in /usr/lib64/python2.7/site-packages
python3.6
[pol#polhost db]$ pip3 install psycopg2
Traceback (most recent call last):
File "/usr/local/bin/pip3", line 7, in <module>
from pip._internal import main
ImportError: No module named _internal
I would be grateful if
a) I could be provided with working python for both 2.7 and 3.6, and
b) much more importantly, receive an explanation of where it is I'm going wrong!
I've included the input file (very short) at the end for testing - outputting of SQLString is sufficient.
============ python code ======================
from __future__ import print_function
import psycopg2
try:
conn = psycopg2.connect("dbname='test' user='postgres' host='localhost'")
print ('Connected!')
except:
print ('I am unable to connect to the database')
cur = conn.cursor()
with open('/home/pol/Downloads/sware/acedb/ace_bis/src/dump/dump_2018-05-18_A_Jade.1.ace') as f:
for line in f:
words = line.split()
print ('Line length is: ' + len(line))
if "\\" in line:
continue
if line[0:3] == " \\ ":
print (line + 'beginning of data')
continue
if (len(words) == 3):
print (words[0], ' 1 ', words[1], '2', words[2], '3')
SQLString = "INSERT INTO jade VALUES ('" + words[0] + "', '" + words[1] + "', '" + words[2] + "');"
print (SQLString)
'''
try:
cur.execute(SQLString)
print('Inserted')
except:
print('Insert failed')
'''
conn.commit()
cur.close()
conn.close()
# EOF
================= Input file ==================
[pol#polhost dump]$ more dump_2018-05-18_A_Jade.1.ace
// Class Jade
Jade "default"
Display "Chromosome" "jade.maps.Vmap"
Display "Map" "jade.maps.Vmap"
Display "Table" "jade.maps.TableDisplay"
Display "Cell" "lignage.Lignage"
Display "Node" "lignage.Lignage"
// End of this dump file
[pol#polhost dump]$
===================================

I can explain name conn not defined:
try:
conn = psycopg2.connect("dbname='test' user='postgres' host='localhost'")
print ('Connected!')
except:
print ('I am unable to connect to the database')
cur = conn.cursor()
In the try block, your code attempts to connect to the database, and fails.
In the except block, you code prints I am unable to connect to the database and then carries on.
In the next line you refer to conn.cursor(). But the original assignment to conn in the try block failed, so the assignment didn't take place, so the name is undefined.

AttributeError: module 'odbc' has no attribute 'connect' - python with pydev

I am very new to python and I just can't seem to find an answer to this error. When I run the code below I get the error
AttributeError: module 'odbc' has no attribute 'connect'
However, the error only shows in eclipse. There's no problem if I run it via command line. I am running python 3.5. What am I doing wrong?
try:
import pyodbc
except ImportError:
import odbc as pyodbc
# Specifying the ODBC driver, server name, database, etc. directly
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=PXLstr,17;DATABASE=Dept_MR;UID=guest;PWD=password')
The suggestion to remove the try...except block did not work for me. Now the actual import is throwing the error as below:
Traceback (most recent call last):
File "C:\Users\a\workspace\TestPyProject\src\helloworld.py", line 2, in <module>
import pyodbc
File "C:\Users\a\AppData\Local\Continuum\Anaconda3\Lib\site-packages\sqlalchemy\dialects\mssql\pyodbc.py", line 105, in <module>
from .base import MSExecutionContext, MSDialect, VARBINARY
I do have pyodbc installed and the import and connect works fine with the command line on windows.
thank you

The problem here is that the pyodbc module is not importing in your try / except block. I would highly recommend not putting import statements in try blocks. First, you would want to make sure you have pyodbc installed (pip install pyodbc), preferably in a virtualenv, then you can do something like this:
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=PXLstr,17;DATABASE=Dept_MR;UID=guest;PWD=password')
cursor = cnxn.cursor()
cursor.execute('SELECT 1')
for row in cursor.fetchall():
print(row)
If you're running on Windows (it appears so, given the DRIVER= parameter), take a look at virtualenvwrapper-win for managing Windows Python virtual environments: https://pypi.python.org/pypi/virtualenvwrapper-win
Good luck!

Flipper's answer helped to establish that the problem was with referencing an incorrect library in External Libraries list in eclipse. After fixing it, the issue was resolved.

What is the name of your python file? If you inadvertently name it as 'pyodbc.py', you got that error. Because it tries to import itself instead of the intended pyodbc module.

here is the solution!
simply install and use 'pypyodbc' instead of 'pyodbc'!
I have my tested example as below. change your data for SERVER_NAME and DATA_NAME and DRIVER. also put your own records.good luck!
import sys
import pypyodbc as odbc
records = [
['x', 'Movie', '2020-01-09', 2020],
['y', 'TV Show', None, 2019]
]
DRIVER = 'ODBC Driver 11 for SQL Server'
SERVER_NAME = '(LocalDB)\MSSQLLocalDB'
DATABASE_NAME = 'D:\ASPNET\SHOJA.IR\SHOJA.IR\APP_DATA\DATABASE3.MDF'
conn_string = f"""
Driver={{{DRIVER}}};
Server={SERVER_NAME};
Database={DATABASE_NAME};
Trust_Connection=yes;
"""
try:
conn = odbc.connect(conn_string)
except Exception as e:
print(e)
print('task is terminated')
sys.exit()
else:
cursor = conn.cursor()
insert_statement = """
INSERT INTO NetflixMovies
VALUES (?, ?, ?, ?)
"""
try:
for record in records:
print(record)
cursor.execute(insert_statement, record)
except Exception as e:
cursor.rollback()
print(e.value)
print('transaction rolled back')
else:
print('records inserted successfully')
cursor.commit()
cursor.close()
finally:
if conn.connected == 1:
print('connection closed')
conn.close()

Psycopg2 concurrency issue

I am trying to do share a psycopg2 connection between multiple threads. As was mentioned in the docs, I am doing that by creating new cursor objects from the shared connection, whenever I use it in a new thread.
def delete(conn):
while True:
conn.commit()
def test(conn):
cur = conn.cursor()
thread.start_new_thread(delete,(conn,))
i = 1
while True:
cur.execute("INSERT INTO mas(taru,s) values (2,%s)",(i,))
print i
i = i +1
conn.commit()
After running, I get output like,
1
2
...
98
99
Traceback (most recent call last):
File "postgres_test_send.py", line 44, in <module>
cur.execute("INSERT INTO mas(taru,s) values (2,%s)",(i,))
psycopg2.InternalError: SET TRANSACTION ISOLATION LEVEL must be called before any query
What's going on here?

The bug is not in the most recent psycopg2 versions: it has probably been fixed in 2.4.2.