I have the following PostgreSql table:
I want to insert a string value of 3 into the first row under the diagnosis column. I'm trying to follow some general INSERT code from the postgresql documentation. Below is my trial code that is not working.
diagnosis = 3;
pip install config
# insert
import psycopg2
from config import config
def insert_diagnosis(diagnosis):
""" insert a new diagnosis prediction into the eyeballtables table """
sql = """INSERT INTO eyeballtables(diagnosis)
VALUES(%s) RETURNING vendor_id;"""
conn = None
vendor_id = None
try:
# read database configuration
params = config()
# connect to the PostgreSQL database
conn = psycopg2.connect(**params)
# create a new cursor
cur = conn.cursor()
# execute the INSERT statement
cur.execute(sql, (diagnosis,))
# get the generated id back
vendor_id = cur.fetchone()[0]
# commit the changes to the database
conn.commit()
# close communication with the database
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
return vendor_id
The code above is not exactly right because it doesn't isolate the insert to the "diagnosis" field in the table. But beyond that I also get the following error even though I did a successful pip install of config:
ImportError: cannot import name 'config' from 'config' (/home/pinzhi/anaconda3/lib/python3.7/site-packages/config.py)
Thought on what I'm doing wrong or if there is a more straightforward way to insert a datapoint into a PostgreSql table I have?
EDIT ---------------------------------------------------------------------
The config error is no longer popping up after I followed the answer provided. But my code above is unable to insert the diagnosis value of 3 into the table.
What am I doing wrong?
it's Config not config
from config import Config
params = Config()
Related
Whenever I want to insert data in a pandas dataframe to the into a postgresql database I get this error
error: extra data after last expected column CONTEXT: COPY recommendations, line 1: "0,4070,"[5963, 8257, 9974, 7546, 11251, 5203, 102888, 8098, 101198, 10950]""
The dataframe consist of three column, the first and second column are of type integers and the third column is a list of integers.
I created a table in PostgreSQL using this function below
def create_table(query: str) -> None:
"""
:param query: A string of the query to create table in the database
:return: None
"""
try:
logger.info("Creating the table in the database")
conn = psycopg2.connect(host=HOST, dbname=DATABASE_NAME, user=USER, password=PASSWORD, port=PORT)
cur = conn.cursor()
cur.execute(query)
conn.commit()
logger.info("Successfully created a table in the database using this query {}".format(query))
return
except (Exception, psycopg2.Error) as e:
logger.error("An error occurred while creating a table using the query {} with exception {}".format(query, e))
finally:
if conn is not None:
conn.close()
logger.info("Connection closed!")
The query passed into this function is this:
create_table_query = '''CREATE TABLE Recommendations
(id INT NOT NULL,
applicantId INT NOT NULL,
recommendation INTEGER[],
PRIMARY KEY(id),
CONSTRAINT applicantId
FOREIGN KEY(applicantId)
REFERENCES public."Applicant"(id)
ON DELETE CASCADE
ON UPDATE CASCADE
); '''
I then use the function below to copy the data frame to the created table in postgres.
def copy_from_file(df: pd.DataFrame, table: str = "recommendations") -> None:
"""
Here we are going save the dataframe on disk as
a csv file, load the csv file
and use copy_from() to copy it to the table
"""
conn = psycopg2.connect(host=HOST, dbname=DATABASE_NAME, user=USER, password=PASSWORD, port=PORT)
# Save the dataframe to disk
tmp_df = "./tmp_dataframe.csv"
df.to_csv(tmp_df, index_label='id', header=False)
f = open(tmp_df, 'r')
cursor = conn.cursor()
try:
cursor.copy_from(f, table, sep=",")
conn.commit()
except (Exception, psycopg2.DatabaseError) as error:
os.remove(tmp_df)
logger.error("Error: %s" % error)
conn.rollback()
cursor.close()
logger.info("copy_from_file() done")
cursor.close()
os.remove(tmp_df)
And then I still get this error: extra data after last expected column CONTEXT: COPY recommendations, line 1: "0,4070,"[5963, 8257, 9974, 7546, 11251, 5203, 102888, 8098, 101198, 10950]"" please any recommendations on how to fix this issue? Thanks
copy_from uses the text format, not the csv format. You told it to use , as the separator, but that doesn't change the protecting method it is trying to use. So the commas inside the quotes are not treated as being protected, they are treated as field separators, and so of course there are too many of them.
I think you need to use copy_expert and tell it to use the csv format.
I am trying to fetch data from AWS MariaDB:
cursor = self._cnx.cursor()
stmt = ('SELECT * FROM flights')
cursor.execute(stmt)
print(cursor.rowcount)
# prints 2
for z in cursor:
print(z)
# Does not iterate
row = cursor.fetchone()
# row is None
rows = cursor.fetchall()
# throws 'No result set to fetch from.'
I can verify that table contains data using MySQL Workbench. Am I missing some step?
EDIT: re 2 answers:
res = cursor.execute(stmt)
# res is None
EDIT:
I created new Python project with a single file:
import mysql.connector
try:
cnx = mysql.connector.connect(
host='foobar.rds.amazonaws.com',
user='devuser',
password='devpasswd',
database='devdb'
)
cursor = cnx.cursor()
#cursor = cnx.cursor(buffered=True)
cursor.execute('SELECT * FROM flights')
print(cursor.rowcount)
rows = cursor.fetchall()
except Exception as exc:
print(exc)
If I run this code with simple cursor, fetchall raises "No result set to fetch from". If I run with buffered cursor, I can see that _rows property of cursor contains my data, but fetchall() returns empty array.
Your issue is that cursor.execute(stmt) returns an object with results and you're not storing that.
results = cursor.execute(stmt)
print(results.fetchone()) # Prints out and pops first row
For the future googlers with the same Problem I found a workaround which may help in some cases:
I didn't find the source of the problem but a solution which worked for me.
In my case .fetchone() also returned none whatever I did on my local(on my own Computer) Database. I tried the exact same code with the Database on our companies server and somehow it worked. So I copied the complete server Database onto my local Database (by using database dumps) just to get the server settings and afterwards I also could get data from my local SQL-Server with the code which didn't work before.
I am a SQL-newbie but maybe some crazy setting on my local SQL-Server prevented me from fetching data. Maybe some more experienced SQL-user knows this setting and can explain.
I want to insert a record in mytable (in DB2 database) and get the id generated in that insert. I'm trying to do that with python 2.7. Here is what I did:
import sqlalchemy
from sqlalchemy import *
import ibm_db_sa
db2 = sqlalchemy.create_engine('ibm_db_sa://user:pswd#localhost:50001/mydatabase')
sql = "select REPORT_ID from FINAL TABLE(insert into MY_TABLE values(DEFAULT,CURRENT TIMESTAMP,EMPTY_BLOB(),10,'success'));"
result = db2.execute(sql)
for item in result:
id = item[0]
print id
When I execute the code above it gives me this output:
10 //or a increasing number
Now when I check in the database nothing has been inserted ! I tried to run the same SQL request on the command line and it worked just fine. Any clue why I can't insert it with python using sqlalchemy ?
Did you try a commit? #Lennart is right. It might solve your problem.
Your code does not commit the changes you have made and thus are rolled back.
If your Database is InnoDB, it is transactional and thus needs a commit.
according to this, you also have to connect to your engine. so in your instance it would look like:
db2 = sqlalchemy.create_engine('ibm_db_sa://user:pswd#localhost:50001/mydatabase')
conn = db2.connect()
trans = conn.begin()
try:
sql = "select REPORT_ID from FINAL TABLE(insert into MY_TABLE values(DEFAULT,CURRENT TIMESTAMP,EMPTY_BLOB(),10,'success'));"
result = conn.execute(sql)
for item in result:
id = item[0]
print id
trans.commit()
except:
trans.rollback()
raise
I do hope this helps.
I wanted to start into using databases in python. I chose postgresql for the database "language". I already created several databases, but now I want simply to check if the database exists with python. For this I already read this answer: Checking if a postgresql table exists under python (and probably Psycopg2) and tried to use their solution:
import sys
import psycopg2
con = None
try:
con = psycopg2.connect(database="testdb", user="test", password="abcd")
cur = con.cursor()
cur.execute("SELECT exists(SELECT * from information_schema.testdb)")
ver = cur.fetchone()[0]
print ver
except psycopg2.DatabaseError, e:
print "Error %s" %e
sys.exit(1)
finally:
if con:
con.close()
But unfortunately, I only get the output
Error relation "information_schema.testdb" does not exist
LINE 1: SELECT exists(SELECT * from information_schema.testdb)
Am I doing something wrong, or did I miss something?
Your question confuses me a little, because you say you want to look to see if a database exists, but you look in the information_schema.tables view. That view would tell you if a table existed in the currently open database. If you want to check if a database exists, assuming you have access to the 'postgres' database, you could:
import sys
import psycopg2, psycopg2.extras
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
dbname = 'db_to_check_for_existance'
con = None
try:
con = psycopg2.connect(database="postgres", user="postgres")
cur = con.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute("select * from pg_database where datname = %(dname)s", {'dname': dbname })
answer = cur.fetchall()
if len(answer) > 0:
print "Database {} exists".format(dbname)
else:
print "Database {} does NOT exist".format(dbname)
except Exception, e:
print "Error %s" %e
sys.exit(1)
finally:
if con:
con.close()
What is happening here is you are looking in the database tables called pg_database. The column 'datname' contains each of the database names. Your code would supply db_to_check_for_existance as the name of the database you want to check for existence. For example, you could replace that value with 'postgres' and you would get the 'exists' answer. If you replace the value with aardvark you would probably get the does NOT exist report.
If you're trying to see if a database exists:
curs.execute("SELECT exists(SELECT 1 from pg_catalog.pg_database where datname = %s)", ('mydb',))
It sounds like you may be confused by the difference between a database and a table.
I am currently connecting to a Sybase 15.7 server using sybpydb. It seems to connect fine:
import sys
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib')
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/lib')
import sybpydb
conn = sybpydb.connect(user='usr', password='pass', servername='serv')
is working fine. Changing any of my connection details results in a connection error.
I then select a database:
curr = conn.cursor()
curr.execute('use db_1')
however, now when I try to run queries, it always returns None
print curr.execute('select * from table_1')
I have tried running the use and select queries in the same execute, I have tried including go commands after each, I have tried using curr.connection.commit() after each, all with no success. I have confirmed, using dbartisan and isql, that the same queries I am using return entries.
Why am I not getting results from my queries in python?
EDIT:
Just some additional info. In order to get the sybpydb import to work, I had to change two environment variables. I added the lib paths (the same ones that I added to sys.path) to $LD_LIBRARY_PATH, i.e.:
setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib:/dba/sybase/ase/15.7/OCS-15_0/lib
and I had to change the SYBASE path from 12.5 to 15.7. All this was done in csh.
If I print conn.error(), after every curr.execute(), I get:
("Server message: number(5701) severity(10) state(2) line(0)\n\tChanged database context to 'master'.\n\n", 5701)
I completely understand where you might be confused by the documentation. Its doesn't seem to be on par with other db extensions (e.g. psycopg2).
When connecting with most standard db extensions you can specify a database. Then, when you want to get the data back from a SELECT query, you either use fetch (an ok way to do it) or the iterator (the more pythonic way to do it).
import sybpydb as sybase
conn = sybase.connect(user='usr', password='pass', servername='serv')
cur = conn.cursor()
cur.execute("use db_1")
cur.execute("SELECT * FROM table_1")
print "Query Returned %d row(s)" % cur.rowcount
for row in cur:
print row
# Alternate less-pythonic way to read query results
# for row in cur.fetchall():
# print row
Give that a try and let us know if it works.
Python 3.x working solution:
import sybpydb
try:
conn = sybpydb.connect(dsn="Servername=serv;Username=usr;Password=pass")
cur = conn.cursor()
cur.execute('select * from db_1..table_1')
# table header
header = tuple(col[0] for col in cur.description)
print('\t'.join(header))
print('-' * 60)
res = cur.fetchall()
for row in res:
line = '\t'.join(str(col) for col in row)
print(line)
cur.close()
conn.close()
except sybpydb.Error:
for err in cur.connection.messages:
print(f'Error {err[0]}, Value {err[1]}')