I have the following procedure that doesn't load data into a table as expected:
def upload_data(parsed_buffer):
parsed_buffer.seek(0)
try:
con = psycopg2.connect(database=database, user=user, password=pw, host=host)
cur = con.cursor()
try:
cur.copy_from(parsed_buffer, 'staging.vcf_ht')
except StandardError, err:
conn.rollback()
print(" Caught error (as expected):\n", err)
except psycopg2.NotSupportedError as e:
now = time.strftime("%H:%M:%S +0000", time.localtime())
print("Copy failed at: " + now + " - " + e.pgerror)
sys.exit(1)
finally:
if con:
con.close()
now = time.strftime("%H:%M:%S +0000", time.localtime())
print('Finished loading data at:' + now)
In other posts they discuss adding the seek function after writes. That is in line 3 of my code. This did not work. A couple of other things I checked. 1. The string buffer is populated with tab delimited data. 2. If I redirect the output to a file and use the \copy command in psql it works as advertised. 3. If I write an insert statement instead of a string buffer this also works (but this is bad for performance). This procedure terminates with throwing any errors.
The problem was no commit statement. Added: con.commit()
Related
Relatively new to python scripts, so bare with.
I have used speedtest-cli before. I have edited the script so it will insert the values into a sql table as below, however having an issue with one of the inserts. It will insert ping, and download ok, however, the upload is always 2.74 or 2.75 for example, but ONLY when run from a crontab.. very weird.
If I run the python script from cli it will insert values fine.
This is my query, and the values ping, download and upload are coming from the speedtest-cli script.
Here is the full script
import re
import subprocess
import time
import mysql.connector
from mysql.connector import Error
from mysql.connector import errorcode
print "----------------------------------"
print 'Started: {} {}'.format(time.strftime('%d/%m/%y %H:%M:%S'), "")
response = subprocess.Popen('speedtest-cli --simple', shell=True, stdout=subprocess.PIPE).stdout.read()
ping = re.findall('Ping:\s(.*?)\s', response, re.MULTILINE)
download = re.findall('Download:\s(.*?)\s', response, re.MULTILINE)
upload = re.findall('Upload:\s(.*?)\s', response, re.MULTILINE)
ping[0] = ping[0].replace(',', '.')
download[0] = download[0].replace(',', '.')
upload[0] = upload[0].replace(',', '.')
try:
if os.stat('/var/www/html/speed/log.txt').st_size == 0:
print 'Date,Time,Ping (ms),Download (Mbit/s),Upload (Mbit/s)'
except:
pass
print 'PING: {}, DOWN: {}, UP: {}'.format(ping[0], download[0], upload[0])
try:
connection = mysql.connector.connect(host='localhost',
database='dev',
user='dev',
password='dev1')
sql_insert_query = ("""INSERT INTO speedtest(ping, download, upload) VALUES (%s,%s,%s)""", (ping[0], download[0], upload[0]))
cursor = connection.cursor()
result = cursor.execute(*sql_insert_query)
connection.commit()
print ("Insert success into speedtest tbl")
except mysql.connector.Error as error :
connection.rollback() #rollback if any exception occured
print("Failed inserting record into speedtest table {}".format(error))
finally:
#closing database connection.
if(connection.is_connected()):
cursor.close()
connection.close()
print("MySQL conn closed")
print 'Finished: {} {}'.format(time.strftime('%d/%m/%y %H:%M:%S'), "")
Manual script runs ok, just from crontab I get unexpected values. Not sure how to solve.
I'm still using Flask-mysql.
I'm getting the database context (the mysql variable) just fine, and can query on the database / get results. It's only the insert that is not working: it's not complaining (throwing Exceptions). It returns True from the insert method.
This should be done inserting the record when it commits, but for some reason, as I watch the MySQL database with MySQL Workbench, nothing is getting inserted into the table (and it's not throwing exceptions from the insert method):
I'm passing in this to insertCmd:
"INSERT into user(username, password) VALUES ('test1','somepassword');"
I've checked the length of the column in the database, and copied the command into MySQL Workbench (where it successfully inserts the row into the table).
I'm at a loss. The examples I've seen all seem to follow this format, and I have a good database context. You can see other things I've tried in the comments.
def insert(mysql, insertCmd):
try:
#connection = mysql.get_db()
cursor = mysql.connect().cursor()
cursor.execute(insertCmd)
mysql.connect().commit()
#mysql.connect().commit
#connection.commit()
return True
except Exception as e:
print("Problem inserting into db: " + str(e))
return False
You need to keep a handle to the connection; you keep overriding it in your loop.
Here is a simplified example:
con = mysql.connect()
cursor = con.cursor()
def insert(mysql, insertCmd):
try:
cursor.execute(insertCmd)
con.commit()
return True
except Exception as e:
print("Problem inserting into db: " + str(e))
return False
If mysql is your connection, then you can just commit on that, directly:
def insert(mysql, insertCmd):
try:
cursor = mysql.cursor()
cursor.execute(insertCmd)
mysql.commit()
return True
except Exception as e:
print("Problem inserting into db: " + str(e))
return False
return False
Apparently, you MUST separate the connect and cursor, or it won't work.
To get the cursor, this will work: cursor = mysql.connect().cursor()
However, as Burchan Khalid so adeptly pointed out, any attempt after that to make a connection object in order to commit will wipe out the work you did using the cursor.
So, you have to do the following (no shortcuts):
connection = mysql.connect()
cursor = connection.cursor()
cursor.execute(insertCmd)
connection.commit()
I'm having a problem where I have a SQL statement that when run in my MySQL Workbench executes properly, but when run with python's mysql package function cursor.execute() doesn't work. The problem SQL statement is:
REPLACE INTO mmm_dev.samp_wp_links SELECT * FROM mmm_master.samp_wp_links;
The statement is supposed to copy all data from mmm_master into mmm_dev. The following the python code that I'm using to execute the query:
cnx = mysql.connector.connect(**config)
cursor = cnx.cursor()
def examine(cursor, cnx):
try:
qry = cursor.execute("REPLACE INTO mmm_dev.samp_wp_links SELECT * FROM mmm_master.samp_wp_links;")
except mysql.connector.Error as err:
print("Failed to select everything")
exit(1)
MySQL Python libraries are PEP 249-compliant:
.commit () Commit any pending transaction to the database.
Note that if the database supports an auto-commit feature, this must
be initially off. An interface method may be provided to turn it back
on.
Database modules that do not support transactions should implement
this method with void functionality.
Call cnx.commit()
Call examine(). Debug with print():
cnx = mysql.connector.connect(**config)
cursor = cnx.cursor()
def examine(cursor):
print("[DEBUG] 1: before query")
try:
qry = cursor.execute("REPLACE INTO mmm_dev.samp_wp_links SELECT * FROM mmm_master.samp_wp_links;")
print("[DEBUG] 2: after query")
except mysql.connector.Error as err:
print("Failed to select everything %s" % err)
exit(1)
print("[DEBUG] 3: success")
examine(cursor)
You have to do a commit.
mydb.commit()
in order to save the changes.
I am trying to connect to my database via Python 2.7 with this code:
import csv
import psycopg2
try:
conn = psycopg2.connect("dbname='student', user='postgres',password='password', host='localhost'")
cursor = conn_cursor()
reader = csv.reader(open('last_file.csv', 'rb'))
print "connected"
except:
print "not Connected"
It did work last week and we don't think we've changed anything, but now it won't connect.
We've tried using it with the database open and closed, nothing worked.
The database does exist in Postgres.
import psycopg2
try:
conn = psycopg2.connect("dbname='database_name' user='postgres_user_name' host='localhost' password='user_passwd'")
except:
print "I am unable to connect to the database"
cur = conn.cursor()
cur.execute("""SELECT * from table_name""")
rows = cur.fetchall()
print "\nShow me the data:\n"
for row in rows:
print " ", row[0]
print " ", row[1]
Exception part add like this to see what is error
except Exception as ex:
print "not Connected"
print "Error: "+ str(ex)
Try this:
import csv
import psycopg2
try:
conn = psycopg2.connect("dbname='student', user='postgres',password='password', host='localhost'")
except:
print "I am unable to connect to the database."
cursor = conn.cursor()
try:
reader = csv.reader(open('last_file.csv', 'rb'))
print "connected"
except:
print "not Connected"
Seems like there are something wrong with your postgres.
Try and see postgres log.
Location of postgres log by default :
tail -f /var/log/postgresql/<>/main/postgresql.log
something like this.
Also don't forget to check firewall. Maybe someone disable it by accident.
Also try for pip install PyGreSQL package. Since psycopg2 (some of versions) is under GPL license. It could be tricky for open source license. Just for your information.
I have a list of MS SQL CREATE scripts and I am trying to automate the process of executing each of these scripts. As this CREATE scripts does not result any records, I would want my automation script to return the SSMS output message that looks like:
'Command executed successfully'
Can I read this output message using pyodbc?
Here is the sample code that I use to execute the script:
conn = pyodbc.connect(r'DRIVER={SQL Server};SERVER=%s;Trusted_Connection=True;'% (db_conn_string))
cursor = conn.cursor()
cursor.execute(query)
It is not really necessary to capture the "Command executed successfully" message because an exception will occur if the command is not executed successfully.
So your Python code can just .execute the statement, catch any exception that occurs, and proceed accordingly, e.g.,
try:
crsr.execute("DROP TABLE dbo.nonexistent")
print("INFO: DROP TABLE succeeded.")
except pyodbc.ProgrammingError as err:
error_code = err.args[0]
if error_code == "42S02": # [table] does not exist or you do not have permission
print("INFO: DROP TABLE did not succeed.")
else:
raise # re-raise unexpected exception
cursor.rowcount
Adding the above line to your code will return the row count for the last executed query.