I am having trouble trying to copy a few tables from a mysql server to postgres server. Here is the code that I am currently using to connect, fetch the data and write the data.
my_sqlconnection = MySQLdb.connect(host='a.b.c.d',
user='user',
passwd='admin'
)
try:
pl_sqlconnection = psycopg2.connect("host='x.y.z.c' dbname='user'
user='uadmin' password='uadmin'" )
except psycopg2.Error as e:
print('PSQL: Unable to connect!\n{0}')
print (e)
print (e.pgcode)
print (e.pgerror)
print (traceback.format_exc())
sys.exit(1)
cursor1 = my_sqlconnection.cursor(MySQLdb.cursors.DictCursor)
cursor2 = pl_sqlconnection.cursor()
for db in db_list.db_set:
restaurant_name = db[:-3]
my_sqlconnection.select_db(db)
sql = "SELECT * FROM Orders"
cursor1.execute(sql)
for row in cursor1:
try:
cursor2.execute("INSERT INTO Orders (all the values) Values
(%(all the values)s)", row)
except psycopg2.Error as e:
print ("cannot execute that query!!")
print (e)
print (e.pgcode)
print (e.pgerror)
print (traceback.format_exc())
sys.exit("Some problem occured with that query! leaving early")
sql2 = "SELECT * FROM USERS"
cursor1.execute(sql2)
for row in cursor1:
try:
cursor2.execute("INSERT INTO Users (all the values) Values
(%(all the values)s)", row)
except psycopg2.Error as e:
print ("cannot execute that query!!")
print (e)
print (e.pgcode)
print (e.pgerror)
print (traceback.format_exc())
sys.exit("Some problem occured with that query! leaving early")
cursor1.close()
cursor2.close()
pl_sqlconnection.commit()
my_sqlconnection.close()
pl_sqlconnection.close()
Now the error that I am getting is
python backup.py
cannot execute that query!!
invalid byte sequence for encoding "UTF8": 0xeef1e5
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
22021
ERROR: invalid byte sequence for encoding "UTF8": 0xeef1e5
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by "client_encoding".
This error is shown specifically when I am trying to execute the 2nd query. When I run only the 1st query, everything runs as it is supposed to. Both the tables are present in the same database. Why is it that the encoding error is shown while executing the 2nd query.
Related
I want to print a message in case the user made a mistake while writing the code but it doesnt work I also tried to add NameError exception, it only works if I raise an exception.Thank you for helping.
`
def cncours(nvcours,num_cours):
try :
sql="Update cours set nomC=%s where num_cours=%s"
result=cursor.fetchone()
cursor.execute(sql,(nvcours,num_cours))
print("Operation Done.")
except TypeError:
print("Plz put the name between quotes")
`
Each DB implementation (MySQL, sqlite,...) may raise their particular exceptions. So instead of catching a general exceptions, you may catch errors depending on the specific DB, then on the particular type (e.g. SyntaxError). I suggest to provoque a syntax error on your SQL statement then see what type is (e.g. errorcode or exception type) then catch it.
For instance, the MySQL connector raises error numbers:
import mysql.connector as cn
try:
#...
except cn.Error as err:
print("Something went wrong: {}".format(err))
if err.errno == errorcode.ER_BAD_TABLE_ERROR: #
Here are some MySQL Error Code Ranges
If you are using MySQLdb:
import MySQLdb
try:
#...
cursor.execute(sql)
res = cursor.fetchone()
# ...
except MySQLdb.Error, e:
print "MySQLdb.Error: {}".format(e.args)
Depending on your schema (column types) and the type from the input variable, you may use:
sql="Update cours set nomC='%s' where num_cours=%s" # Added quotes on the first replacement
Besides what you are asking, I think that the command order is inverted.
sql="Update cours set nomC=%s where num_cours=%s"
cursor.execute(sql,(nvcours,num_cours)) # First
result=cursor.fetchone() # Second
print("Operation Done.")
https://www.tutorialspoint.com/python/python_database_access.htm
# execute SQL query using execute() method.
cursor.execute("SELECT VERSION()")
# Fetch a single row using fetchone() method.
data = cursor.fetchone()
I wrote this code to retrieve data from a cube:
rs = win32com.client.Dispatch('ADODB.RecordSet')
rs.__init__
conn = win32com.client.Dispatch('ADODB.Connection')
conn.__init__
conn.CommandTimeout=3000
conn.Open(connString)
#Retrieving data
try:
rs.Open(mdxQuery, conn, 1, 3)
except Exception as e:
print("An error occured while trying to retrieve data:")
print(str(e))
sys.exit(1)
try:
t=rs.GetRows()
except Exception as ex:
print(str(ex))
it seems the mdx query is ok, and retrieve some result (I can see it from debug mode)
but when I try to assign the result of GetRows() function to a variable, it fails, with the following message:
ADODB.Field : Either BOF or EOF is True, or the current record has been deleted. Requested operation requires a current record.
It seems to me GetRows() function is not able to handle NULL values. Is there a way to tell that method to skip NULL values or just fill it with blanks, and avoid this error?
Thanks
I am trying to test the following with python but I get the invalid syntax error:
db = None
try:
db = mdb.connect("localhost","user","pass","dbName")
with db:
cur = db.cursor()
cur.execute("SELECT * from product")
rows = cur.fetchall()
for row in rows:
print row
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
The error is the following:
File "script.py", line 11
with db:
^
SyntaxError: invalid syntax
How do I fix this?
I'd be nice to see all your imports declarations too to see what could be missing from your code. I assume that you at least have the import MySQLdb somewhere. I made a dummy database with a couple of data rows to test this out. Not sure how python let you get away with the indentation blocks being all messed up, but maybe it's just your code indentation posting error on here.
This is the code I tried and seemed to have no issues:
import MySQLdb as mdb
db = None
try:
db = mdb.connect("localhost","user", "password", "test_data")
## with db: ## try taking this out for Python 2.4
cur = db.cursor()
cur.execute("SELECT * FROM PRODUCT")
rows = cur.fetchall()
for row in rows:
print row
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
Suppose, I have a modifying statement:
cursor = conn.cursor()
# some code
affected_rows1 = cursor.execute(update_statement1, params1)
# some code
conn.commit()
cursor.close()
Should I wrap the block of code with a try ... except and explicitly rollback a transaction when an exception is raised, and which MySQLdb exceptions should I catch to rollback? I used to catch any StandardError in this case, but now I have a hesitation that the block of code would even need an explicit rollback at all.
The following example is slightly more difficult, and I understand that it does require an explicit rollback if the first update statement succeeded. Still, which exceptions should I catch in this case:
cursor = conn.cursor()
# some code
affected_rows1 = cursor.execute(update_statement1, params1)
# some code
affected_rows2 = cursor.execute(update_statement2, params2)
#some code
conn.commit()
cursor.close()
This link shows the various types of Errors that you can catch. MySQLdb.Error is the standard base class from which all other MySQL Errors are derived.
I usually use MySQLdb.Error because it lets you focus on errors relating to MySQLdb itself. By contrast StandardError will catch almost all the exceptions (not something you want if you want better debugging capability). Plus the use of MySQLdb.Error allows you to display the exact error message (MySQL error number and all) so that you can debug it faster.
Coming to the first part of the question, in case of database statements it is (usually) necessary to rollback transactions (if they are supported) in case of error.
The methodology that I follow is to wrap each execute statement in a try except clause (catching MySQLdb.Error) and using rollback if there is an an error before printing the error message and exiting.
However, there is a catch. In MySQLdb the changes that you make to DB are not actually written to the database until you explicilty call commit. So, logically, rollback is not necessary.
As an example,
conn = MySQLdb.connection(db=, host=, passwd=, user=)
cur = conn.cursor()
#Say you have a table X with one entry id = 1 and total = 50
cur.execute("update X set total = 70 where id = 1")
#Actual DB has not yet changed
cur.execute("update X set total = 80 where id = 1")
#Actual DB has still not changed
If you exit the program without commiting, the value in DB will still be 50 because you never called commit().
This is how you would ideally do it:
conn = MySQLdb.connection(db=, host=, passwd=, user=)
cur = conn.cursor()
#Say you have a table X with one entry id = 1 and total = 50
try:
cur.execute("update X set total = 70 where id = 1")
except MySQLdb.Error,e:
print e[0], e[1]
conn.rollback()
cur.close()
conn.close()
#print lengthy error description!!
sys.exit(2)
#Note: Value in table is still 50
#If you do conn.commit() here, value becomes 70 in table too!!
try:
cur.execute("update X set total = 80 where id = 1")
except MySQLdb.Error,e:
print e[0], e[1]
conn.rollback()
cur.close()
conn.close()
#print lengthy error description!!
sys.exit(2)
#Value in DB will be
#a) 50 if you didn't commit anywhere
#b) 70 if you committed after first execute statement
conn.commit()
#Now value in DB is 80!!
cur.close()
conn.close()
IMHO, you should rollback transactions if you continue to use the same connection. Else everything before the error will get commit when you finish the transactions.
For the exception to catch, I always use MySQLdb.Error but I'm not sure that's correct.
Its advised to wrap execute() in a sub. This is how i do it.
def executeSQL(self, stmt):
cursor = self.dbHand.cursor()
if not stmt.endswith(";"):
stmt += ';'
try:
cursor.execute(stmt)
except MySQLdb.Error, e:
self.logger.error("Caught MYSQL exception :%s: while executing stmt :%s:.\n"%(e,stmt))
return False
I am trying to import a large text file into a MySQL database. The SQL statement is as follows:
LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre
CHARACTER SET UTF8 FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES (export_date, storefront_id, genre_id, album_id, album_rank)
The above works when I run it in phpMyAdmin, however when I write a simple function in Python that uses the above SQL statement I get an error.
Here is the Python code,
def test():
dbConnection = MySQLdb.connect(
charset='utf8',
host='localhost',
user='root',
passwd='root',
db='epf')
cursor = dbConnection.cursor()
exStr = """LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre CHARACTER SET UTF8
FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES
(export_date, storefront_id, genre_id, album_id, album_rank)"""
try:
cursor.execute(exStr)
except MySQLdb.Warning, e:
print "Warning %s" % (str(e))
except MySQLdb.IntegrityError, e:
print "Error %d: %s" % (e.args[0], e.args[1])
#Clean up
cursor.close()
dbConnection.close()
The error I get is as follows,
Warning Data truncated for column 'album_rank' at row 1
My question now is, why does the raw SQL statement work but when I try to run the Python code, no data is imported into the database?
The Python DBAPI is implicitly transactional. Try adding dbConnection.commit() after the execute.