I've seen some similar questions about this on StackOverflow but haven't found an answer that works; see http://stackoverflow.com/questions/4408714/execute-sql-file-with-python-mysqldb AND http://stackoverflow.com/questions/10593876/execute-sql-file-in-python-with-mysqldb?lq=1
Here is my code:
import pymysql
import sys
import access # holds credentials
import mysql_connector # connects to MySQL, is fully functional
class CreateDB(object):
def __init__(self):
self.cursor = None
self.conn = pymysql.connect(host, user, passwd)
def create_database(self):
try:
with self.conn.cursor() as cursor:
for line in open('file.sql'):
cursor.execute(line)
self.conn.commit()
except Warning as warn:
f = open(access.Credentials().error_log, 'a')
f.write('Warning: %s ' % warn + '\nStop.\n')
sys.exit()
create = CreateDB()
create.create_database()
When I run my script I get the following error:
pymysql.err.InternalError: (1065, 'Query was empty')
My .sql file is successfully loaded when I import directly through MySQL and there is a single query on each line of the file. Does anybody have a solution for this? I have followed the suggestions on other posts but have not had any success.
Take care of empty lines in the end of the file by:
if line.strip(): cursor.execute(line)
You can execute all the SQL in the file at once, by using the official MySQL Connector/Python and the Multi parameter in its cursor.execute method.
Quote from the second link:
If multi is set to True, execute() is able to execute multiple statements specified in the operation string. It returns an iterator that enables processing the result of each statement.
Example code from the link, slightly modified:
import mysql.connector
file = open('script.sql')
sql = file.read()
cnx = mysql.connector.connect(user='u', password='p', host='h', database='d')
cursor = cnx.cursor()
for result in cursor.execute(sql, multi=True):
if result.with_rows:
print("Rows produced by statement '{}':".format(
result.statement))
print(result.fetchall())
else:
print("Number of rows affected by statement '{}': {}".format(
result.statement, result.rowcount))
cnx.close()
Related
To start off, here is an example of what I mean by the 'completion status'. This is a query run via MySQL's CLI
mysql> USE DATABASE1;
Database changed
The line,Database Changed is what I am after. The mysql.connector lib for Python only returns the result of queries that return tables of data from what I've seen so far (I will preface this by saying that I am a beginner to SQL in general, so admittedly I have not tried everything out there).
I am trying to create a MySQL gui-based front end as part of a project, so I am trying to replicate the MySQL CLI 'experience',so to speak, to as high a degree as possible. So stuff like getting out such output text is imperative
For instance,
import mysql.connector
mydb = mysql.connector.connect(host = 'localhost', user = 'root', password = key)
mycursor = mydb.cursor()
mycursor.execute('USE DATABASE1')
for i in mycursor:
print(i)
Returns... nothing (Understandable, iterating on the cursor would only provide table data after all). I want to find a way to extract Database Changed through mysql.connector. I've scoured through the docs but can't find any method that would yield me such a thing. Is there a way to eke it out?
You could do (I did after reading this example)
When you only want to print database changes when they happen, you can do something like this:
import mysql.connector
from mysql.connector import errorcode
_current_database = ''
cnx = mysql.connector.connect(host = 'localhost',
user = 'test', password = 'test', auth_plugin='mysql_native_password')
def Execute_MySQL_and_print_result(sqlstatement):
global _current_database
global cnx
try:
mycursor = cnx.cursor(buffered=True)
mycursor.execute(sqlstatement)
if (cnx.database != _current_database):
_current_database = cnx.database
print("Database changed to: ", cnx.database)
j=1
for i in mycursor:
print(j, i)
j=j+1
except mysql.connector.Error as err:
print("OOPS, something went wrong: ", err.errno)
print(err.msg)
Execute_MySQL_and_print_result('USE TEST')
Execute_MySQL_and_print_result('SHOW DATABASES')
Execute_MySQL_and_print_result('USE TEST')
Execute_MySQL_and_print_result('SHOW DATABASES')
Execute_MySQL_and_print_result('SELECT * FROM nonexistingTable')
output:
Database changed to: test
1 ('information_schema',)
2 ('sakila',)
3 ('test',)
4 ('world',)
1 ('information_schema',)
2 ('sakila',)
3 ('test',)
4 ('world',)
OOPS, something went wrong: 1146
Table 'test.nonexistingtable' doesn't exist
I added the numbers before the database names to make it clear when the second SHOW DATABASE is starting output.
Note: The second USE TEST has no output, because active database did not change.
I am getting
File "getweather.py", line 15 SyntaxError: unexpected EOF while
parsing
I retyped it into a new completely new file. But am still seeing the same issue. Can anyone see what could be causing it?
from urlib.request import urlopen
import json
import psycopg2
import psycopg2.extras
try:
db = psycopg2.connect("dbname=myapp user=postgres password=postgres")
cursor = db.cursor()
query = "select city from myapp"
cursor.execute(query)
data = cursor.fetchall()
for row in data:
print (row)
db.close()
You can't have a try clause without an accompanying except; that's the cause of your error.
See the docs on Handling Exceptions.
Writing a script to convert raw data for MySQL import I worked with a temporary textfile so far which I later imported manually using the LOAD DATA INFILE... command.
Now I included the import command into the python script:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
cursor = db.cursor()
query = """
LOAD DATA INFILE 'temp.txt' INTO TABLE myDB.values
FIELDS TERMINATED BY ',' LINES TERMINATED BY ';';
"""
cursor.execute(query)
cursor.close()
db.commit()
db.close()
This works but temp.txt has to be in the database directory which isn't suitable for my needs.
Next approch is dumping the file and commiting directly:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
sql = "INSERT INTO values(`timestamp`,`id`,`value`,`status`) VALUES(%s,%s,%s,%s)"
cursor=db.cursor()
for line in lines:
mode, year, julian, time, *values = line.split(",")
del values[5]
date = datetime.strptime(year+julian, "%Y%j").strftime("%Y-%m-%d")
time = datetime.strptime(time.rjust(4, "0"), "%H%M" ).strftime("%H:%M:%S")
timestamp = "%s %s" % (date, time)
for i, value in enumerate(values[:20], 1):
args = (timestamp,str(i+28),value, mode)
cursor.execute(sql,args)
db.commit()
Works as well but takes around four times as long which is too much. (The same for construct was used in the first version to generate temp.txt)
My conclusion is that I need a file and the LOAD DATA INFILE command to be faster. To be free where the textfile is placed the LOCAL option seems useful. But with MySQL Connector (1.1.7) there is the known error:
mysql.connector.errors.ProgrammingError: 1148 (42000): The used command is not allowed with this MySQL version
So far I've seen that using MySQLdb instead of MySQL Connector can be a workaround. Activity on MySQLdb however seems low and Python 3.3 support will probably never come.
Is LOAD DATA LOCAL INFILE the way to go and if so is there a working connector for python 3.3 available?
EDIT: After development the database will run on a server, script on a client.
I may have missed something important, but can't you just specify the full filename in the first chunk of code?
LOAD DATA INFILE '/full/path/to/temp.txt'
Note the path must be a path on the server.
To use LOAD DATA INFILE with every accessible file you have to set the
LOCAL_FILES client flag while creating the connection
import mysql.connector
from mysql.connector.constants import ClientFlag
db = mysql.connector.connect(client_flags=[ClientFlag.LOCAL_FILES], <other arguments>)
I wanted a script that iterates through csv files in a folder and dump them into a MySQL database. I was able to dump one csv file into it.. But have troubles passing the file name in to the SQL script.
This is the code I use
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
cursor.execute(" LOAD DATA LOCAL INFILE '%s' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES ",(n))
And I get the following error
raise errorclass, errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'file1.csv'' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY' at line 1")
If I use the file name directly instead of passing it, it works fine.
If you can see in the error thrown out, there seems to be an error in the SQL Script.
This would be the whole code
import csv
import MySQLdb
import sys
import os
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
print n
cursor.execute(" LOAD DATA LOCAL INFILE %s INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES " %n)
connection.commit()
cursor.close()
First, replace '%s' with %s in the query. MySQLdb handles any quoting automatically.
Here's the code with some corrections and changes:
import MySQLdb
import os
CSV_DIR = "C:\csv-files"
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db',
local_infile=1)
cursor = connection.cursor()
try:
for filename in sorted(os.listdir(CSV_DIR)):
cursor.execute("""LOAD DATA LOCAL INFILE %s
INTO TABLE new_table
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES""",
(os.path.join(CSV_DIR, filename),))
connection.commit()
finally:
cursor.close()
NOTE: I set local_infile parameter to 1 in MySQLdb.connect and pass filename in tuple to execute.
Works for me.
I am currently connecting to a Sybase 15.7 server using sybpydb. It seems to connect fine:
import sys
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib')
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/lib')
import sybpydb
conn = sybpydb.connect(user='usr', password='pass', servername='serv')
is working fine. Changing any of my connection details results in a connection error.
I then select a database:
curr = conn.cursor()
curr.execute('use db_1')
however, now when I try to run queries, it always returns None
print curr.execute('select * from table_1')
I have tried running the use and select queries in the same execute, I have tried including go commands after each, I have tried using curr.connection.commit() after each, all with no success. I have confirmed, using dbartisan and isql, that the same queries I am using return entries.
Why am I not getting results from my queries in python?
EDIT:
Just some additional info. In order to get the sybpydb import to work, I had to change two environment variables. I added the lib paths (the same ones that I added to sys.path) to $LD_LIBRARY_PATH, i.e.:
setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib:/dba/sybase/ase/15.7/OCS-15_0/lib
and I had to change the SYBASE path from 12.5 to 15.7. All this was done in csh.
If I print conn.error(), after every curr.execute(), I get:
("Server message: number(5701) severity(10) state(2) line(0)\n\tChanged database context to 'master'.\n\n", 5701)
I completely understand where you might be confused by the documentation. Its doesn't seem to be on par with other db extensions (e.g. psycopg2).
When connecting with most standard db extensions you can specify a database. Then, when you want to get the data back from a SELECT query, you either use fetch (an ok way to do it) or the iterator (the more pythonic way to do it).
import sybpydb as sybase
conn = sybase.connect(user='usr', password='pass', servername='serv')
cur = conn.cursor()
cur.execute("use db_1")
cur.execute("SELECT * FROM table_1")
print "Query Returned %d row(s)" % cur.rowcount
for row in cur:
print row
# Alternate less-pythonic way to read query results
# for row in cur.fetchall():
# print row
Give that a try and let us know if it works.
Python 3.x working solution:
import sybpydb
try:
conn = sybpydb.connect(dsn="Servername=serv;Username=usr;Password=pass")
cur = conn.cursor()
cur.execute('select * from db_1..table_1')
# table header
header = tuple(col[0] for col in cur.description)
print('\t'.join(header))
print('-' * 60)
res = cur.fetchall()
for row in res:
line = '\t'.join(str(col) for col in row)
print(line)
cur.close()
conn.close()
except sybpydb.Error:
for err in cur.connection.messages:
print(f'Error {err[0]}, Value {err[1]}')