Develop Reference

Fail to run 'execute(sqlcmd)' of sqlite3 using Python,Ubuntu

I want to use sqlite3 to deal with data in Ubuntu with python. But I always failed and get errors. Codes related to database are as follows:
sqlite = "%s.db" % name
#connnect to the database
conn = sqlite3.connect(sqlite)
print "Opened database successfully"
c = conn.cursor()
#set default separator to "\t" in database
c.execute(".separator "\t"")
print "Set separator of database successfully"
#create table data_node
c.execute('''create table data_node(Time int,Node Text,CurSize int,SizeVar int,VarRate real,Evil int);''')
print "Table data_node created successfully"
node_info = "%s%s.txt" % (name,'-PIT-node')
c.execute(".import %\"s\" data_node") % node_info
print "Import to data_node successfully"
#create table data_face
data_info = "%s%s.txt" % (name,'-PIT-face')
c.execute('''create table data_face(Time int,Node Text,TotalEntry real,FaceId int,FaceEntry real,Evil int);''')
c.execute(".import \"%s\" data_face") % face_info
#get the final table : PIT_node
c.execute('''create table node_temp as select FIRST.Time,FIRST.Node,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio from tracer_temp FIRST,tracer_temp SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.Type='InData' AND SECOND.Type='OutInterests';''')
c.execute('''create table PIT_node as select A.Time,A.Node,B.SatisRatio,A.CurSize,A.SizeVar,A.VarRate,A.Evil from data_node A,node_temp B WHERE A.Time=B.Time AND A.Node=B.Node;''')
#get the final table : PIT_face
c.execute('''create table face_temp as select FIRST.Time,FIRST.Node,FIRST.FaceId,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio,SECOND.Packets from data_tracer FIRST,data_tracer SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.FaceId=SECOND.FaceId AND FIRST.Type='OutData' AND SECOND.Type='InInterests';''')
c.execute('''create table PIT_face as select A.Time,A.Node,A.FaceId,B.SatisRatio,B.Packets,ROUND(A.FaceEntry/A.TotalEntry,4),A.Evil from data_face as A,face_temp as B WHERE A.Time=B.Time AND A.Node=B.Node AND A.FaceId = B.FaceId;''')
conn.commit()
conn.close()
These sql-commands are right. When I run the code, it always shows sqlite3.OperationalError: near ".": syntax error. So how to change my code and are there other errors in other commands such as create table?

You have many problems in your code as posted, but the one you're asking about is:
c.execute(".separator "\t"")
This isn't valid Python syntax. But, even if you fix that, it's not valid SQL.
The "dot-commands" are special commands to the sqlite3 command line shell. It intercepts them and uses them to configure itself. They mean nothing to the actual database, and cannot be used from Python.
And most of them don't make any sense outside that shell anyway. For example, you're trying to set the column separator here. But the database doesn't return strings, it returns row objects—similar to lists. There is nowhere for a separator to be used. If you want to print the rows out with tab separators, you have to do that in your own print statements.
So, the simple fix is to remove all of those dot-commands.
However, there is a problem—at least one of those dot-commands actually does something:
c.execute(".import %\"s\" data_node") % node_info
You will have to replace that will valid calls to the library that do the same thing as the .import dot-command. Read what it does, and it should be easy to understand. (You basically want to open the file, parse the columns for each row, and do an executemany on an INSERT with the rows.)

Fast MySQL Import

Writing a script to convert raw data for MySQL import I worked with a temporary textfile so far which I later imported manually using the LOAD DATA INFILE... command.
Now I included the import command into the python script:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
cursor = db.cursor()
query = """
LOAD DATA INFILE 'temp.txt' INTO TABLE myDB.values
FIELDS TERMINATED BY ',' LINES TERMINATED BY ';';
"""
cursor.execute(query)
cursor.close()
db.commit()
db.close()
This works but temp.txt has to be in the database directory which isn't suitable for my needs.
Next approch is dumping the file and commiting directly:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
sql = "INSERT INTO values(`timestamp`,`id`,`value`,`status`) VALUES(%s,%s,%s,%s)"
cursor=db.cursor()
for line in lines:
mode, year, julian, time, *values = line.split(",")
del values[5]
date = datetime.strptime(year+julian, "%Y%j").strftime("%Y-%m-%d")
time = datetime.strptime(time.rjust(4, "0"), "%H%M" ).strftime("%H:%M:%S")
timestamp = "%s %s" % (date, time)
for i, value in enumerate(values[:20], 1):
args = (timestamp,str(i+28),value, mode)
cursor.execute(sql,args)
db.commit()
Works as well but takes around four times as long which is too much. (The same for construct was used in the first version to generate temp.txt)
My conclusion is that I need a file and the LOAD DATA INFILE command to be faster. To be free where the textfile is placed the LOCAL option seems useful. But with MySQL Connector (1.1.7) there is the known error:
mysql.connector.errors.ProgrammingError: 1148 (42000): The used command is not allowed with this MySQL version
So far I've seen that using MySQLdb instead of MySQL Connector can be a workaround. Activity on MySQLdb however seems low and Python 3.3 support will probably never come.
Is LOAD DATA LOCAL INFILE the way to go and if so is there a working connector for python 3.3 available?
EDIT: After development the database will run on a server, script on a client.

I may have missed something important, but can't you just specify the full filename in the first chunk of code?
LOAD DATA INFILE '/full/path/to/temp.txt'
Note the path must be a path on the server.

To use LOAD DATA INFILE with every accessible file you have to set the
LOCAL_FILES client flag while creating the connection
import mysql.connector
from mysql.connector.constants import ClientFlag
db = mysql.connector.connect(client_flags=[ClientFlag.LOCAL_FILES], <other arguments>)

Dumping CSV files to MySQL database using embedded query in python script

I have a SQL query which opens up a csv file and dumps it in a table of a database. I am trying to dump multiple files at once using a python script to iterate among the files. I tried embedding the same SQL query inside the script, but it throws out an error.
This is the script I have.
import csv
import MySQLdb
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
query = """ LOAD DATA INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
cursor.execute(query)
conenction.commit()
cursor.close()
And for some reason the python script looks up example.csv at a different location
This is the error that is thrown out :
raise errorclass, errorvalue
InternalError: (29, "File 'C:\\Documents and Settings\\All Users\\Application Data\\MySQL\\MySQL Server 5.5\\data\\example.csv' not found (Errcode: 2)")
Any help would be greatly appreciated. I am also searching on stackoverflow for help to dump the scv files into differnt tables of a database. Any ideas on that?

You probably need the load data local syntax to make sure the data is being read relative to the client and not the server. Change
query = """ LOAD DATA INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
to
query = """ LOAD DATA LOCAL INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """

Watch your spelling! conenction.commit() should be connection.commit()

Problem with Python and MySQL

I am trying to import a large text file into a MySQL database. The SQL statement is as follows:
LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre
CHARACTER SET UTF8 FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES (export_date, storefront_id, genre_id, album_id, album_rank)
The above works when I run it in phpMyAdmin, however when I write a simple function in Python that uses the above SQL statement I get an error.
Here is the Python code,
def test():
dbConnection = MySQLdb.connect(
charset='utf8',
host='localhost',
user='root',
passwd='root',
db='epf')
cursor = dbConnection.cursor()
exStr = """LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre CHARACTER SET UTF8
FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES
(export_date, storefront_id, genre_id, album_id, album_rank)"""
try:
cursor.execute(exStr)
except MySQLdb.Warning, e:
print "Warning %s" % (str(e))
except MySQLdb.IntegrityError, e:
print "Error %d: %s" % (e.args[0], e.args[1])
#Clean up
cursor.close()
dbConnection.close()
The error I get is as follows,
Warning Data truncated for column 'album_rank' at row 1
My question now is, why does the raw SQL statement work but when I try to run the Python code, no data is imported into the database?

The Python DBAPI is implicitly transactional. Try adding dbConnection.commit() after the execute.

Error message in python-mysql cursor: 1054 unknown column "x" in 'field list'

This is my first post! I also just started programming, so please bear with me!
I am trying to load a bunch of .csv files into a database, in order to later perform various reports on the data. I started off by creating a few tables in mysql with matching field names and data types to what will be loaded into the tables. I am manipulating the filename (in order to parse out the date to use as a field in my table) and cleaning up the data with python.
So my problem right now (haha...) is that I get this error message when I attempt the 'Insert Into' query to mysql.
Traceback (most recent call last):
File "C:\Program Files\Python\load_domains2.py", line 80, in <module>
cur.execute(sql)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1054, "Unknown column 'a1200e.com' in 'field list'")
'a1200e.com' refers to a specific domain name I'm inserting into that column. My query is as follows:
sql="""INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)""" %(cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql)
I am very new at all this, so I'm sure my code isn't at all efficient, but I just wanted to lay everything out so it's clear to me. What I don't understand is that I have ensured my table has correctly defined data types (corresponding to those in my query). Is there something I'm missing? I've been trying to work this out for a while, and don't know what could be wrong :/
Thanks so much!!!
Val

Thomas is, as usual, absolutely correct: feel free to let MySQLdb handle the quoting issues.
In addition to that recommendation:
The csv module is your friend.
MySQLdb uses the "format" parameter style as detailed in PEP 249.
What does that mean for you?
All parameters, whatever type, should be passed to MySQLdb as strings (like this %s). MySQLdb will make sure that the values are properly converted to SQL literals.
By the way, MySQLdb has some good documentation.
Feel free to include more detail about your source data. That may make diagnosing the problem easier.
Here's one way to insert values to a MySQL database from a .csv file:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import csv
import MySQLdb
import os
def main():
db = MySQLdb.connect(db="mydb",passwd="mypasswd",) # connection string
filename = 'data.csv'
f = open(filename, "rb") # open your csv file
reader = csv.reader(f)
# assuming the first line of your csv file has column names
col_names = reader.next() # first line of .csv file
reader = csv.DictReader(f, col_names) # apply column names to row values
to_db = [] # this list holds values you will insert to db
for row in reader: # loop over remaining lines in .csv file
to_db.append((row['col1'],row['col2']))
# or if you prefer one-liners
#to_db = [(row['col1'],row['col2']) for row in reader]
f.close() # we're done with the file now
cursor = db.cursor()
cursor.executemany('''INSERT INTO mytable (col1,col2)
VALUES (%s, %s)''', to_db) # note the two arguments
cursor.close()
db.close()
if __name__ == "__main__":
main()

You should be using DB-API quoting instead of including the data in the SQL query directly:
sql = """INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)"""
args = (cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql, args)
This makes the DB-API module quote the values appropriately, and resolves a whole host of issues that you might get when doing it by hand (and usually incorrectly.)