Develop Reference

how to populate database using python faker

I am trying to populate a table using python faker and I am getting this error . Here is my code
import psycopg2
from faker import Faker
fake = Faker()
conn = psycopg2.connect(database="testdb", user="****", password="****", host="127.0.0.1", port="5432")
print "Opened database successfully"
cur = conn.cursor()
for i in range (10):
Id =fake.random_digit_not_null()
name = fake.name()
age=fake.random_number(digits=None)
adress =fake.address()
salary = fake.random_int(min=0, max=9999)
cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
VALUES (Id,name,age,adress,salary)");
conn.commit()
print "Records created successfully";
conn.close()
here is the error
Traceback (most recent call last):
File "fakegenerator.py", line 16, in <module>
VALUES (Id,name,age,adress,salary)");
psycopg2.ProgrammingError: column "id" does not exist
LINE 1: ...OMPANY (ID,NAME,AGE,ADDRESS,SALARY) VALUES (Id,name,ag...
^
HINT: There is a column named "id" in table "company", but it cannot be referenced from this part of the query.

You're not filling in the values into your query, instead you're sending the string as-is to the database. This would actually fill your query with values:
cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) VALUES (%s, %s, %s, %s, %s)", (Id, name, age, adress, salary));
This wraps the variables filled with the values you want to insert into a tuple and let's psycopg2 handle quoting your strings correctly which is less work for you and keeps you safe from SQL injection, should you use your code as a base for productive code. This is documented in the module's documentation.

the sql request in cur.execute seems to be a problem try this
cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
VALUES ({},{},{},{},{})".format(Id,name,age,adress,salary));

Testing Python to MySQL error

I have created a little script that allows me to save data to MySQLdb. At first it was working fine when I was using:
cursor.execute('INSERT INTO people (name, text) VALUES ("dan", "test2")')
The above would save "dan" into the title and "test2" into the text. I wanted to test to see if I was able to define something and fill it in this way. For example if I was to scrape a site and say (dan = soup.title.string) or something like that it would be able to populate this data into the database. I have tried to have a look around but cannot seem to find anything.
import MySQLdb
import sys
try:
db = MySQLdb.connect(
host = 'localhost',
user = 'root',
passwd = '',
db = 'python',
)
except:
print "db not found"
dan = "dandandan"
test2 = "testing101"
cursor = db.cursor()
cursor.execute('INSERT INTO people (name, text) VALUES (dan, test2)')
cursor.execute('SELECT * FROM people')
result = cursor.fetchall()
db.commit()
db.close()
The error I am receiving is:
C:\Users\********\Desktop>python mysqltest.py
Traceback (most recent call last):
File "mysqltest.py", line 18, in <module>
cursor.execute('INSERT INTO people (name) VALUES (dan)')
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defau
lterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1054, "Champ 'dan' inconnu dans field list"
)

You need to use parameters.
cursor.execute('INSERT INTO people (name, text) VALUES (%s,%s)', (dan, test2))

Use prepared statements:
cursor.execute("INSERT INTO people (name, text) VALUES (%s,%s)", (dan, test2))
From the documentation :
paramstyle
String constant stating the type of parameter marker formatting
expected by the interface. Set to 'format' = ANSI C printf format
codes, e.g. '...WHERE name=%s'. If a mapping object is used for
conn.execute(), then the interface actually uses 'pyformat' = Python
extended format codes, e.g. '...WHERE name=%(name)s'. However, the API
does not presently allow the specification of more than one style in
paramstyle.
Note that any literal percent signs in the query string passed to
execute() must be escaped, i.e. %%.
Parameter placeholders can only be used to insert column values. They
can not be used for other parts of SQL, such as table names,
statements, etc.

How to insert a table consist of several dictionaries into MySQL tables?

I have the below result from my python code:
[
{filename:'1,2',Name:'Gorge',registration number: '6657', registration date: '2012-09-10 14:31:13'},
{filename:'5,43',Name:'mazu',registration number:'45', registration date:'2012-10-08 17:28:47'}]
and as soon as I want to put it in a MySQL table, I got this error:
Traceback (most recent call last):
File "C:\Users\jio\Datasets 1\MyTable_info.py", line 63, in <module>
cur.executemany(query,records)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 243, in executemany
self.errorhandler(self, ProgrammingError, msg.args[0])
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: not enough arguments for format string
My python code to insert the result in MySQL table is the below code:
con = MySQLdb.connect(host = "******", port=***, user = "***", passwd="*****", db="****")
with con:
cur=con.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS info(id INT(10) auto_increment primary key,file_name VARCHAR(10),
Name VARCHAR(50),Registration ID INT(50),registration time INT(50))''')
query= "INSERT INTO info (file_name, Name, Registration ID, registration time) VALUES ( %s, %s, %s, %s )"
cur.executemany(query,records)
con.commit()
Does anyone has an idea why I get this error and what does the error mean?

You are trying to insert a String into a field INT(50).
Take a look in the last field of the table, registration time field. It is an Integer and you are trying to insert values like '2012-10-08 17:28:47' or '2012-09-10 14:31:13'.
For a quick fix just change registration time field type as a VARCHAR(50).
But maybe, for perfomance issues, you should think to use some kind of TIMESTAMP field instead of a VARCHAR for this kind of purposes.
Apart from don't use INT type for field where you would like to add some kind of String.
Modify the records variable from
[ {filename:'1,2',Name:'Gorge',registration number: '6657', registration date: '2012-09-10 14:31:13'},
{filename:'5,43',Name:'mazu',registration number:'45', registration date:'2012-10-08 17:28:47'}]
to
[('1,2','Gorge','6657', '2012-09-10 14:31:13'),('5,43','mazu','45','2012-10-08 17:28:47')]
And avoid use whitespaces for the columns names as well

Try to quote field names. What the contents of records?

Don't use spaces in field names.
If you are required to, quote them with the ` character.

python inserting single quotes (') around MySQL table name

I have a database called project1 with the following tables:
_systbl1
_systbl2
_systbl3
dataset1
dataset2
dataset3
MySQL user odbc will need to be granted SELECT permissions on dataset% tables whenever a new one is added.
To accomplish this, I'm using a simple python script, like so:
#!/usr/bin/python
import MySQLdb
db = MySQLdb.connect(
host="localhost",
user="user",
passwd="pass",
db="project1"
)
# Create Cursor Object
cur = db.cursor()
# get list of tables beginning with dataset
cur.execute("SHOW TABLES FROM project1 LIKE 'dataset%';")
# run GRANT statement for each table
for row in cur.fetchall() :
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;", row)
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`%`;", row)
Unfortunately, it gives me the following error:
Traceback (most recent call last):
File "mysql_query.py", line 20, in <module>
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;", row)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1146, "Table 'project1.'dataset1'' doesn't exist")
As you can see in the last line of the error, the problem is that python is putting a single quote around the table names when generating the query.
What am I missing here?

Do not use SQL parameters for table names. SQL parameters are escaped by the database adapter to not be interpreted as anything but literal values.
You'll have to interpolate those yourself instead, but be absolutely certain that your table name does not hold untrusted data (prevent SQL injection attacks):
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;" % row)
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`%%`;" % row)
(where the % character in the grant has been escaped by doubling it to %%).

Instead use:
cur.execute("GRANT SELECT ON `project`.`%s` TO `odbc`#`localhost`" % row)
This will not use the normal escaping of the input. Beware of a backtick in any of your table names, though.

sql = """CREATE TABLE IF NOT EXISTS `""" + project + """` ( `id` INT(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`))"""

Python MySQL Statement returning Error

hey, I'm very new to all this so please excuse stupidity :)
import os
import MySQLdb
import time
db = MySQLdb.connect(host="localhost", user="root", passwd="********", db="workspace")
cursor = db.cursor()
tailoutputfile = os.popen('tail -f syslog.log')
while 1:
x = tailoutputfile.readline()
if len(x)==0:
break
y = x.split()
if y[2] == 'BAD':
timestring = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(time.time()))
cursor.execute("INSERT INTO releases (date, cat, name) values (timestring, y[4], y[7]")
if y[2] == 'GOOD':
print y[4] + '\t' + y[7]
so i run the program and this is the error message I am getting
user#machine:~/$ python reader.py
Traceback (most recent call last):
File "reader.py", line 17, in ?
cursor.execute("INSERT INTO releases (date, cat, name) values (timestring, y[4], y[7]")
File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 163, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.4/site-packages/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[4], y[7]' at line 1")
user#machine:~/$
So i'm assuming that the error is obviously coming from the SQL Statement
cursor.execute("INSERT INTO releases (date, cat, name) values (timestring, y[4], y[7]")
Here is an example of what y[4] and y[7] will look like.
YES Mail.Sent.To.User:user#work.com.11.2.2008:23.17
Is this error happening because I should be escaping those values before I try and Insert them into the Database?
Or am I completely missing the point??
Any help would be appreciated!
thanks in advance.

As pointed out, you're failing to copy the Python variable values into the query, only their names, which mean nothing to MySQL.
However the direct string concatenation option:
cursor.execute("INSERT INTO releases (date, cat, name) VALUES ('%s', '%s', '%s')" % (timestring, y[4], y[7]))
is dangerous and should never be used. If those strings have out-of-bounds characters like ' or \ in, you've got an SQL injection leading to possible security compromise. Maybe in your particular app that can never happen, but it's still a very bad practice, which beginners' SQL tutorials really need to stop using.
The solution using MySQLdb is to let the DBAPI layer take care of inserting and escaping parameter values into SQL for you, instead of trying to % it yourself:
cursor.execute('INSERT INTO releases (date, cat, name) VALUES (%s, %s, %s)', (timestring, y[4], y[7]))

cursor.execute("INSERT INTO releases (date, cat, name) values (timestring, y[4], y[7]")
should be
cursor.execute("INSERT INTO releases (date, cat, name) values (timestring, '%s', '%s')" % (y[4], y[7]))
Your best bet to debug things like this is to put the query into a variable and use that:
query = "INSERT INTO releases (date, cat, name) values (timestring, '%s', '%s')" % (y[4], y[7])
print query
cursor.execute(query)
That print statement would make it very obvious what the problem is.
If you're going to be using list variables a lot like this it can get very confusing, consider using the list just once and putting the variables into a dictionary. It's a bit longer to type, but is much, much easier to keep track of what's going on.

never use "direct string concatenation" with SQL, because it's not secure, more correct variant:
cursor.execute('INSERT INTO releases (date, cat, name) VALUES (%s, %s, %s)', (timestring, y[4], y[7]))
it automatically escaping forbidden symbols in values (such as ", ' etc)