I am working on a learning how to execute SQL in python (I know SQL, not Python).
I have an external sql file. It creates and inserts data into three tables 'Zookeeper', 'Handles', 'Animal'.
Then I have a series of queries to run off the tables. The below queries are in the zookeeper.sql file that I load in at the top of the python script. Example for the first two are:
--1.1
SELECT ANAME,zookeepid
FROM ANIMAL, HANDLES
WHERE AID=ANIMALID;
--1.2
SELECT ZNAME, SUM(TIMETOFEED)
FROM ZOOKEEPER, ANIMAL, HANDLES
WHERE AID=ANIMALID AND ZOOKEEPID=ZID
GROUP BY zookeeper.zname;
These all execute fine in SQL. Now I need to execute them from within Python. I have been given and completed code to read in the file. Then execute all the queries in the loop.
The 1.1 and 1.2 is where I am getting confused. I believe in the loop this is the line where I should put in something to run the first and then second query.
result = c.execute("SELECT * FROM %s;" % table);
but what? I think I am missing something very obvious. I think what is throwing me off is % table. In query 1.1 and 1.2, I am not creating a table, but rather looking for a query result.
My entire python code is below.
import sqlite3
from sqlite3 import OperationalError
conn = sqlite3.connect('csc455_HW3.db')
c = conn.cursor()
# Open and read the file as a single buffer
fd = open('ZooDatabase.sql', 'r')
sqlFile = fd.read()
fd.close()
# all SQL commands (split on ';')
sqlCommands = sqlFile.split(';')
# Execute every command from the input file
for command in sqlCommands:
# This will skip and report errors
# For example, if the tables do not yet exist, this will skip over
# the DROP TABLE commands
try:
c.execute(command)
except OperationalError, msg:
print "Command skipped: ", msg
# For each of the 3 tables, query the database and print the contents
for table in ['ZooKeeper', 'Animal', 'Handles']:
**# Plug in the name of the table into SELECT * query
result = c.execute("SELECT * FROM %s;" % table);**
# Get all rows.
rows = result.fetchall();
# \n represents an end-of-line
print "\n--- TABLE ", table, "\n"
# This will print the name of the columns, padding each name up
# to 22 characters. Note that comma at the end prevents new lines
for desc in result.description:
print desc[0].rjust(22, ' '),
# End the line with column names
print ""
for row in rows:
for value in row:
# Print each value, padding it up with ' ' to 22 characters on the right
print str(value).rjust(22, ' '),
# End the values from the row
print ""
c.close()
conn.close()
Your code already contains a beautiful way to execute all statements from a specified sql file
# Open and read the file as a single buffer
fd = open('ZooDatabase.sql', 'r')
sqlFile = fd.read()
fd.close()
# all SQL commands (split on ';')
sqlCommands = sqlFile.split(';')
# Execute every command from the input file
for command in sqlCommands:
# This will skip and report errors
# For example, if the tables do not yet exist, this will skip over
# the DROP TABLE commands
try:
c.execute(command)
except OperationalError, msg:
print("Command skipped: ", msg)
Wrap this in a function and you can reuse it.
def executeScriptsFromFile(filename):
# Open and read the file as a single buffer
fd = open(filename, 'r')
sqlFile = fd.read()
fd.close()
# all SQL commands (split on ';')
sqlCommands = sqlFile.split(';')
# Execute every command from the input file
for command in sqlCommands:
# This will skip and report errors
# For example, if the tables do not yet exist, this will skip over
# the DROP TABLE commands
try:
c.execute(command)
except OperationalError, msg:
print("Command skipped: ", msg)
To use it
executeScriptsFromFile('zookeeper.sql')
You said you were confused by
result = c.execute("SELECT * FROM %s;" % table);
In Python, you can add stuff to a string by using something called string formatting.
You have a string "Some string with %s" with %s, that's a placeholder for something else. To replace the placeholder, you add % ("what you want to replace it with") after your string
ex:
a = "Hi, my name is %s and I have a %s hat" % ("Azeirah", "cool")
print(a)
>>> Hi, my name is Azeirah and I have a Cool hat
Bit of a childish example, but it should be clear.
Now, what
result = c.execute("SELECT * FROM %s;" % table);
means, is it replaces %s with the value of the table variable.
(created in)
for table in ['ZooKeeper', 'Animal', 'Handles']:
# for loop example
for fruit in ["apple", "pear", "orange"]:
print(fruit)
>>> apple
>>> pear
>>> orange
If you have any additional questions, poke me.
A very simple way to read an external script into an sqlite database in python is using executescript():
import sqlite3
conn = sqlite3.connect('csc455_HW3.db')
with open('ZooDatabase.sql', 'r') as sql_file:
conn.executescript(sql_file.read())
conn.close()
First make sure that a table exists if not, create a table then follow the steps.
import sqlite3
from sqlite3 import OperationalError
conn = sqlite3.connect('Client_DB.db')
c = conn.cursor()
def execute_sqlfile(filename):
c.execute("CREATE TABLE clients_parameters (adress text, ie text)")
#
fd = open(filename, 'r')
sqlFile = fd.readlines()
fd.close()
lvalues = [tuple(v.split(';')) for v in sqlFile[1:] ]
try:
#print(command)
c.executemany("INSERT INTO clients_parameters VALUES (?, ?)", lvalues)
except OperationalError as msg:
print ("Command skipped: ", msg)
execute_sqlfile('clients.sql')
print(c.rowcount)
according me, it is not possible
solution:
import .sql file on mysql server
after
import mysql.connector
import pandas as pd
and then you use .sql file by convert to dataframe
Related
I know there are some other posts out there, but I was not able to find the specific question I had in mind.
I'm using US_baby_names csv file. and want to import this csv file line by line into sqlite3 as a table.
I'm able to create the table called storage.
I'm then trying to read lines in the csv file and put it into that table, but I must be doing something wrong.
import sqlite3 as sql
from sqlite3 import Error
import csv
def CreateConnection ( dbFileName ):
try:
conn = sql.connect(dbFileName)
return conn
except Error as e:
print(e)
return None
def CreateNew( dbConnection, new):
sql = """INSERT INTO storage (dat, Id, Name, Year, group, subgroup, Count)
VALUES (?,?,?,?,?,?,?)"""
try:
cursor = dbConnection.cursor()
cursor.execute(sql, new)
return cursor.lastrowid
except Error as e:
print(e)
def Main():
database = "storage.db"
dbConnection = CreateConnection(database)
with open('storage.csv', 'rb') as fin:
dr = csv.DictReader(fin)
to_db = [(i['dat'], i['Id'], i['Name'], i['Year'], i['group'], i['subgroup'], i['Count']) \
for i in dr]
cursor.executemany(CreateNew(sql, to_db))
dbConnection.close()
if __name__ == "__main__":
Main()
I believe my cursor.executemany is wrong, but I'm not able to figure out what else to do..
Thanks
You are almost right with much of your code, but:
in cursor.execute(sql, new) you are passing an iterable, new, to sqlite3.execute() (which requires a simple SQL statement), instead of sqlite3.executemany().
Moreover, the result of CreateNew() is an integer, lastrowid, and you pass that result to executemany().
You must use Connection.commit() to save the changes to the database, and Connection.rollback() to discard them.
You must open the file for the csv.DictReader class as a text file, in r or rt mode.
Finally, remember that sqlite3.Connection is a context manager, so you can use it in a with statement.
This should be your desired outcome:
import sqlite3 as sql
from sqlite3 import Error
import csv
def create_table(conn):
sql = "CREATE TABLE IF NOT EXISTS baby_names("\
"dat TEXT,"\
"Id INTEGER PRIMARY KEY,"\
"Name TEXT NOT NULL,"\
"Year INTEGER NOT NULL,"\
"Gender TEXT NOT NULL,"\
"State TEXT NOT NULL,"\
"Count INTEGER)"
conn.execute(sql)
conn.execute("DELETE FROM baby_names")
def select_all(conn):
for r in conn.execute("SELECT * FROM baby_names").fetchall():
print(r)
def execute_sql_statement(conn, data):
sql = "INSERT INTO baby_names "\
"(dat, Id, Name, Year, Gender, State, Count) "\
"VALUES (?,?,?,?,?,?,?)"
try:
cursor = conn.executemany(sql, data)
except Error as e:
print(e)
conn.rollback()
return None
else:
conn.commit()
return cursor.lastrowid
def main():
with sql.connect('baby_names.db') as conn, open('US_Baby_Names_right.csv', 'r') as fin:
create_table(conn)
dr = csv.DictReader(fin)
data = [(i['dat'], i['Id'], i['Name'], i['Year'], i['Gender'], i['State'], i['Count']) for i in dr ]
lastrowid = execute_sql_statement(conn, data)
select_all(conn)
main()
I added a create_table() function just to test my code. I also made up a sample test file as follows:
dat,Id,Name,Year,Gender,State,Count
1,1,John,1998,M,Washington,2
2,2,Luke,2000,M,Arkansas,10
3,3,Carrie,1999,F,Texas,3
The output of the select_all() function is:
('1',1,'John',1998,'M','Washington',2)
('2',2,'Luke',2000,'M','Arkansas',10)
('3',3,'Carrie',1999,'F','Texas',3)
I am trying to insert data into my Postgres Database, I am able to insert some data which is something else but not the actual data
This is my data generator and data sender into my query execution function(
(Python-Falsk)
def runid():
cmdStr = 'cd /root/Dart/log; ls -rt DartRunner*.log | tail -1 '
ret = execCmdLocal(cmdStr)
logName = ret[1].strip()
runId = ""
print('The DartRunner Log generated is: %s'%logName)
with open('/root/Dart/log/' + logName, "r") as fd:
for line in fd:
if 'runId' in line:
runId = line.split()[-1]
print('Run Id: %s'%runId)
break
print (runId) # output : Run Id: 180628-22
post_runid(runId) # output is given in below link
return jsonify({"run_id": runId})
This my database(postgres) execution method:
(Python)
def post_runid(run_id):
query = "insert into runid(runid) values(%s)"
cur.execute(query %run_id)
conn.commit()
My output looks something like this:
The above two rows are manually inserted by me but the below two rows are executed from the code, the below two rows must be same as the above ones but for some reason they are not as the original data but being generated in series
Changing %s to '%s' in the query fixed the problem.
def post_runid(run_id):
query = "insert into runid(runid) values('%s')"
cur.execute(query, (run_id,))
conn.commit()
I'm trying to add info into a record from two different files, I'm trying to achieve this by opening the first file and adding to a record, then opening the second and updating that record.
with open('C:/Users/ELITEBOOK/documents/github/chatbot/chatbot/bot/human_text.txt', 'r') as table2, open('C:/Users/ELITEBOOK/documents/github/chatbot/chatbot/bot/robo_text.txt','r') as table3:
for line in table2.readlines():
message_text = line
#for robo_line in table3.readlines():
message_intent = ''
message_entities = ''
test = 'hello'
#cursor.execute('START TRANSACTION')
cursor.execute("INSERT INTO conversation (text) VALUES ('%s')" % line)
cnx.commit()
#cursor.execute((line))
for robo_line in table3.readlines():
#message_reply = robo_line
cursor.execute("UPDATE conversation SET reply = '%s' WHERE text = %s " % (robo_line, line))
#cursor.execute(robo_line)
cnx.commit()
I am receiving a Unknown column 'start' in 'where clause' error, "start" is just the string from the first line in my second text file. I'm using string formatters right now because otherwise I get a syntax error, this code is only being used to update the DB once, not in production.
You need to put quotes around the value, since it's a string, just like you did for the string you're setting reply to.
cursor.execute("UPDATE conversation SET reply = '%s' WHERE text = '%s' " % (robo_line, line))
But it would be better to use a prepared statement rather than string formatting, to prevent SQL injection. Then you don't put quotes around placeholders, cursor.execute replaces them safely.
cursor.execute("UPDATE conversation SET reply = %s WHERE text = %s ", (robo_line, line))
Also, your looping is wrong. You don't want to loop through the entire table3 for every line in table2, you just want to read both files in parallel. See Read two textfile line by line simultaneously -python
with open('C:/Users/ELITEBOOK/documents/github/chatbot/chatbot/bot/human_text.txt', 'r') as table2, open('C:/Users/ELITEBOOK/documents/github/chatbot/chatbot/bot/robo_text.txt','r') as table3:
for line, robo_line in zip(table2, table3):
message_text = line
message_intent = ''
message_entities = ''
test = 'hello'
#cursor.execute('START TRANSACTION')
cursor.execute("INSERT INTO conversation (text, reply) VALUES (%s, %s)", (line, robo_line))
cnx.commit()
I have a csv file like this:
nohaelprince#uwaterloo.ca, 01-05-2014
nohaelprince#uwaterloo.ca, 01-05-2014
nohaelprince#uwaterloo.ca, 01-05-2014
nohaelprince#gmail.com, 01-05-2014
I am reading the above csv file and extracting domain name and also the count of emails address by domain name and date as well. All these things I need to insert into MySQL table called domains.
Below is the code in which is giving me error as TypeError: not enough arguments for format string and it's happening when I try to insert into domains table.
#!/usr/bin/python
import fileinput
import csv
import os
import sys
import time
import MySQLdb
from collections import defaultdict, Counter
domain_counts = defaultdict(Counter)
# ======================== Defined Functions ======================
def get_file_path(filename):
currentdirpath = os.getcwd()
# get current working directory path
filepath = os.path.join(currentdirpath, filename)
return filepath
# ===========================================================
def read_CSV(filepath):
with open('emails.csv') as f:
reader = csv.reader(f)
for row in reader:
domain_counts[row[0].split('#')[1].strip()][row[1]] += 1
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="root", # your username
passwd="abcdef1234", # your password
db="test") # name of the data base
cur = db.cursor()
q = """INSERT INTO domains(domain_name, cnt, date_of_entry) VALUES(%s, %s, STR_TO_DATE(%s, '%d-%m-%Y'))"""
for domain, data in domain_counts.iteritems():
for email_date, email_count in data.iteritems():
cur.execute(q, (domain, email_count, email_date))
db.commit()
# ======================= main program =======================================
path = get_file_path('emails.csv')
read_CSV(path) # read the input file
What is wrong I am doing?
As of now my data type for date_of_entry column is date in MySQL.
You need the "%d-%m-%Y" in your sql statement in exact this way. But python (or the execute command) tries first to use it for string formatting and throws this error.
I think you have to escape it and you should try following:
q = """INSERT INTO domains(domain_name, cnt, date_of_entry) VALUES(%s, %s, STR_TO_DATE(%s, '%%d-%%m-%%Y'))"""
Try this:
q = """INSERT INTO domains(domain_name, cnt, date_of_entry) VALUES(%s, %s, STR_TO_DATE(%s, 'd-m-Y'))"""
So we have changed from STR_TO_DATE(%s, '%d-%m-%Y')) to STR_TO_DATE(%s, 'd-m-Y'))
It is detecting the %s as a format string and failing on that. You need to surround it with quotes I guess
INSERT INTO domains(domain_name, cnt, date_of_entry) VALUES(%s, %s, STR_TO_DATE('%s', '%d-%m-%Y'))
I am trying to make a random code generator in python that writes to a database. I have the codes generating and writing to the database, but instead of adding full codes to the database it loops through letters. Here is my code for the code generator:
import string
import random
import sqlite3
def id_generator():
db = sqlite3.connect('codes.db')
c = db.cursor()
number_of_codes = 10
stringLength = 9
id_code = input("what letter should this begin with: \n")
id_code = id_code.upper()
dict_ofcodes = []
for x in range(0, number_of_codes):
codez = (''.join(random.choice(string.ascii_uppercase) for i in range(stringLength)))
final_codez = (id_code + codez)
dict_ofcodes.insert(x, final_codez)
print (dict_ofcodes)
dict_ofcodes_tuple = tuple(dict_ofcodes)
print(dict_ofcodes_tuple)
for x in range(0, number_of_codes):
c.executemany(''' INSERT INTO codes(codes) VALUES(?)''', dict_ofcodes_tuple[x])
db.commit()
db.close()
id_generator()
Here is what it prints
['AALRRIULNC', 'AZTKZBKTLK', 'ATWMWYWICO', 'AWQJIJYEJH', 'AQFIONPUNJ', 'AMJRXUIJXM', 'AUDRLSBLSG', 'ABXYXDUMPD', 'AUAXRQURBH', 'ADQEVIRDFU']
('AALRRIULNC', 'AZTKZBKTLK', 'ATWMWYWICO', 'AWQJIJYEJH', 'AQFIONPUNJ', 'AMJRXUIJXM', 'AUDRLSBLSG', 'ABXYXDUMPD', 'AUAXRQURBH', 'ADQEVIRDFU')
It writes to the database single letters of the codes:
A
F
Y
and so on
The code I used to create the schema is contained in the a python file
import sqlite3
def writeDB():
db = sqlite3.connect('codes.db')
c = db.cursor()
# Create table
c.execute('''CREATE TABLE codes (codes TEXT)''')
# Save (commit) the changes
db.commit()
#can also close the connection if done with it.
# be sure any changes have been committed or they will be lost.
db.close()
writeDB()
I created the file with the mac terminal.
How could I write the full codes to the database?
The problem is with this line:
c.executemany(''' INSERT INTO codes(codes) VALUES(?)''', dict_ofcodes_tuple[x])
executemany is used to iterate over a list of parameters and call the sql statement for each parameter. So your dict_ofcodes_tupel[x] is treated as a character array and the INSERT is called for each character.
If you want to insert the entire string as one, use execute() instead.
c.execute(''' INSERT INTO codes(codes) VALUES(?)''', (dict_ofcodes_tuple[x],))
or
c.execute(''' INSERT INTO codes(codes) VALUES(?)''', [dict_ofcodes_tuple[x]])