Python & SQLite3 Selecting from two tables

Python & SQLite3 Selecting from two tables - python

I have written this code in python, which I basically opens up my SQLite3 database and looks at each row in the table 'contact' and then takes each 'id' number and then looks at the matching 'id' in the table 'Users'. My problem is that it only outputs the first one and does not loop through all the rows.
import sqlite3
conn = sqlite3.connect('sqlite3.db')
cursor = conn.cursor()
cursor2 = conn.cursor()
cursor3 = conn.cursor()
text_file = open("Output.txt", "w");
try:
cursor.execute("SELECT Id, address FROM contact;") # Get address details by ID
for row in cursor:
ID = row[0]
address= row[1]
cursor2.execute("SELECT name FROM Users WHERE id= " + str(ID) + ";") # Get users's name by ID
row2 = cursor2.fetchone()
sendername = row2[0]
text_file.write(firstname, lastname, address);
finally:
conn.close()
Any suggestions, I'm very new to python.

You can ask the database to do a join instead:
cursor.execute("""\
SELECT u.name, c.address
FROM contact c
INNER JOIN Users u ON u.id = c.Id
""")
with open('Output.txt', 'w') as outfh:
for name, address in cursor:
outfh.write('{} {}\n'.format(name, address)
The INNER JOIN tells SQLite to only pick rows for which there is an actual match on the id columns. If you marked the id column as a foreign key in the contact table, you could use a NATURAL INNER JOIN as well, and omit the ON clause.

If I understand you:
cursor.execute("SELECT Users.name, contact.address FROM Users, contact WHERE contact.Id = Users.id;")
for row in cursor:
name= row[0]
address= row[1]
text_file.write(name+" "+address)

Related

Insert data from csv to postgreSQL database via Python

I'm brand new to postgreSQL or SQL at all.
I'm trying to create a table in a database via Python and then load data from a .csv file into the table.
My code looks like this:
import csv
import psycopg2
#Establish connection to database
con = psycopg2.connect(
host = "localhost",
database = "kundeavgang",
user = "postgres",
password = "postgres",
)
#Cursor
cur = con.cursor()
#If a mistake is made, start from scratch
cur.execute("DROP TABLE IF EXISTS kundeavgang")
#Create table
cur.execute('''
CREATE TABLE "kundeavgang"(
"customerID" TEXT,
"gender" TEXT,
"SeniorCitizen" TEXT,
"Partner" TEXT,
"Dependents" TEXT,
"tenure" INT,
"PhoneService" TEXT,
"MultipleLines" TEXT,
"InternetService" TEXT,
"OnlineSecurity" TEXT,
"DeviceProtection" TEXT,
"TechSupport" TEXT,
"StreamingMovies" TEXT,
"Contract" TEXT,
"PaperlessBilling" TEXT,
"PaymentMethod" TEXT,
"MonthlyCharges" FLOAT,
"TotalCharges" FLOAT,
"Churn" TEXT
)
''')
#Acsess .csv file
with open('kundeavgang.csv') as csvFile:
reader = csv.reader(csvFile)
skipHeader = next(reader) #Account for header
for row in reader:
customerID = row[0]
gender = row[1]
SeniorCitizen = row[2]
Partner = row[3]
Dependents = row[4]
tenure = row[5]
PhoneService = row[6]
MultipleLines = row[7]
InternetService = row[8]
OnlineSecurity = row[9]
OnlineBackup = row[10]
DeviceProtection = row[11]
TechSupport = row[12]
StreamingTV = [13]
StreamingMovies = row[14]
Contract = row[15]
PaperlessBilling = row[16]
PaymentMethod = row[17]
MonthlyCharges = row[18]
TotalCharges = row[19]
Churn = row[20]
cur.execute('''INSERT INTO kundeavgang(customerID,gender,SeniorCitizen,Partner,Dependents,tenure,PhoneService,MultipleLines,InternetService,OnlineSecurity,OnlineBackup,DeviceProtection,TechSupport,StreamingTV,StreamingMovies,Contract,PaperlessBilling,PaymentMethod,TotalCharges,Churn)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)''',(customerID,gender,SeniorCitizen,Partner,Dependents,tenure,PhoneService,MultipleLines,InternetService,OnlineSecurity,OnlineBackup,DeviceProtection,TechSupport,StreamingTV,StreamingMovies,Contract,PaperlessBilling,PaymentMethod,TotalCharges,Churn))
#Commit the transaction
con.commit()
#End connection
con.close()
In pgAdmin, the table comes up as existing in the database. However, I cannot find the actual table. Further, I have no idea about this line of code:
cur.execute('''INSERT INTO kundeavgang(customerID,gender,SeniorCitizen,Partner,Dependents,tenure,PhoneService,MultipleLines,InternetService,OnlineSecurity,OnlineBackup,DeviceProtection,TechSupport,StreamingTV,StreamingMovies,Contract,PaperlessBilling,PaymentMethod,TotalCharges,Churn)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)''',(customerID,gender,SeniorCitizen,Partner,Dependents,tenure,PhoneService,MultipleLines,InternetService,OnlineSecurity,OnlineBackup,DeviceProtection,TechSupport,StreamingTV,StreamingMovies,Contract,PaperlessBilling,PaymentMethod,TotalCharges,Churn))
What does all the %s stand for? I found it off an online example which was not very helpful, so I tried it without knowing what it means. I have seen some examples where question marks are inserted instead, but also this without explanation.
Lastly, as the code stands now, I get the error message:
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)''',(customerID,gender,SeniorCitizen,Partner,Dependents,tenure,PhoneService,MultipleLines,InternetService,OnlineSecurity,OnlineBackup,DeviceProtection,TechSupport,StreamingTV,StreamingMovies,Contract,PaperlessBilling,PaymentMethod,TotalCharges,Churn))
IndexError: tuple index out of range
All help or explanations will be appreciated.

For bulk inserts from text files, consider copy_from or copy_expert of psycopg2. Also, be sure to commit your execution:
cur.execute("DROP TABLE IF EXISTS kundeavgang")
con.commit()
cur.execute('''CREATE TABLE "kundeavgang" ... ''')
con.commit()
with open('kundeavgang.csv') as csvFile:
next(csvFile) # SKIP HEADERS
cur.copy_from(csvFile, "kundeavgang", sep=",")
# POSTGRES COPY COMMAND FOR CSV MODE
# cur.copy_expert("""COPY "kundeavgang" FROM STDIN WITH CSV""", csvFile)
con.commit()

The %s are placeholders for the values that will be inserted and passed through the following tuple:
(customerID,gender,SeniorCitizen,Partner,Dependents,tenure,PhoneService,MultipleLines,InternetService,OnlineSecurity,OnlineBackup,DeviceProtection,TechSupport,StreamingTV,StreamingMovies,Contract,PaperlessBilling,PaymentMethod,TotalCharges,Churn)
The problem that your insert statement going to insert to 20 columns, you provide 20 values in your tuple but you have 22 placeholders (%s).

The problem is a mismatch between the number of columns to be populated and the length of the list provided. This is an easy mistake to make when dealing with a lot of columns. One way to reduce risk of error is to use the length of the columns or values list to build the statement.
cols = [name1, name2,...]
vals = [val1, val2, ...]
assert len(cols) == len(vals), 'mismatch between number of columns and number of values'
template = """INSERT INTO tbl ({}) VALUES ({})"""
stmt = template.format(', '.join(cols), ','.join(['%s'] * len(vals)))
cur.execute(stmt, vals)
Note that when building the column names dynamically it's good practice to quote them - psycopg2 provides tools for this.

Change the line reader = csv.reader(csvFile) to:
reader = csv.reader(csvFile, delimiter=';')

Retrieve data with a SELECT and treat it separately

I need to do a SELECT to an SQL Server table and treat the information I get separately.
For example, let's say I have this table named Table1
And I do this SELECT in python:
SELECT name, phone, date FROM Table1
In the print the result would be:
[['Sara Miller',611111111],['Jane Smith',622222222],['Amanda
Laurens',633333333]]
I need to treat each row and each name and phone number separately to send SMS... So, how can I access each one using Python?
For example, to send an SMS to the number 611111111 saying
"Dear Sara Miller, tomorrow (20/05/2020) you have an appointment in
the Clinic"
The SMS part I have covered, using an API, the problem is I can't figure out how to treat received data from SQL Server.
The code I have at the moment is:
conn = pypyodbc.connect("Connection parameters, waorking OK")
cursor = conn.cursor()
cursor.execute('SELECT name, phone, date FROM Table1')
result = cursor.fetchall()
final_result = [list(i) for i in result]
print(final_result)
If I need to clarify something please let me know.

I haven't really worked with pypyodbc so I'm not sure what the format of the data that cursor.fetchall returns so I have listed two approaches which should cover both scenarios.
conn = pypyodbc.connect("Connection parameters, waorking OK")
cursor = conn.cursor()
cursor.execute('SELECT name, phone, date FROM Table1')
for row in cursor.fetchall():
name = row[0]
phone = row[1]
date = row[2]
# do something with these variables
If the result returned is a dict instead of a list then it becomes:
for row in cursor.fetchall():
name = row['name']
phone = row['phone']
date = row['date']
# do something with these variables
Or as #DanGuzman mentions we can also do:
for row in cursor.fetchall():
name = row.name
phone = row.phone
date = row.date
# do something with these variables

Print a value in Python from oracle database

I have an issue when displaying a value in python retrieved from oracle table into CLOB field:
Oracle query:
SELECT EXTRACTVALUE(xmltype(t.xml), '/DCResponse/ResponseInfo/ApplicationId')
FROM table t
WHERE id = 2
Value displayed in Oracle Client
5701200
Python code
import cx_Oracle
conn = cx_Oracle.Connection("user/pwd#localhost:1521/orcl")
cursor = conn.cursor()
cursor.execute("""SELECT EXTRACTVALUE(xmltype(t.xml),'/DCResponse/ResponseInfo/ApplicationId') FROM table t where id = 2""")
for row in cursor:
print(row)
Python Console: Nothing is displayed!!! I want to show:5701200
Please Help.
Best Regards
Giancarlo

There are only a few issues with your code :
Replace cx_Oracle.Connection with cx_Oracle.connect
Be careful about the indentation related to the print(row)
Triple double-quotes, within the SELECT statement, are redundant,
replace them with Single double-quotes
Prefer Using print(row[0]) in order to return the desired number rather than
a tuple printed.
import cx_Oracle
conn = cx_Oracle.connect('user/pwd#localhost:1521/orcl')
cursor = conn.cursor()
query = "SELECT EXTRACTVALUE(xmltype(t.xml),'/DCResponse/ResponseInfo/ApplicationId')"
query += " FROM tab t "
query += " WHERE t.ID = 2 "
cursor.execute( query )
for row in cursor:
print(row[0])
Assigning a query to a variable not required, as stated in my case, but preferable to use in order to display the long SELECT statement decently.

If you want to iterate over result, use this one:
for row in cursor.execute("sql_query")
print(row)
or you can fetch each row like this:
cursor = conn.cursor()
cursor.execute("sql_query")
while True:
row = cursor.fetchone()
print(row)

what slows down parsing?

I have big XML files to parse (about 200k lines and 10MB). The structure is following:
<el1>
<el2>
<el3>
<el3-1>...</el3-1>
<el3-2>...</el3-2>
</el3>
<el4>
<el4-1>...</el4-1>
<el4-2>...</el4-2>
</el4>
<el5>
<el5-1>...</el4-1>
<el5-2>...</el5-2>
</el5>
</el2>
</el1>
Here is my code:
tree = ElementTree.parse(filename)
doc = tree.getroot()
cursor.execute(
'INSERT INTO first_table() VALUES()',
())
cursor.execute('SELECT id FROM first_table ORDER BY id DESC limit 1')
row = cursor.fetchone()
v_id1 = row[0]
for el1 in doc.findall('EL1'):
cursor.execute(
'INSERT INTO second_table() VALUES(v_id1)',
(v_id1))
cursor.execute(
'SELECT id FROM second_table ORDER BY id DESC limit 1')
row = cursor.fetchone()
v_id2 = row[0]
for el2 in el1.findall('EL2'):
cursor.execute(
'INSERT INTO third_table(v_id2) VALUES()',
(v_id2))
cursor.execute(
'SELECT id FROM third_table ORDER BY id DESC limit 1')
row = cursor.fetchone()
v_id3 = row[0]
for el3 in el2.findall('EL3'):
cursor.execute(
'INSERT INTO fourth_table(v_id3) VALUES()',
(v_id3))
cursor.execute(
'SELECT id FROM fourth_table ORDER BY id DESC limit 1')
row = cursor.fetchone()
v_id4 = row[0]
for el4 in el3.findall('EL4'):
cursor.execute(
'INSERT INTO fifth_table(v_id4) VALUES()',
(v_id4))
for el5 in el4.findall('EL5'):
cursor.execute(
'INSERT INTO sixth_table(v_id4) VALUES()',
(v_id4))
cursor.execute(
'SELECT id FROM sixth_table ORDER BY id DESC limit 1')
row = cursor.fetchone()
v_id5 = row[0]
...
conn.commit()
Basically I get values from attributes and send them into the database. When I need to process nested elements, I have to SELECT last inserted ID from the database and INSERT it as a foreign key into the next INSERT statement.
The whole process takes about 50s but apparently it's too long for the data I have. The SELECT statements for sure take some time, but I already selecting only 1 attribute on last row.
I don't know if it can be faster since I'm not good at programming so I ask you guys.

You have 4 nested for loops. That's why. It is O(n^4).

Replace L, in SQL results in python

I'm running pyodbc connected to my db and when i run a simply query I get a load of results back such as
(7L, )(12L,) etc.
How do I replace the the 'L, ' with '' so I can pass the ids into another query
Thanks
Here's my code
import pyodbc
cnxn = pyodbc.connect('DSN=...;UID=...;PWD=...', ansi=True)
cursor = cnxn.cursor()
rows = cursor.execute("select id from orders")
for row in rows:
test = cursor.execute("select name from customer where order_id = %(id)s" %{'id':row})
print test

Use parameters:
...
test = cursor.execute("select name from customer where order_id = ?", row.id)
...
The L after the number indicates that the value is a long type.

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.

Python & SQLite3 Selecting from two tables - python

If I understand you: cursor.execute("SELECT Users.name, contact.address FROM Users, contact WHERE contact.Id = Users.id;") for row in cursor: name= row[0] address= row[1] text_file.write(name+" "+address)

Related

Insert data from csv to postgreSQL database via Python

Retrieve data with a SELECT and treat it separately

Print a value in Python from oracle database

what slows down parsing?

Replace L, in SQL results in python

Categories

Resources