Develop Reference

how to fetch multiple tables using spark sql

I am fetching data from mysql using pyspark which for only one table.I want to fetch all tables from mysql db. Don't want call jdbc connection again and again. see code below
Is it possible to simplify my code? Thank you in advance
url = "jdbc:mysql://localhost:3306/dbname"
table_df=sqlContext.read.format("jdbc").option("url",url).option("dbtable","table_name").option("user","root").option("password", "root").load()
sqlContext.registerDataFrameAsTable(table_df, "table1")
table_df_1=sqlContext.read.format("jdbc").option("url",url).option("dbtable","table_name_1").option("user","root").option("password", "root").load()
sqlContext.registerDataFrameAsTable(table_df_1, "table2")

you need somehow to acquire the list of the tables you have in mysql.
Either you find some sql commands to do that, or you manually create a file containing everything.
Then, assuming you can create a list of tablenames in python tablename_list, you can simply loop over it like this :
url = "jdbc:mysql://localhost:3306/dbname"
reader = (
sqlContext.read.format("jdbc")
.option("url", url)
.option("user", "root")
.option("password", "root")
)
for tablename in tablename_list:
reader.option("dbtable", tablename).load().createTempView(tablename)
This will create a temporary view with the same tablename. If you want another name, you can probably change the initial tablename_list with a list of tuples (tablename_in_mysql, tablename_in_spark).

#Steven already gave a perfect answer. As he said, in order to find a Python list of tablenames, you can use:
#list of the tables in the server
table_names_list = spark.read.format('jdbc'). \
options(
url='jdbc:postgresql://localhost:5432/', # database url (local, remote)
dbtable='information_schema.tables',
user='YOUR_USERNAME',
password='YOUR_PASSWORD',
driver='org.postgresql.Driver'). \
load().\
filter("table_schema = 'public'").select("table_name")
#DataFrame[table_name: string]
# table_names_list.collect()
# [Row(table_name='employee'), Row(table_name='bonus')]
table_names_list = [row.table_name for row in table_names_list.collect()]
print(table_names_list)
# ['employee', 'bonus']
Note that this is in PostgreSQL. You can easily change url and driver arguments.

Can't get MySQL Connector/Python to Return Dictionary

I have a Python application, in which I'm calling a MySQL stored procedure from my view, like so:
import mysql.connector
proc = 'audit_report'
parms = [data['schoolid'], dateToISO(data['startdatedefault'],'from'), dateToISO(data['enddatedefault'],'to'), joinIntList(data['studypgms'], joinWith), joinIntList(data['fedpgms'], joinWith), joinIntList(data['statuses'], joinWith), data['fullssndefault']]
conn = mysql.connector.connect(user='usr', database='db', password='pwd')
cursor = conn.cursor(dictionary=True)
cursor.callproc(proc, parms)
for result in cursor.stored_results():
print(result.fetchall())
I am getting the data returned as a list of tuples, the standard output. Since I'm using connector version 2.1.7, the docs say adding
dictionary=True
to the cursor declaration should cause the rowset to be returned as a list of dictionaries, with column name as the key of each dictionary. Main difference between my application and the example in the docs is that I'm using cursor.callproc(), whereas the examples use cursor.execute() with actual sql code.
I tried
print(cursor.column_names)
to see if I could get the column names that way, but all I get is
('#_audit_report_arg1', '#_audit_report_arg2', '#_audit_report_arg3', '#_audit_report_arg4', '#_audit_report_arg5', '#_audit_report_arg6', '#_audit_report_arg7')
which looks more like the input parameters to the stored procedure.
Is there any way to actually get the column names of the returned data? The procedure is somewhat complex and contains crosstab-type manipulation, but calling the same stored procedure from MySQL Workbench happily supplies the column names.
Normally, knowing what the output is supposed to be, I could hard-code column names, except this procedure crosstabs the data for the last few columns, and it is unpredictable what they will be until after the query runs.
Thanks...

You can use pymysql in python3 and it should work fine !!
import pymysql.cursors
connection = pymysql.connect(host='',
user='',
password='',
db='test',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
# Read a single record
sql = "query"
cursor.execute(sql)
result = cursor.fetchone()
num_fields = len(cursor.description)
field_names = [i[0] for i in cursor.description]
print (field_names)
finally:
connection.close()

Python MySQL cursor fails to fetch rows

I am trying to fetch data from AWS MariaDB:
cursor = self._cnx.cursor()
stmt = ('SELECT * FROM flights')
cursor.execute(stmt)
print(cursor.rowcount)
# prints 2
for z in cursor:
print(z)
# Does not iterate
row = cursor.fetchone()
# row is None
rows = cursor.fetchall()
# throws 'No result set to fetch from.'
I can verify that table contains data using MySQL Workbench. Am I missing some step?
EDIT: re 2 answers:
res = cursor.execute(stmt)
# res is None
EDIT:
I created new Python project with a single file:
import mysql.connector
try:
cnx = mysql.connector.connect(
host='foobar.rds.amazonaws.com',
user='devuser',
password='devpasswd',
database='devdb'
)
cursor = cnx.cursor()
#cursor = cnx.cursor(buffered=True)
cursor.execute('SELECT * FROM flights')
print(cursor.rowcount)
rows = cursor.fetchall()
except Exception as exc:
print(exc)
If I run this code with simple cursor, fetchall raises "No result set to fetch from". If I run with buffered cursor, I can see that _rows property of cursor contains my data, but fetchall() returns empty array.

Your issue is that cursor.execute(stmt) returns an object with results and you're not storing that.
results = cursor.execute(stmt)
print(results.fetchone()) # Prints out and pops first row

For the future googlers with the same Problem I found a workaround which may help in some cases:
I didn't find the source of the problem but a solution which worked for me.
In my case .fetchone() also returned none whatever I did on my local(on my own Computer) Database. I tried the exact same code with the Database on our companies server and somehow it worked. So I copied the complete server Database onto my local Database (by using database dumps) just to get the server settings and afterwards I also could get data from my local SQL-Server with the code which didn't work before.
I am a SQL-newbie but maybe some crazy setting on my local SQL-Server prevented me from fetching data. Maybe some more experienced SQL-user knows this setting and can explain.

Python code not deleting record from database

This is the code I have to delete a record from two tables in my database that share the same ID code and I'm not too sure where I've gone wrong. Anything missing? I've checked this a million times
def deletePhoto(photoID):
"""
Middleware function to delete a photo post
"""
#connect to the database
conn, cursor = getConnectionAndCursor()
#create sql to delete from the ratings table
sql = """
DELETE
FROM ratings
WHERE photoID= %s
"""
#set the parameters
parameters = (photoID)
#execute the sql
cursor.execute(sql, parameters)
#create sql to delete from the photo table
sql = """
DELETE
FROM photo
WHERE photoID = %s
"""
#set the parameters
parameters = (photoID)
#execute the sql
cursor.execute(sql, parameters)
#fetch the data
data = cursor.rowcount
#clean up
conn.commit()
cursor.close()
conn.close()

You might try adding a sleeper after your executes.
It can take some time for the server to process your query.
import time
time.sleep(x)
x in seconds

You need to pass in a sequence for the second argument. Using just parentheses does not create a sequence. To top this off, if photoID then that is a sequence too, one that consists of individual characters.
To create a tuple, you need to use a comma. Parentheses are optional here:
parameters = photoID,
or
parameters = (photoID,)
If you find it easier to avoid mistakes here, you could also make it a list:
parameters = [photoID]
You only have to do this once.
As a side note, you can use the MySQLdb connection object, as well as the cursor, as context managers:
with connection, cursor:
ratings_delete = """
DELETE FROM ratings
WHERE photoID= %s
"""
cursor.execute(ratings_delete, (photoID,))
photo_delete = """
DELETE FROM photo
WHERE photoID = %s
"""
cursor.execute(photo_delete, (photoID,))
The with statement will then take care of closing the cursor and connection for you, and if nothing has gone wrong in the block (no exceptions were raised), will also commit the transaction for you.

Confirmation that a postgres 'update' query worked in python

I've written my first 'update' query in python, while it seems correct, I'm not sure how to receive back the output to confirm it worked..
This is supposed to load a CSV file and replace the values in the first column with those in the second:
def main():
try:
conn=psycopg2.connect("dbname='subs' user='subs' host='localhost' password=''")
except:
print "I am unable to connect to the database."
sys.exit()
with open("dne.txt", "r+") as f:
for line in f:
old = line.split(',')[0].strip()
new = line.split(',')[1].strip()
cur = conn.cursor()
cur.execute("UPDATE master_list SET subs = '{0}' WHERE subs = '{1}';".format(new, old))
conn.commit()
results = cur.fetchall()
for each in results:
print str(each)
if __name__=="__main__":
main()
I thought the results (UPDATE 1 for each change?) would come back as a tuple, but I got an error instead:
psycopg2.ProgrammingError: no results to fetch
I'm not sure if this means my query just didn't work and there were no updates, or if I can't use fetchall() like I'm trying to.
Any feedback or suggestions welcome!

The UPDATE statement won't return any values as you are asking the database to update its data not to retrieve any data.
By far the best way to get the number of rows updated is to use cur.rowcount. This works with other drivers too, like with Psycopg2 for Postgresql it's the same syntax.
cur.execute("UPDATE master SET sub = ('xyz') WHERE sub = 'abc'")
print(cur.rowcount)
A more roundabout way of checking the update is by running a SELECT against the table after updating it; you should get the data returned. In my example below the first SELECT will return the row(s) where the update will happen. The second SELECT after the update should then return no rows as you have already updated all fields. The third SELECT should return the rows you have updated, plus any that already existed with the 'xyz' value.
import sqlite3
def main():
try:
conn=sqlite3.connect(":memory:")
cur = conn.cursor()
cur.execute("create table master(id text, sub text)")
cur.execute("insert into master(id, sub) values ('1', 'abc')")
cur.execute("insert into master(id, sub) values ('2', 'def')")
cur.execute("insert into master(id, sub) values ('3', 'ghi')")
conn.commit()
except:
print("I am unable to connect to the database.")
sys.exit()
cur.execute("select id, sub from master where sub='abc'")
print(cur.fetchall())
cur.execute("UPDATE master SET sub = ('xyz') WHERE sub = 'abc'")
conn.commit()
cur.execute("select id, sub from master where sub='abc'")
print(cur.fetchall())
cur.execute("select id, sub from master where sub='xyz'")
print(cur.fetchall())
if __name__=="__main__":
main()

In PostgreSQL 9.5 or later you can add RETURNING * to end your query that then returns the modified rows.
PostgreSQL docs: https://www.postgresql.org/docs/9.5/dml-returning.html
Sometimes it is useful to obtain data from modified rows while they
are being manipulated. The INSERT, UPDATE, and DELETE commands all
have an optional RETURNING clause that supports this. Use of RETURNING
avoids performing an extra database query to collect the data, and is
especially valuable when it would otherwise be difficult to identify
the modified rows reliably.