I am using this sample sqlite database and my code is
import sqlite3
conn = sqlite3.connect('chinook.db')
conn.execute("SELECT * FROM tracks")
rows = conn.cursor().fetchall()
It should have worked, but rows is empty?
What am I doing wrong here?
The Connection.execute shortcut returns a cursor instance, which you need to use with fetchall. In your code, you're creating a new, independent cursor.
Thus:
import sqlite3
conn = sqlite3.connect('chinook.db')
cursor = conn.execute("SELECT * FROM tracks")
rows = cursor.fetchall()
or even shorter (not recommended, but for those who like obscured one-liners):
rows = sqlite3.connect('chinook.db').execute("SELECT * FROM tracks").fetchall()
Or don't use Connection.execute shortcut, to avoid confusion:
import sqlite3
conn = sqlite3.connect('chinook.db')
cursor = conn.cursor()
cursor.execute("SELECT * FROM tracks")
rows = cursor.fetchall()
Related
I'm trying to connect to an SQL database and, within a loop, create separate dataframes for each different instance of Id, containing all the data related to that Id. I've tried a number of ways, without any success so far. I'm pretty new to all of this, so I'm probably making some rookie mistakes.
Attempt 1:
import pandas as pd
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=Server_name;'
'Database=Database;'
'UID=Username;'
'PWD=password;'
'Trusted_Connection=yes;')
Name = ['HR','ZA','PR','FW']
for x in Name:
SQL = '''
SELECT *
FROM Database
WHERE Id = {x}'''.format(x = x)
cursor = conn.cursor()
cursor.execute(SQL)
df = pd.read_sql_query(SQL)
On this code, I get an 'invalid column name' programming error on the first Name 'HL'.
Attempt 2:
import pandas as pd
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=Server_name;'
'Database=Database;'
'UID=Username;'
'PWD=password;'
'Trusted_Connection=yes;')
SQL = '''
SELECT *
FROM Database
conn.autocommit = True
cursor.execute(SQL)
for [Id] in cursor:
df = pd.Dataframe(SQL,conn)
On this code, I get a 'ValueError: too many values to unpack (expected 1)' - on the for statement.
I want to put a lot more code in the for loop so I need it to be set up to work through each Id. I hope that makes sense. Any guidance would be greatly appreciated. Thanks
UPDATE:
Thanks for all comments/answers. For some reason I just couldn't get it to work in either of the formats above so I took it back to where I started from now I understand how to include the syntax for the loop variable. The following now works:
import pandas as pd
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=Server_name;'
'Database=Database;'
'UID=Username;'
'PWD=password;'
'Trusted_Connection=yes;')
Name = ['HR','ZA','PR','FW']
for x in Name:
SQL = pd.read_sql_query(
'''
SELECT *
FROM Database_table
WHERE Id = '{x}'
'''.format(x = x), conn)
df = pd.DataFrame(SQL)
I think that if you try a variation on your first attempt like:
for x in Name:
SQL = '''
SELECT *
FROM Database
WHERE Id = ?'''
cursor = conn.cursor()
cursor.execute(SQL)
df = pd.read_sql_query(SQL, params={x})
It should probably work :)
So currently when I execute SELECT query and retrieve data I have to get results like this:
connection = psycopg2.connect(user="admin",
password="admin",
host="127.0.0.1",
port="5432",
database="postgres_db")
cursor = connection.cursor()
cursor.execute("SELECT * FROM user")
users = cursor.fetchall()
for row in users:
print(row[0])
print(row[1])
print(row[2])
What I want to do is, use column names instead of integers, like this:
for row in users:
print(row["id"])
print(row["first_name"])
print(row["last_name"])
Is this possible, and if it is, then how to do it?
You need to use RealDictCursor, then you can access the results like a dictionary:
import psycopg2
from psycopg2.extras import RealDictCursor
connection = psycopg2.connect(user="...",
password="...",
host="...",
port="...",
database="...",
cursor_factory=RealDictCursor)
cursor = connection.cursor()
cursor.execute("SELECT * FROM user")
users = cursor.fetchall()
print(users)
print(users[0]['user'])
Output:
[RealDictRow([('user', 'dbAdmin')])]
dbAdmin
no need to call fetchall() method, the psycopg2 cursor is an iterable object you can directly do:
cursor.execute("SELECT * FROM user")
for buff in cursor:
row = {}
c = 0
for col in cursor.description:
row.update({str(col[0]): buff[c]})
c += 1
print(row["id"])
print(row["first_name"])
print(row["last_name"])
I am trying to execute the following script. but I don't get neither the desired results nor a error message ,and I can't figure out where I'm doing wrong.
import pyodbc
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=mySRVERNAME;"
"Database=MYDB;"
"uid=sa;pwd=MYPWD;"
"Trusted_Connection=yes;")
cursor = cnxn.cursor()
cursor.execute('select DISTINCT firstname,lastname,coalesce(middlename,\' \') as middlename from Person.Person')
for row in cursor:
print('row = %r' % (row,))
any ideas ? any help is appreciated :)
You have to use a fetch method along with cursor. For Example
for row in cursor.fetchall():
print('row = %r' % (row,))
EDIT :
The fetchall function returns all remaining rows in a list.
If there are no rows, an empty list is returned.
If there are a lot of rows, *this will use a lot of memory.*
Unread rows are stored by the database driver in a compact format and are often sent in batches from the database server.
Reading in only the rows you need at one time will save a lot of memory.
If we are going to process the rows one at a time, we can use the cursor itself as an interator
Moreover we can simplify it since cursor.execute() always returns a cursor :
for row in cursor.execute("select bla, anotherbla from blabla"):
print row.bla, row.anotherbla
Documentation
I found this information useful to retrieve data from SQL database to python as a data frame.
import pandas as pd
import pymssql
con = pymssql.connect(server='use-et-aiml-cloudforte-aiops- db.database.windows.net',user='login_username',password='login_password',database='database_name')
cursor = con.cursor()
query = "SELECT * FROM <TABLE_NAME>"
cursor.execute(query)
df = pd.read_sql(query, con)
con.close()
df
import mysql.connector as mc
connection creation
conn = mc.connect(host='localhost', user='root', passwd='password')
print(conn)
#create cursor object
cur = conn.cursor()
print(cur)
cur.execute('show databases')
for i in cur:
print(i)
query = "Select * from employee_performance.employ_mod_recent"
emp_data = pd.read_sql(query, conn)
emp_data
I have some Python code the selects data from Oracle spatial and inserts into Spatialite. My problem is that the cursor contains the geometry in binary and I can’t figure out how to read the binary into the Spatialite insert statement. Just to added this all works if I use WKT but some of the geometries are too long hence the reason for the binary format.
Can anyone help please?
# Import system modules
import cx_Oracle
from pyspatialite import dbapi2 as sl_db
def db_connect():
# Build connect from TNS names
o_db = cx_Oracle.connect("xxxxx", "xxxxx", "xxxxx_gl_dev")
cursor = o_db.cursor()
return cursor
def db_lookup(cursor):
# Select records
sql = "SELECT sdo_util.to_wkbgeometry(a.shape), a.objectid FROM span a WHERE a.objectid = 1382372"
cursor.execute(sql)
row = cursor.fetchall()
return row
def db_insert(row):
# Insert Rows in new spatailite table
database_name = 'C:\\Temp\\MYDATABASE.sqlite'
db_connection = sl_db.connect(database_name)
db_cursor = db_connection.cursor()
sql = 'INSERT INTO "SPAN_OFL" ("geometry", "OBJECTID") Values GeomFromWKB(?,27700),?);'
db_cursor.executemany(sql, row)
db_connection.commit()
db_connection.close()
# main code
cursor = db_connect()
row = db_lookup(cursor)
db_insert(row)
I am trying to retrieve data from an SQL server using pyodbc and print it in a table using Python. However, I can only seem to retrieve the column name and the data type and stuff like that, not the actual data values in each row of the column.
Basically I am trying to replicate an Excel sheet that retrieves server data and displays it in a table. I am not having any trouble connecting to the server, just that I can't seem to find the actual data that goes into the table.
Here is an example of my code:
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=SQLSRV01;DATABASE=DATABASE;UID=USER;PWD=PASSWORD')
cursor = cnxn.cursor()
cursor.execute("SELECT * FROM sys.tables")
tables = cursor.fetchall()
#cursor.execute("SELECT WORK_ORDER.TYPE,WORK_ORDER.STATUS, WORK_ORDER.BASE_ID, WORK_ORDER.LOT_ID FROM WORK_ORDER")
for row in cursor.columns(table='WORK_ORDER'):
print row.column_name
for field in row:
print field
However the result of this just gives me things like the table name, the column names, and some integers and 'None's and things like that that aren't of interest to me:
STATUS_EFF_DATE
DATABASE
dbo
WORK_ORDER
STATUS_EFF_DATE
93
datetime
23
16
3
None
0
None
None
9
3
None
80
NO
61
So I'm not really sure where I can get the values to fill up my table. Would it should be in table='WORK_ORDER', but could it be under a different table name? Is there a way of printing the data that I am just missing?
Any advice or suggestions would be greatly appreciated.
You are so close!
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=SQLSRV01;DATABASE=DATABASE;UID=USER;PWD=PASSWORD')
cursor = cnxn.cursor()
cursor.execute("SELECT WORK_ORDER.TYPE,WORK_ORDER.STATUS, WORK_ORDER.BASE_ID, WORK_ORDER.LOT_ID FROM WORK_ORDER")
for row in cursor.fetchall():
print row
(the "columns()" function collects meta-data about the columns in the named table, as opposed to the actual data).
you could try using Pandas to retrieve information and get it as dataframe
import pyodbc as cnn
import pandas as pd
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=SQLSRV01;DATABASE=DATABASE;UID=USER;PWD=PASSWORD')
# Copy to Clipboard for paste in Excel sheet
def copia (argumento):
df=pd.DataFrame(argumento)
df.to_clipboard(index=False,header=True)
tableResult = pd.read_sql("SELECT * FROM YOURTABLE", cnxn)
# Copy to Clipboard
copia(tableResult)
# Or create a Excel file with the results
df=pd.DataFrame(tableResult)
df.to_excel("FileExample.xlsx",sheet_name='Results')
I hope this helps!
Cheers!
In order to receive actual data stored in the table, you should use one of fetch...() functions or use the cursor as an iterator (i.e. "for row in cursor"...). This is described in the documentation:
cursor.execute("select user_id, user_name from users where user_id < 100")
rows = cursor.fetchall()
for row in rows:
print row.user_id, row.user_name
Just do this:
import pandas as pd
import pyodbc
cnxn = pyodbc.connect("Driver={SQL Server}\
;Server=SERVER_NAME\
;Database=DATABASE_NAME\
;Trusted_Connection=yes")
df = pd.read_sql("SELECT * FROM myTableName", cnxn)
df.head()
Instead of using the pyodbc library, use the pypyodbc library... This worked for me.
import pypyodbc
conn = pypyodbc.connect("DRIVER={SQL Server};"
"SERVER=server;"
"DATABASE=database;"
"Trusted_Connection=yes;")
cursor = conn.cursor()
cursor.execute('SELECT * FROM [table]')
for row in cursor:
print('row = %r' % (row,))
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=db-server;'
'Database=db;'
'Trusted_Connection=yes;')
sql = "SELECT * FROM [mytable] "
cursor.execute(sql)
for r in cursor:
print(r)
Why pyodbc you can try with pymssql. For more information follow this link: https://stackoverflow.com/a/70445445/8614314.
import pandas as pd
import pymssql
con = pymssql.connect(<conncetion to the server and db>)
cursor = con.cursor()
query = "<Your query>"
cursor.execute(query)
df = pd.read_sql(query, con)
con.close()
Upvoted answer din't work for me, It was fixed by editing connection line as follows(replace semicolons with coma and also remove those quotes):
import pyodbc
cnxn = pyodbc.connect(DRIVER='{SQL Server}',SERVER=SQLSRV01,DATABASE=DATABASE,UID=USER,PWD=PASSWORD)
cursor = cnxn.cursor()
cursor.execute("SELECT WORK_ORDER.TYPE,WORK_ORDER.STATUS, WORK_ORDER.BASE_ID, WORK_ORDER.LOT_ID FROM WORK_ORDER")
for row in cursor.fetchall():
print row