Develop Reference

Unable to create data frame from MS Access query results

I'm trying to insert information from an MS Access database MDB file, unfortunately I don't know how to delimitate the columns from the database table with Python.
I'm getting the error
ValueError: Shape of passed values is (109861, 1), indices imply (3,1)
and the code I'm using is:
import os
import shutil
import pyodbc
import pandas as pd
import csv
from datetime import datetime
conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\\Users\\sguerra\\Desktop\\Python\\Measurements-2020-12-15.mdb;')
cursor = conn.cursor()
cursor.execute('select * from Measurements')
new = cursor.fetchall()
columns = ['Prod_Date','Prod_Time','CCE_SKU']
df = pd.DataFrame(new,columns)
for row in df.itertuples():
cursor.execute('''
insert into MITSF_1.dbo.MeasurementsTest ([Prod_Date],[Prod_Time],[CCE_SKU])
VALUES (?,?,?)
''',
row.Prod_Date,
row.Prod_Time,
row.CCE_SKU
)
conn.commit()

You are using the same cursor to try and execute both the select and the insert, so both of those statements would be operating on the same database. To keep things simple, you should use pandas' read_sql_query() to read the required columns from Access and then use to_sql() to write them to SQL Server:
df = pd.read_sql_query(
"SELECT [Prod_Date],[Prod_Time],[CCE_SKU] FROM Measurements",
conn,
)
from sqlalchemy import create_engine
engine = create_engine(
"mssql+pyodbc://scott:tiger#192.168.0.199/MITSF_1"
"?driver=ODBC+Driver+17+for+SQL+Server",
fast_executemany=True,
)
df.to_sql("MeasurementsTest", engine, schema="dbo",
index=False, if_exists="append",
)

how to connect to sqlite from sqlalchemy

I have a sqlite db in my home dir.
stephen#stephen-AO725:~$ pwd
/home/stephen
stephen#stephen-AO725:~$ sqlite db1
SQLite version 2.8.17
Enter ".help" for instructions
sqlite> select * from test
...> ;
3|4
5|6
sqlite> .quit
when I try to connect from a jupiter notebook with sqlalchemy and pandas, sth does not work.
db=sqla.create_engine('sqlite:////home/stephen/db1')
pd.read_sql('select * from db1.test',db)
~/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py in do_execute(self, cursor, statement, parameters, context)
578
579 def do_execute(self, cursor, statement, parameters, context=None):
--> 580 cursor.execute(statement, parameters)
581
582 def do_execute_no_params(self, cursor, statement, context=None):
DatabaseError: (sqlite3.DatabaseError) file is not a database
[SQL: select * from db1.test]
(Background on this error at: http://sqlalche.me/e/4xp6)
I also tried:
db=sqla.create_engine('sqlite:///~/db1')
same result

Personally, just to complete the code of #Stephen with the modules required:
# 1.-Load module
import sqlalchemy
import pandas as pd
#2.-Turn on database engine
dbEngine=sqlalchemy.create_engine('sqlite:////home/stephen/db1.db') # ensure this is the correct path for the sqlite file.
#3.- Read data with pandas
pd.read_sql('select * from test',dbEngine)
#4.- I also want to add a new table from a dataframe in sqlite (a small one)
df_todb.to_sql(name = 'newTable',con= dbEngine, index=False, if_exists='replace')
Another way to read is using sqlite3 library, which may be more straighforward:
#1. - Load libraries
import sqlite3
import pandas as pd
# 2.- Create your connection.
cnx = sqlite3.connect('sqlite:////home/stephen/db1.db')
cursor = cnx.cursor()
# 3.- Query and print all the tables in the database engine
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
print(cursor.fetchall())
# 4.- READ TABLE OF SQLITE CALLED test
dfN_check = pd.read_sql_query("SELECT * FROM test", cnx) # we need real name of table
# 5.- Now I want to delete all rows of this table
cnx.execute("DELETE FROM test;")
# 6. -COMMIT CHANGES! (mandatory if you want to save these changes in the database)
cnx.commit()
# 7.- Close the connection with the database
cnx.close()
Please let me know if this helps!

import sqlalchemy
engine=sqlalchemy.create_engine(f'sqlite:///db1.db')
Note: that you need three slashes in sqlite:/// in order to use a relative path for the DB. If you want an absolute path, use four slashes: sqlite:////
Source: Link

The issue is no backward compatibility as noted by Everila. anaconda installs its own sqlite, which is sqlite3.x and that sqlite cannot load databases created by sqlite 2.x
after creating a db with sqlite 3 the code works fine
db=sqla.create_engine('sqlite:////home/stephen/db1')
pd.read_sql('select * from test',db)
which confirms the 4 slashes are needed.

None of the sqlalchemy solutions worked for me with python 3.10.6 and sqlalchemy 2.0.0b4, it could be a beta issue or version 2.0.0 changed things. #corina-roca's solution was close, but not right as you need to pass a connection object, not an engine object. That's what the documentation says, but it didn't actually work. After a bit of experimentation, I discovered that engine.raw_connect() works, although you get a warning on the CLI. Here are my working examples
The sqlite one works out of the box - but it's not ideal if you are thinking of changing databases later
import sqlite3
conn = sqlite3.connect("sqlite:////home/stephen/db1")
df = pd.read_sql_query('SELECT * FROM test', conn)
df.head()
# works, no problem
sqlalchemy lets you abstract your db away
from sqlalchemy import create_engine, text
engine = create_engine("sqlite:////home/stephen/db1")
conn = engine.connect() # <- this is also what you are supposed to
# pass to pandas... it doesn't work
result = conn.execute(text("select * from test"))
for row in result:
print(row) # outside pands, this works - proving that
# connection is established
conn = engine.raw_connection() # with this workaround, it works; but you
# get a warning UserWarning: pandas only
# supports SQLAlchemy connectable ...
df = pd.read_sql_query(sql='SELECT * FROM test', con=conn)
df.head()

Pyhive Presto insert select * from not running

I can us PYHIVE to connect to PRESTO and select data back just fine. I am trying to use PYHIVE to run "insert into x select from y" on presto and it is not running. I am sure I am missing something simple.
from pyhive import presto
import requests
from requests.auth import HTTPBasicAuth
import pandas as pd
req_kw = {'auth': HTTPBasicAuth(user, pw),'verify':False}
conn = presto.connect(host=ht,port=prt,protocol='https',catalog='hive',username=user,requests_kwargs=req_kw)
cursor = conn.cursor()
query='select count(1) from dim.date_dim '
cursor.execute(query)
print(cursor.fetchall())
query='insert into flowersc.date_dim select * from dim.date_dim'
cursor.execute(query)
query='select count(1) from flowersc.date_dim '
cursor.execute(query)
print(cursor.fetchall())
no errors occur
but the results show no data loaded
[(16624,)]
[(0,)]
Any help is greatly appreciated.

You need to check (fetch) result in
query='insert into flowersc.date_dim select * from dim.date_dim'
cursor.execute(query).next() # added .next()
This is needed due to a change in Presto in May 2018 (https://github.com/prestosql/presto/commit/568449b8d058ed8281cc5277bb53902fd044cad7). But it's also a good practise to verify query results, i.e. check that your INSERT statement succeeds.

Error while trying to execute the query in Denodo using Python SQLAlchemy

I'm trying to get a table from Denodo using Python and sqlalchemy library. That's my code
from sqlalchemy import create_engine
import os
sql = """SELECT * FROM test_table LIMIT 10 """
engine = create_engine('mssql+pyodbc://DenodoODBC', encoding='utf-8')
con = engine.connect().connection
cursor = con.cursor()
cursor.execute(sql)
df = cursor.fetchall()
cursor.close()
con.close()
When I'm trying to run it for the first time I get the following error.
DBAPIError: (pyodbc.Error) (' \x10#', "[ \x10#] ERROR: Function 'schema_name' with arity 0 not found\njava.sql.SQLException: Function 'schema_name' with arity 0 not found;\nError while executing the query (7) (SQLExecDirectW)")
[SQL: SELECT schema_name()]
I think the problem might be with create_engine because when I'm trying to run the code for the second time without creating an engine again, everything is fine.
I hope somebody can explain me what is going on. Thanks :)

I would like to write the most efficient code to import Microsoft SQL data to a data frame

I would like to load a SQL query into a data frame as efficiently as possible. I read different sources and everyone seems to use a different approach. I am not sure why... Some are using cursors some aren't.
Currently I have:
import pandas as pd
import pyodbc
con = pyodbc.connect('Driver={something};'
'Server=something;'
'Database=something;'
'Trusted_Connection=yes;'
)
sql="""
SQL CODE
"""
df = pd.read_sql_query(con,sql)
And for some reason, this doesn't work in my machine.

Just pack it in a function. Also I add username and password (just in case)
import pandas as pd
import pyodbc
def GetSQLData(dbName, query):
sPass = 'MyPassword'
sServer = 'MyServer\\SQL1'
uname = 'MyUser'
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=" + sServer + ";"
"Database=" + dbName + ";"
"uid=" + uname + ";pwd=" + sPass)
df = pd.read_sql(query, cnxn)
return df

Try this solution
import pyodbc
import pandas as pd
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=something;"
"Database=something;"
"Trusted_Connection=yes;")
cursor = cnxn.cursor()
cursor.execute('SELECT * FROM something')
for row in cursor:
print(row)

import pandas as pd
import pyodbc
con = pyodbc.connect('Driver={something};'
'Server=something;'
'Database=something;'
'Trusted_Connection=yes;'
)
cursor = con.cursor()
cursor.execute("SQL Syntax")
Not quite sure what your last line is doing, but the cursor method works well and runs with efficiently with minimal code.
This should run. You can test it by adding in sqllist = list(cursor.fetchall()) and then print(sqllist)