Develop Reference

[Python to MS SQL]: Alternative to DataFrame.to_sql without using sqlalchemy.create_engine() and using pypyodbc

Scenario:
1.I am trying to insert the dataframe directly into SQL Table.
engine_azure = sqlalchemy.create_engine(sqlchemy_conn_str,echo=True,fast_executemany = True, poolclass=NullPool)
conn = engine_azure.connect()
df_final_result.to_sql('Employee', engine_azure,schema='dbo', index = False, if_exists = 'replace')
2.So is there any alternative to the above .to_sql using pypyodbc connection?
3.Below code we can use but i have 90 columns, so i want to avoid code with below iteration.
import pyodbc
cnxn = pyodbc.connect(driver='{ODBC Driver 17 for SQL Server}', server='xyz', database='xyz',
trusted_connection='yes'
cursor = cnxn.cursor()
for index, row in df2.iterrows():
cursor.execute("INSERT INTO Employee(ContactNumber,Name,Salary,Address) values(?,?,?,?,?,?)", row.ContactNumber, row.Name, row.Salary,row['Address'])
cnxn.commit()
cnxn.close()

JSON response into database

Ok, I have tried several kinds of solutions recommended by others on this site and other sited. However, I can't get it work as I would like it to do.
I get a XML-response which I normalize and then save to a CSV. This first part works fine.
Instead of saving it to CSV I would like to save it into an existing table in an access database. The second part below:
Would like to use an existing table instead of creating a new one
The result is not separated with ";" into different columns. Everything ends up in the same column not separated, see image below
response = requests.get(u,headers=h).json()
dp = pd.json_normalize(response,'Units')
response_list.append(dp)
export = pd.concat(response_list)
export.to_csv(r'C:\Users\username\Documents\Python Scripts\Test\Test2_'+str(now)+'.csv', index=False, sep=';',encoding='utf-8')
access_path = r"C:\Users\username\Documents\Python Scripts\Test\Test_db.accdb"
conn = pyodbc.connect("DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};DBQ={};" \
.format(access_path))
strSQL = "SELECT * INTO projects2 FROM [text;HDR=Yes;FMT=sep(;);" + \
"Database=C:\\Users\\username\\Documents\\Python Scripts\\Test].Testdata.csv;"
cur = conn.cursor()
cur.execute(strSQL)
conn.commit()
conn.close()

If you already have the data in a well-formed pandas DataFrame then you don't really need to dump it to a CSV file; you can use the sqlalchemy-access dialect to push the data directly into an Access table using pandas' to_sql() method:
from pprint import pprint
import urllib
import pandas as pd
import sqlalchemy as sa
connection_string = (
r"DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};"
r"DBQ=C:\Users\Public\Database1.accdb;"
r"ExtendedAnsiSQL=1;"
)
connection_uri = f"access+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
engine = sa.create_engine(connection_uri)
with engine.begin() as conn:
# existing data in table
pprint(
conn.execute(sa.text("SELECT * FROM user_table")).fetchall(), width=30
)
"""
[('gord', 'gord#example.com'),
('jennifer', 'jennifer#example.com')]
"""
# DataFrame to insert
df = pd.DataFrame(
[
("newdev", "newdev#example.com"),
("newerdev", "newerdev#example.com"),
],
columns=["username", "email"],
)
df.to_sql("user_table", engine, index=False, if_exists="append")
with engine.begin() as conn:
# updated table
pprint(
conn.execute(sa.text("SELECT * FROM user_table")).fetchall(), width=30
)
"""
[('gord', 'gord#example.com'),
('jennifer', 'jennifer#example.com'),
('newdev', 'newdev#example.com'),
('newerdev', 'newerdev#example.com')]
"""
(Disclosure: I am currently the maintainer of the sqlalchemy-access dialect.)

Solved with the following code
SE_export_Tuple = list(zip(SE_export.Name,SE_export.URL,SE_export.ImageUrl,......,SE_export.ID))
print(SE_export_Tuple)
access_path = r"C:\Users\username\Documents\Python Scripts\Test\Test_db.accdb"
conn = pyodbc.connect("DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};DBQ={};" \
.format(access_path))
cursor = conn.cursor()
mySql_insert_query="INSERT INTO Temp_table (UnitName,URL,ImageUrl,.......,ID) VALUES (?,?,?,......,?)"
cursor.executemany(mySql_insert_query,SE_export_Tuple)
conn.commit()
conn.close()
However, when I add many fields I get an error at "executemany", saying:
cursor.executemany(mySql_insert_query,SE_export_Tuple)
Error: ('HY004', '[HY004] [Microsoft][ODBC Microsoft Access Driver]Invalid SQL data type (67) (SQLBindParameter)')

How to execute multiple queries in pandas?

How to execute the following queries with sqlalchemy?
import pandas as pd
import urllib
from sqlalchemy import create_engine
from sqlalchemy.types import NVARCHAR
params = urllib.parse.quote_plus(r'DRIVER={SQL Server};SERVER=localhost\SQLEXPRESS;Trusted_Connection=yes;DATABASE=my_db;autocommit=true;MultipleActiveResultSets=True')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str, encoding = 'utf-8-sig')
with engine.connect() as con:
con.execute('Declare #latest_date nvarchar(8);')
con.execute('SELECT #latest_date = max(date) FROM my_table')
df = pd.read_sql_query('SELECT * from my_db where date = #latest_date', conn_str)
However, an error occured:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Must declare the scalar variable "#latest_date". (137) (SQLExecDirectW)')
How to solve this problem?
Thanks.

You don't need to declare a variable and use so many queries, you can do it just with one query:
SELECT *
FROM my_db
WHERE date = (SELECT max(date)
FROM my_db)
And then you can use, i use backticks because date is a reserved word:
with engine.connect() as con:
query="SELECT * FROM my_db WHERE `date` = (SELECT max(`date`) FROM my_db)"
df = pd.read_sql(query, con=con)

Using Python to extract data from MS Access

I need to extract a table from Access and print it in python. I have successfully connected the Access data base but I am not sure how to pull the table from Access and move it into a python data frame. I have inserted my code below.
odbc_conn_str = 'DRIVER={Microsoft Access Driver (*.mdb,
*.accdb)};DBQ=%s;UID=%s;PWD=%s' % (db_file, user, password)
conn = pyodbc.connect(odbc_conn_str)
cur = conn.cursor()
SQLCommand = 'select *from table1'
df = cur.execute(SQLCommand)
print(df)
conn.commit()
I get no errors but all this returns is
<pyodbc.Cursor object at 0x0BCFF3A0>

The fetchall() will retrieve the result
odbc_conn_str = 'DRIVER={Microsoft Access Driver (*.mdb,
*.accdb)};DBQ=%s;UID=%s;PWD=%s' % (db_file, user, password)
conn = pyodbc.connect(odbc_conn_str)
cur = conn.cursor()
SQLCommand = 'select * from table1'
cur.execute(SQLCommand)
df = cur.fetchall()
print(df)
You don't need to commit a select statement

Connecting to SQL server using PYODBC

I am able to connect to the SQL server 2008 R2 using Python in jupyter notebook, but when I select top 10 rows from a table, the results are not rendered on the screen. I do not get any error. I need to know how can I select the data from a table in SQL and the result gets displayed on the screen. Below is code that I used:
import pyodbc
con = pyodbc.connect('Trusted_Connection=yes', driver = '{ODBC Driver 13 for SQL Server}',server = 'ServerName', database = 'DBname')
cursor.execute("select top 10 accountid from Table")
rows = cursor.fetchall()
for row in rows:
print(row)

It looks like you missed creating the actual cursor:
import pyodbc
con = pyodbc.connect('Trusted_Connection=yes', driver = '{ODBC Driver 13 for SQL Server}',server = 'ServerName', database = 'DBname')
cursor = con.cursor()
cursor.execute("select top 10 accountid from Table")
rows = cursor.fetchall()
for row in rows:
print(row)
Good luck!