I am very new to python and SQL. I have a DB query in Excel which is very large that I am trying to move to Python directly. Currently i read it in from excel using read_excel().
Ive installed import pyodbc and tried to take the connection string from excel and put it into python.
default code that i have written is:
import pyodbc
conn_str = 'Provider.....'
conn = pyodbc.connect(conn_str)
SQL_Query = pd.read_sql_query(
'''SELECT *
FROM [MarketData].[Gas].[KplerVesselHistory]''', conn)
The excel connection info is as follows:
Connection type: OLE DB Query
Connection string: Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=True;Initial Catalog=MarketData;Data Source=FOTDAL02PR;Use Procedure for Prepare=1;Auto Translate=True;Packet Size=4096;Workstation ID=[blank];Use Encryption for Data=False;Tag with column collation when possible=False
Command type: SQL
The error I'm getting is:
Traceback (most recent call last):
File "C:\dev\bin\pycharm\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<string>", line 3, in <module>
pyodbc.InterfaceError: ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
Something like this might work:
This connects using the ODBC driver for SQL Server, runs the query and returns it to Pandas DataFrame and then prints the output.
import pyodbc
import pandas
driver = '{ODBC Driver 17 for SQL Server}'
server = 'server_name\instance_name'
database = 'MarketData'
sql = 'SELECT * FROM [MarketData].[Gas].[KplerVesselHistory]'
conn_str = 'Driver={};Server={};Trusted_Connection=True;Database={}'.format(driver, server, database)
conn = pyodbc.connect(conn_str)
df = pandas.read_sql(sql, conn)
print(df)
Related
With python, I am trying to read a table from SQL Server, populate one field and load the resulting table back to SQL Server as a new table. I'm using "petl".
I can read the table with no problems, I populate the field successfully, but then I get an error when I try to load the resulting table back to SQL Server.
This is the python code I'm using:
import petl as etl
import pyodbc
def populate_field(z, row):
...
con = pyodbc.connect('Trusted_Connection=yes', driver='{SQL Server}', server=r'my_server', database='my_db')
query = r'SELECT TOP 10 * FROM my_table'
tab = etl.fromdb(con, query)
tab = etl.convert(tab, 'my_field', populate_field, pass_row=True)
etl.todb(tab, con, 'my_new_table')
And this is the error message I get on "etl.todb(tab, con, 'my_new_table')"
Error: ('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)')
I am trying, for the first time ever, to send data from a data frame in Spyder to Azure SQL Server...I think it's called Synapse. I created a small table in the database and when I run the code below, I see the results I expect to see.
import pyodbc
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
driver= '{ODBC Driver 17 for SQL Server}'
with pyodbc.connect('DRIVER='+driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password) as conn:
with conn.cursor() as cursor:
cursor.execute("SELECT * From Order_Table")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
So, the connection is fine. I guess I am just stuck on the syntax to push a data frame to SQL Server in Azure. I tested the code below.
import pyodbc
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
driver= '{ODBC Driver 17 for SQL Server}'
conn = pyodbc.connect('DRIVER='+driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password)
all_data.to_sql('health', conn, if_exists='replace', index=True)
When I run that code, I get this error.
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Statement(s) could not be prepared. (8180)")
Try to do it by importing Pandas and pyodbc.
Below are few basic steps which we follow usually:
Connect to SQL Server.
Install all your python packages in your local.
Load the data into CSV.
Later you can use below Python script to load it from dataframe.
import pyodbc
import pandas as pd
df = pd.read_csv("c:\\user\\username\department.csv")
server = 'yourservername'
database = 'AdventureWorks'
username = 'username'
password = 'yourpassword'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
#Insert Dataframe into SQL Server:
for index, row in df.iterrows():
cursor.execute("INSERT INTO HumanResources.DepartmentTest (DepartmentID,Name,GroupName) values(?,?,?)", row.DepartmentID, row.Name, row.GroupName)
cnxn.commit()
cursor.close()
After completing the configs you can run below command to get the data from SQL:
SELECT count(*) from HumanResources.DepartmentTest;
Refer to this official doc for detailed explanation.
Now I have created two different connection for example I have a .py file called connec as the below code:
from sqlalchemy import create_engine
import pyodbc
# pyodbc connection connect to server
conn = pyodbc.connect(
"driver={SQL Server};server=WINKPN-3B5JTT2\SMARTRNO_EXPRESS; database=master; trusted_connection=true",
autocommit=True, Trusted_Connection='Yes')
crsr = conn.cursor()
# connect db (connect to database name) using SQL-Alchemy
engine = create_engine(
'mssql+pyodbc://WINKPN-3B5JTT2\SMARTRNO_EXPRESS/MyNewDatabase?driver=SQL+Server+Native+Client+11.0')
connection = engine.connect()
So now the first connection is related to the connection of the server, and the second one is the connection the database, the both connection works fine, but together I found the below error:
Traceback (most recent call last):
File "C:/Users/haroo501/PycharmProjects/ToolUpdated/app.py", line 16, in <module>
delete_and_create_db.delete_database()
File "C:\Users\haroo501\PycharmProjects\ToolUpdated\database\delete_and_create_db.py", line 9, in delete_database
connec.crsr.execute(delete_bd_query)
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Cannot drop database "MyNewDatabase" because it is currently in use. (3702) (SQLExecDirectW)')
When trying to run the below code:
from database import connec
def create_db():
create_bd_query = "CREATE DATABASE MyNewDatabase"
connec.crsr.execute(create_bd_query)
def delete_database():
delete_bd_query = "DROP DATABASE MyNewDatabase"
connec.crsr.execute(delete_bd_query
So I think there's a problem to use both connection or something, may be I put the two connection i def and call them when need, but I tried it doesn't work, Any one have an idea how to solve this?
I'm trying to use Python to upload from a Pandas dataframe to a SQL Server table, but I can't successfully create a connection using sqlalchemy. I understand I first need to create an engine object using create_engine(), and create a connection object using engine.connect(), but no string I enter in create_engine() seems to work. I've tried the following:
engine = create_engine('mssql+pyodbc://myServer/myDB')
conn = engine.connect()
and:
engine = create_engine('mssql+pyodbc://Server=myServer;Database=myDB;')
conn = engine.connect()
and:
engine = create_engine('mssql+pyodbc://Driver={SQL Server};Server=myServer;Database=myDB;Trusted_Connection=yes;')
conn = engine.connect()
but all result in the following error:
InterfaceError: (pyodbc.InterfaceError) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
I've also tried:
engine = create_engine('mssql+pyodbc://Driver={SQL Server Native Client 11.0};Server=myServer;Database=myDB;Trusted_Connection=yes;')
conn = engine.connect()
which results in the following error:
DBAPIError: (pyodbc.Error) ('IM010', '[IM010] [Microsoft][ODBC Driver
Manager] Data source name too long (0) (SQLDriverConnect)')
While I can successfully connect using pyodbc like this:
conn = pyodbc.connect('DRIVER={SQL Server};Server=myServer;Database=myDB;Trusted_Connection=yes;')
I can't seem to make this work for sqlalchemy.
Any help would be appreciated.
The solution:
engine = create_engine('mssql+pyodbc://ERRSTSDBP2/ActPri?driver=SQL+Server+Native+Client+11.0')
Thanks to norbeq for getting me most of the way there!
You can try connection like this:
engine = create_engine('mssql+pyodbc://user:password#host:port/myDB')
conn = engine.connect()
While trying to write a pandas' dataframe into sql-server, I get this error:
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][SQL Server Native Client 11.0][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Statement(s) could not be prepared. (8180)")
It seems pandas is looking into sqlite instead of the real database.
It's not a connection problem since I can read from the sql-server with the same connection using pandas.read_sql
The connection has been set using
sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
It's not a database permission problem either since I can write line by line using the same connection parameters as:
cursor = conn.cursor()
cursor.execute('insert into test values (1, 'test', 10)')
conn.commit()
I could just write a loop to instert line by line but I would like to know why to_sql isn't working for me, and I am affraid it won't be as efficient.
Environment:
Python: 2.7
Pandas: 0.20.1
sqlalchemy: 1.1.12
Thanks in advance.
runnable example:
import pandas as pd
from sqlalchemy import create_engine
import urllib
params = urllib.quote_plus("DRIVER={SQL Server Native Client 11.0};SERVER=
<servername>;DATABASE=<databasename>;UID=<username>;PWD=<password>")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
test = pd.DataFrame({'col1':1, 'col2':'test', 'col3':10}, index=[0])
conn=engine.connect().connection
test.to_sql("dbo.test", con=conn, if_exists="append", index=False)
According to the to_sql doc, the con parameter is either an SQLAchemy engine or the legacy DBAPI2 connection (sqlite3). Because you are passing the connection object rather than the SQLAlchemy engine object as the parameter, pandas is inferring that you're passing a DBAPI2 connection, or a SQLite3 connection since its the only one supported. To remedy this, just do:
myeng = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
# Code to create your df
...
# Now write to DB
df.to_sql('table', myeng, index=False)
try this.
good to connect MS SQL server(SQL Authentication) and update data
from sqlalchemy import create_engine
params = urllib.parse.quote_plus(
'DRIVER={ODBC Driver 13 for SQL Server};'+
'SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
#df: pandas.dataframe; mTableName:table name in MS SQL
#warning: discard old table if exists
df.to_sql(mTableName, con=engine, if_exists='replace', index=False)
So I ran into this same thing. I tried looking through the code, couldn't figure out why it wasn't working but it looks like it gets stuck on this call.
pd.io.sql._is_sqlalchemy_connectable(engine)
I found that if I run this first it returns True, but as soon as I run it after running df.to_sql() it returns False. Right now I'm running it before I do the df.to_sql() and it actually works.
Hope this helps.