This is a fairly common question but even using the answers on SO like here but I still can't connect.
When I setup my connection to pyodbc I can connect with the following:
cnxn = pyodbc.connect('DRIVER={SQL Server Native Client 11.0};SERVER=ip,port;DATABASE=db;UID=user;PWD=pass')
cursor = cnxn.cursor()
cursor.execute("some select query")
for row in cursor.fetchall():
print(row)
and it works.
However to do a .read_sql() in pandas I need to connect with sqlalchemy.
I have tried with both hosted connections and pass-through pyodbc connections like the below:
quoted = urllib.parse.quote_plus('DRIVER={SQL Server Native Client 11.0};Server=ip;Database=db;UID=user;PWD=pass;Port=port;')
engine = sqlalchemy.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(quoted))
engine.connect()
I have tried with both SERVER=ip,port format and the separate Port=port parameter like above but still no luck.
The error I'm getting is Login failed for user 'user'. (18456)
Any help is much appreciated.
I assume that you want to create a DataFrame so when you have a cnxn you can pass it to Pandas read_sql_query function.
Example:
cnxn = pyodbc.connect('your connection string')
query = 'some query'
df = pandas.read_sql_query(query, conn)
Related
First I query my cosmos data and put it into a dataframe. Then I connect to my database and try using to_sql:
params = urllib.parse.quote_plus(r'Driver=SQL Server;Server={SERVERNAME},1433;Database=cosmosTest;Trusted_Connection=yes;TrustServerCertificate=no;Connection Timeout=0;')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str,echo=True)
conn = engine.connect()
df.to_sql('Document', conn, if_exists='replace', index = False)
When I try and run my script I run into this error message
('HY004', '[HY004] [Microsoft][ODBC SQL Server Driver]Invalid SQL data type (0) (SQLBindParameter)')
I'm not sure if I'm receiving this error due to something I'm doing wrong or if the data I'm querying from cosmos doesn't play well with SQL Server. I've checked a couple other posts about this issue, but they didn't really pertain to me. Any suggestions on what I can try here?
I am trying to connect to HANA in order to pull some metadata in a pandas dataframe. There are lots of mixed approaches and I couldn't find anything concrete.
All I have is:
username
password
serverip
servername
and table names.
The admin has provided all the read access to the required account for the specific tables.
What is the quickest way to get this done? I do not have the option of installing anything on SAPs site.
I have tried the below snippets but I get the error 'target machine actively refused it' and to debug at SAPs end is a lost cause. Thank you in advance.
import pyhdb
connection = pyhdb.connect(
host="123.com",
port=123,
user="user",
password="pswrd"
)
cursor = connection.cursor()
cursor.execute("SELECT * FROM Tablename")
cursor.fetchone()
connection.close()
and
from hdbcli import dbapi
conn = dbapi.connect(
address="123.com",
port=123,
user="user",
password="pswrd"
)
cursor = conn.cursor()
Given your server address and port examples, I'm not sure you got the right idea for how to connect to a HANA database.
Since you want to use pandas it is probably a good idea to have a look at the SAP HANA Machine Learning library for Python.
Check the tutorial blog post for this:
https://blogs.sap.com/2019/11/05/hands-on-tutorial-machine-learning-push-down-to-sap-hana-with-python/
To do any of this, there is no need to install or debug anything on the HANA system.
To connect to hana DB :
from hdbcli import dbapi
conn = dbapi.connect(
address="serverhost",
port=39015,
user="UserName",
password="Password",
databasename='DBNAME'
)
Make sure you enter the correct port number
To get the sql results in pandas dataframe:
query = 'select * from table'
df = pd.read_sql_query(query, conn)
df.head()
I'm trying to follow the method for inserting a Panda data frame into SQL Server that is mentioned here as it appears to be the fastest way to import lots of rows.
However I am struggling with figuring out the connection parameter.
I am not using DSN , I have a server name, a database name, and using trusted connection (i.e. windows login).
import sqlalchemy
import urllib
server = 'MYServer'
db = 'MyDB'
cxn_str = "DRIVER={SQL Server Native Client 11.0};SERVER=" + server +",1433;DATABASE="+db+";Trusted_Connection='Yes'"
#cxn_str = "Trusted_Connection='Yes',Driver='{ODBC Driver 13 for SQL Server}',Server="+server+",Database="+db
params = urllib.parse.quote_plus(cxn_str)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
conn = engine.connect().connection
cursor = conn.cursor()
I'm just not sure what the correct way to specify my connection string is. Any suggestions?
I have been working with pandas and SQL server for a while and the fastest way I found to insert a lot of data in a table was in this way:
You can create a temporary CSV using:
df.to_csv('new_file_name.csv', sep=',', encoding='utf-8')
Then use pyobdc and BULK INSERT Transact-SQL:
import pyodbc
conn = pyodbc.connect(DRIVER='{SQL Server}', Server='server_name', Database='Database_name', trusted_connection='yes')
cur = conn.cursor()
cur.execute("""BULK INSERT table_name
FROM 'C:\\Users\\folders path\\new_file_name.csv'
WITH
(
CODEPAGE = 'ACP',
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)""")
conn.commit()
cur.close()
conn.close()
Then you can delete the file:
import os
os.remove('new_file_name.csv')
It was a second to charge a lot of data at once into SQL Server. I hope this gives you an idea.
Note: don't forget to have a field for the index. It was my mistake when I started to use this lol.
Connection string parameter values should not be enclosed in quotes so you should use Trusted_Connection=Yes instead of Trusted_Connection='Yes'.
While trying to write a pandas' dataframe into sql-server, I get this error:
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][SQL Server Native Client 11.0][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Statement(s) could not be prepared. (8180)")
It seems pandas is looking into sqlite instead of the real database.
It's not a connection problem since I can read from the sql-server with the same connection using pandas.read_sql
The connection has been set using
sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
It's not a database permission problem either since I can write line by line using the same connection parameters as:
cursor = conn.cursor()
cursor.execute('insert into test values (1, 'test', 10)')
conn.commit()
I could just write a loop to instert line by line but I would like to know why to_sql isn't working for me, and I am affraid it won't be as efficient.
Environment:
Python: 2.7
Pandas: 0.20.1
sqlalchemy: 1.1.12
Thanks in advance.
runnable example:
import pandas as pd
from sqlalchemy import create_engine
import urllib
params = urllib.quote_plus("DRIVER={SQL Server Native Client 11.0};SERVER=
<servername>;DATABASE=<databasename>;UID=<username>;PWD=<password>")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
test = pd.DataFrame({'col1':1, 'col2':'test', 'col3':10}, index=[0])
conn=engine.connect().connection
test.to_sql("dbo.test", con=conn, if_exists="append", index=False)
According to the to_sql doc, the con parameter is either an SQLAchemy engine or the legacy DBAPI2 connection (sqlite3). Because you are passing the connection object rather than the SQLAlchemy engine object as the parameter, pandas is inferring that you're passing a DBAPI2 connection, or a SQLite3 connection since its the only one supported. To remedy this, just do:
myeng = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
# Code to create your df
...
# Now write to DB
df.to_sql('table', myeng, index=False)
try this.
good to connect MS SQL server(SQL Authentication) and update data
from sqlalchemy import create_engine
params = urllib.parse.quote_plus(
'DRIVER={ODBC Driver 13 for SQL Server};'+
'SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
#df: pandas.dataframe; mTableName:table name in MS SQL
#warning: discard old table if exists
df.to_sql(mTableName, con=engine, if_exists='replace', index=False)
So I ran into this same thing. I tried looking through the code, couldn't figure out why it wasn't working but it looks like it gets stuck on this call.
pd.io.sql._is_sqlalchemy_connectable(engine)
I found that if I run this first it returns True, but as soon as I run it after running df.to_sql() it returns False. Right now I'm running it before I do the df.to_sql() and it actually works.
Hope this helps.
I'm new to Python + Flask + Flask Appbuilder but I am a professional Java developer. I've been working on a small app that I initially used SqlLite and now I want to move into SQL Server, which will be the production database.
I can't seem to get the connection right.
I have tried using a DSN but I get an error message indicating there is a mismatch between the driver and something else (Python?). The searches on this error seem to indicate the driver is 32 bit and Python is 64. Still I can't get that working so I thought I'd try to connect directly. I'd prefer not using a DSN anyway. I've searched the web and can't find an example that works for me.
I have imported pyodbc. This is the current way I'm trying to connect:
params = urllib.quote_plus("DRIVER={SQL Server};SERVER=devsql07:1433;DATABASE=DevOpsSnippets;UID=<user>;PWD=<password>")
SQLALCHEMY_DATABASE_URI = "mssql+pyodbc:///?odbc_connect=%s" % params
This produces the following error message:
2016-02-17 07:11:38,115:ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: (pyodbc.Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]Invalid connection. (14) (SQLDriverConnect); [01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (ParseConnectParams()). (14)')
Can anyone help me get this connection correct?
I really appreciate any help.
if you are using pyodbc you should be able to connect this way
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=yourServer;DATABASE=yourDatabase;UID=;PWD=')
#putting to use
SQL = "select Field1, Field2 from someTable"
cursor = cnxn.cursor()
cursor.execute(SQL)
row = cursor.fetchall()
for r in row:
print r[0] #field1
print r[1] #field2
The port should be specified through a comma. Specify the connection string as
DRIVER={SQL Server};SERVER=devsql07,1433;DATABASE.....