Develop Reference

How to specify a search path with SQL Alchemy and pg8000?

I'm trying to connect to a postgres db using SQL Alchemy and the pg8000 driver. I'd like to specify a search path for this connection. With the Psycopg driver, I could do this by doing something like
engine = create_engine(
'postgresql+psycopg2://dbuser#dbhost:5432/dbname',
connect_args={'options': '-csearch_path={}'.format(dbschema)})
However, this does not work for the pg8000 driver. Is there a good way to do this?

You can use pg8000 pretty much in the same way as psycopg2, just need to swap scheme from postgresql+psycopg2 to postgresql+pg8000.
The full connection string definition is in the SQLAlchemy pg8000 docs:
postgresql+pg8000://user:password#host:port/dbname[?key=value&key=value...]
But while psycopg2.connect will pass kwargs to the server (like options and its content), pg8000.connect will not, so there is no setting search_path with pg8000.

The SQLAlchemy docs describe how to do this. For example:
from sqlalchemy import create_engine, event, text
engine = create_engine("postgresql+pg8000://postgres:postgres#localhost/postgres")
#event.listens_for(engine, "connect", insert=True)
def set_search_path(dbapi_connection, connection_record):
existing_autocommit = dbapi_connection.autocommit
dbapi_connection.autocommit = True
cursor = dbapi_connection.cursor()
cursor.execute("SET SESSION search_path='myschema'")
cursor.close()
dbapi_connection.autocommit = existing_autocommit
with engine.connect() as connection:
result = connection.execute(text("SHOW search_path"))
for row in result:
print(row)
However, as it says in the docs:
SQLAlchemy is generally organized around the concept of keeping this
variable at its default value of public

Python to MS SQL Error: Error when connecting to SQL using sqlalchemy.create_engine() using pypyodbc

Scenario:
I am trying to Convert the SQL output directly to Table using dataframe.to_sql, so for that i am using sqlalchemy.create_engine() and its throwing error when trying to createngine()
sqlchemyparams= urllib.parse.quote_plus(ConnectionString)
sqlchemy_conn_str = 'mssql+pypyodbc:///?odbc_connect={}'.format(sqlchemyparams)
engine_azure = sqlalchemy.create_engine(sqlchemy_conn_str,echo=True,fast_executemany =
True, poolclass=NullPool)
df_top_features.to_sql('Topdata', engine_azure,schema='dbo', index = False, if_exists =
'replace')
2.It will work fine if i use:pyodbc
sqlchemy_conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(sqlchemyparams)
So is there any way i can using pypyodbc in sqlchem_conn_str

SQLAlchemy does not have a pypyodbc driver defined for the mssql dialect, so
mssql+pypyodbc:// …
simply will not work. There may be some way to "fool" your code into using pypyodbc when you specify mssql+pyodbc://, similar to doing
import pypyodbc as pyodbc
in plain Python, but it is not recommended.
In cases where pyodbc cannot be used, the recommended alternative would be mssql+pymssql://.

Here's what I do
import sqlalchemy as sa
from sqlalchemy import create_engine, event
from sqlalchemy.engine.url import URL
Then create varaibles to holder the server, database, username and password and pass it to...
params = urllib.parse.quote_plus("DRIVER={SQL Server};"
"SERVER="+server+";"
"DATABASE="+database+";"
"UID="+username+";"
"PWD="+password+";")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
then upload data to sql using.
dfc.to_sql('jobber',con=engine,index=False, if_exists='append')
Using https://www.dataquest.io/blog/sql-insert-tutorial/ as a source.

Why is the SQL server DB engine not connecting with SQLAlchemy engine?

I am trying to connect to the SQL server database on python platform using SqlAlchemy. I am using windows authentication to connect to my the SQL server. On connecting the server the SqlAlchemy engine is throwing an error:
Below is the code I have implemented:
import os
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
import pyodbc
Driver Server Name Instance Database
DATABASE_URL='mssql+pyodbc://DESKTOP-N32LSOV\PRANAV/AdventureworksDW2016CTP3?trusted_connection=yes'
Engine = create_engine(DATABASE_URL)
cn = Engine.connect()
When the above code is run, this error is produced:
Error:sqlalchemy.exc.InterfaceError: (pyodbc.InterfaceError) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)') (Background on this error at: http://sqlalche.me/e/rvf5)
I tried using pymssql driver inplace of pyodbc driver but still the error persists. On contrary when i try to connect to the using the below syntax it connects. I guess i am missing some attribute in the mssql url.
pyodbc.connect(r'Driver={SQL Server};Server=DESKTOP-N32LSOV\PRANAV;Database=master;Trusted_Connection=yes;')
Any help will be appreciated.

You need to specify both that you want to use ODBC and what ODBC driver to use.
engine = sqlalchemy.create_engine('mssql+pyodbc://localhost/Sandbox?driver=SQL+Server+Native+Client+11.0')
If you add the driver= part to your database url, it should work.

If all else fails, I would try using the creator argument to create_engine (documentation):
def creator():
return pyodbc.connect(r'Driver={SQL Server};Server=DESKTOP-N32LSOV\PRANAV;Database=master;Trusted_Connection=yes;')
Engine = create_engine('mssql://', creator=creator)
Using creator= bypasses all connection parameters specified in the URL, so you should only pass information to specify the DB dialect in the URL.
Looking at the mssql+pyodbc dialect/driver documentation, there is also the ?odbc_connect option:
import urllib.parse
CONNECTION_STRING = r'Driver={SQL Server};Server=DESKTOP-N32LSOV\PRANAV;Database=master;Trusted_Connection=yes;'
Engine = create_engine('mssql+pyodbc:///?odbc_connect=' + urllib.parse.quote_plus(CONNECTION_STRING))

Python: SQL Alchemy Create Engine Syntax Issues

Let's say I have the following connection information for a MSSQL server:
'Driver={SQL Server};'
'Server=VCAB18RPACRGZ12\GNRSRZ11,1414;'
'Database=sampleDB;'
'uid=sampleID;'
'pwd=samplePW'
I want to write a python dataframe to the MSSQL server as a table. I have the following code:
from sqlalchemy import create_engine
connection = create_engine('mssql+pyodbc://sampleID:samplePW#myhost:VCAB18RPACRGZ12\GNRSRZ11,1414/sampleDB?driver=SQL+Server+Native+Client+10.0')
My above connection code is erroring out. I'm not sure exactly where my connection information is supposed to go in the create_engine statement.
This is my error ...
ValueError: invalid literal for int() with base 10:
'VCAB18RPACRGZ12\GNRSRZ11,1414'

Your Server Address is not correct.
If 1414 is the port#, you should use ":" instead of ",".
The SQLAlchemy uses pyodbc as the default DBAPI. pymssql is also available.
Below is the connection string sample:
# pyodbc -DSN
engine = create_engine('mssql+pyodbc://scott:tiger#mydsn')
# pymssql
engine = create_engine('mssql+pymssql://scott:tiger#hostname:port/dbname')
# pyodbc -DSN Less connection
from sqlalchemy import create_engine
#assumes driver name=[SQL+Server+Native+Client+10.0]
#engine = create_engine('mssql+pyodbc://username:password#hostname:port/databasename?driver=SQL+Server+Native+Client+10.0')
engine = create_engine(r'mssql+pyodbc://sampleID:samplePW#VCAB18RPACRGZ12\GNRSRZ11:1414/sampleDB?driver=SQL+Server+Native+Client+10.0')
print engine

How do I connect to SQL Server via sqlalchemy using Windows Authentication?

sqlalchemy, a db connection module for Python, uses SQL Authentication (database-defined user accounts) by default. If you want to use your Windows (domain or local) credentials to authenticate to the SQL Server, the connection string must be changed.
By default, as defined by sqlalchemy, the connection string to connect to the SQL Server is as follows:
sqlalchemy.create_engine('mssql://*username*:*password*#*server_name*/*database_name*')
This, if used using your Windows credentials, would throw an error similar to this:
sqlalchemy.exc.DBAPIError: (Error) ('28000', "[28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for us
er '***S\\username'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for us
er '***S\\username'. (18456)") None None
In this error message, the code 18456 identifies the error message thrown by the SQL Server itself. This error signifies that the credentials are incorrect.

In order to use Windows Authentication with sqlalchemy and mssql, the following connection string is required:
ODBC Driver:
engine = sqlalchemy.create_engine('mssql://*server_name*/*database_name*?trusted_connection=yes')
SQL Express Instance:
engine = sqlalchemy.create_engine('mssql://*server_name*\\SQLEXPRESS/*database_name*?trusted_connection=yes')

If you're using a trusted connection/AD and not using username/password, or otherwise see the following:
SAWarning: No driver name specified; this is expected by PyODBC when using >DSN-less connections
"No driver name specified; "
Then this method should work:
from sqlalchemy import create_engine
server = <your_server_name>
database = <your_database_name>
engine = create_engine('mssql+pyodbc://' + server + '/' + database + '?trusted_connection=yes&driver=ODBC+Driver+13+for+SQL+Server')

A more recent response if you want to connect to the MSSQL DB from a different user than the one you're logged with on Windows. It works as well if you are connecting from a Linux machine with FreeTDS installed.
The following worked for me from both Windows 10 and Ubuntu 18.04 using Python 3.6 & 3.7:
import getpass
from sqlalchemy import create_engine
password = getpass.getpass()
eng_str = fr'mssql+pymssql://{domain}\{username}:{password}#{hostip}/{db}'
engine = create_engine(eng_str)
What changed was to add the Windows domain before \username.
You'll need to install the pymssql package.

Create Your SqlAlchemy Connection URL      From Your pyodbc Connection String      OR Your Known Connection Parameters
I found all the other answers to be educational, and I found the SqlAlchemy Docs on connection strings helpful too, but I kept failing to connect to MS SQL Server Express 19 where I was using no username or password and trusted_connection='yes' (just doing development at this point).
Then I found THIS method in the SqlAlchemy Docs on Connection URLs built from a pyodbc connection string (or just a connection string), which is also built from known connection parameters (i.e. this can simply be thought of as a connection string that is not necessarily used in pyodbc). Since I knew my pyodbc connection string was working, this seemed like it would work for me, and it did!
This method takes the guesswork out of creating the correct format for what you feed to the SqlAlchemy create_engine method. If you know your connection parameters, you put those into a simple string per the documentation exemplified by the code below, and the create method in the URL class of the sqlalchemy.engine module does the correct formatting for you.
The example code below runs as is and assumes a database named master and an existing table named table_one with the schema shown below. Also, I am using pandas to import my table data. Otherwise, we'd want to use a context manager to manage connecting to the database and then closing the connection like HERE in the SqlAlchemy docs.
import pandas as pd
import sqlalchemy
from sqlalchemy.engine import URL
# table_one dictionary:
table_one = {'name': 'table_one',
'columns': ['ident int IDENTITY(1,1) PRIMARY KEY',
'value_1 int NOT NULL',
'value_2 int NOT NULL']}
# pyodbc stuff for MS SQL Server Express
driver='{SQL Server}'
server='localhost\SQLEXPRESS'
database='master'
trusted_connection='yes'
# pyodbc connection string
connection_string = f'DRIVER={driver};SERVER={server};'
connection_string += f'DATABASE={database};'
connection_string += f'TRUSTED_CONNECTION={trusted_connection}'
# create sqlalchemy engine connection URL
connection_url = URL.create(
"mssql+pyodbc", query={"odbc_connect": connection_string})
""" more code not shown that uses pyodbc without sqlalchemy """
engine = sqlalchemy.create_engine(connection_url)
d = {'value_1': [1, 2], 'value_2': [3, 4]}
df = pd.DataFrame(data=d)
df.to_sql('table_one', engine, if_exists="append", index=False)
Update
Let's say you've installed SQL Server Express on your linux machine. You can use the following commands to make sure you're using the correct strings for the following:
For the driver: odbcinst -q -d
For the server: sqlcmd -S localhost -U <username> -P <password> -Q 'select ##SERVERNAME'

pyodbc
I think that you need to put:
"+pyodbc" after mssql
try this:
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
It works for me
Luck!

If you are attempting to connect:
DNS-less
Windows Authentication for a server not locally hosted.
Without using ODBC connections.
Try the following:
import sqlalchemy
engine = sqlalchemy.create_engine('mssql+pyodbc://' + server + '/' + database + '?trusted_connection=yes&driver=SQL+Server')
This avoids using ODBC connections and thus avoids pyobdc interface errors from DPAPI2 vs DBAPI3 conflicts.
I would recommend using the URL creation tool instead of creating the url from scratch.
connection_url = sqlalchemy.engine.URL.create("mssql+pyodbc",database=databasename, host=servername, query = {'driver':'SQL Server'})
engine = sqlalchemy.create_engine(connection_url)
See this link for creating a connection string with SQL Server Authentication (non-domain, uses username and password)