sqlalchemy, a db connection module for Python, uses SQL Authentication (database-defined user accounts) by default. If you want to use your Windows (domain or local) credentials to authenticate to the SQL Server, the connection string must be changed.
By default, as defined by sqlalchemy, the connection string to connect to the SQL Server is as follows:
sqlalchemy.create_engine('mssql://*username*:*password*#*server_name*/*database_name*')
This, if used using your Windows credentials, would throw an error similar to this:
sqlalchemy.exc.DBAPIError: (Error) ('28000', "[28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for us
er '***S\\username'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for us
er '***S\\username'. (18456)") None None
In this error message, the code 18456 identifies the error message thrown by the SQL Server itself. This error signifies that the credentials are incorrect.
In order to use Windows Authentication with sqlalchemy and mssql, the following connection string is required:
ODBC Driver:
engine = sqlalchemy.create_engine('mssql://*server_name*/*database_name*?trusted_connection=yes')
SQL Express Instance:
engine = sqlalchemy.create_engine('mssql://*server_name*\\SQLEXPRESS/*database_name*?trusted_connection=yes')
If you're using a trusted connection/AD and not using username/password, or otherwise see the following:
SAWarning: No driver name specified; this is expected by PyODBC when using >DSN-less connections
"No driver name specified; "
Then this method should work:
from sqlalchemy import create_engine
server = <your_server_name>
database = <your_database_name>
engine = create_engine('mssql+pyodbc://' + server + '/' + database + '?trusted_connection=yes&driver=ODBC+Driver+13+for+SQL+Server')
A more recent response if you want to connect to the MSSQL DB from a different user than the one you're logged with on Windows. It works as well if you are connecting from a Linux machine with FreeTDS installed.
The following worked for me from both Windows 10 and Ubuntu 18.04 using Python 3.6 & 3.7:
import getpass
from sqlalchemy import create_engine
password = getpass.getpass()
eng_str = fr'mssql+pymssql://{domain}\{username}:{password}#{hostip}/{db}'
engine = create_engine(eng_str)
What changed was to add the Windows domain before \username.
You'll need to install the pymssql package.
Create Your SqlAlchemy Connection URL From Your pyodbc Connection String OR Your Known Connection Parameters
I found all the other answers to be educational, and I found the SqlAlchemy Docs on connection strings helpful too, but I kept failing to connect to MS SQL Server Express 19 where I was using no username or password and trusted_connection='yes' (just doing development at this point).
Then I found THIS method in the SqlAlchemy Docs on Connection URLs built from a pyodbc connection string (or just a connection string), which is also built from known connection parameters (i.e. this can simply be thought of as a connection string that is not necessarily used in pyodbc). Since I knew my pyodbc connection string was working, this seemed like it would work for me, and it did!
This method takes the guesswork out of creating the correct format for what you feed to the SqlAlchemy create_engine method. If you know your connection parameters, you put those into a simple string per the documentation exemplified by the code below, and the create method in the URL class of the sqlalchemy.engine module does the correct formatting for you.
The example code below runs as is and assumes a database named master and an existing table named table_one with the schema shown below. Also, I am using pandas to import my table data. Otherwise, we'd want to use a context manager to manage connecting to the database and then closing the connection like HERE in the SqlAlchemy docs.
import pandas as pd
import sqlalchemy
from sqlalchemy.engine import URL
# table_one dictionary:
table_one = {'name': 'table_one',
'columns': ['ident int IDENTITY(1,1) PRIMARY KEY',
'value_1 int NOT NULL',
'value_2 int NOT NULL']}
# pyodbc stuff for MS SQL Server Express
driver='{SQL Server}'
server='localhost\SQLEXPRESS'
database='master'
trusted_connection='yes'
# pyodbc connection string
connection_string = f'DRIVER={driver};SERVER={server};'
connection_string += f'DATABASE={database};'
connection_string += f'TRUSTED_CONNECTION={trusted_connection}'
# create sqlalchemy engine connection URL
connection_url = URL.create(
"mssql+pyodbc", query={"odbc_connect": connection_string})
""" more code not shown that uses pyodbc without sqlalchemy """
engine = sqlalchemy.create_engine(connection_url)
d = {'value_1': [1, 2], 'value_2': [3, 4]}
df = pd.DataFrame(data=d)
df.to_sql('table_one', engine, if_exists="append", index=False)
Update
Let's say you've installed SQL Server Express on your linux machine. You can use the following commands to make sure you're using the correct strings for the following:
For the driver: odbcinst -q -d
For the server: sqlcmd -S localhost -U <username> -P <password> -Q 'select ##SERVERNAME'
pyodbc
I think that you need to put:
"+pyodbc" after mssql
try this:
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
It works for me
Luck!
If you are attempting to connect:
DNS-less
Windows Authentication for a server not locally hosted.
Without using ODBC connections.
Try the following:
import sqlalchemy
engine = sqlalchemy.create_engine('mssql+pyodbc://' + server + '/' + database + '?trusted_connection=yes&driver=SQL+Server')
This avoids using ODBC connections and thus avoids pyobdc interface errors from DPAPI2 vs DBAPI3 conflicts.
I would recommend using the URL creation tool instead of creating the url from scratch.
connection_url = sqlalchemy.engine.URL.create("mssql+pyodbc",database=databasename, host=servername, query = {'driver':'SQL Server'})
engine = sqlalchemy.create_engine(connection_url)
See this link for creating a connection string with SQL Server Authentication (non-domain, uses username and password)
Related
I've already had a working connection through ODBC using Cloudera ODBC Driver for Apache Hive, where I had my DSN set and all I needed was to call pyodbc.connect(f"DSN={mydsn}", autocommit=True).
Since I'm planning to use pandas on the query result, I've read that SQLAlchemy is the preferred choice and I'd like to avoid warnings resulting from other ways of connection. My DSN for Hive was using Zookeeper and "Hosts" field was filled in the form of host1:2181,host2:2181,host3:2181. I'm trying to connect to these 3 hosts and I've tried changing connection url in analogous way to the one provided in here, but I got invalid literal for int() with base 10: '2181,host2:2181,host3:2181 etc.
from sqlalchemy import create_engine
query = """SELECT TOP 10 * from eb.mobile_sa"""
conn_url = f'hive://{UID}#host1:2181,host2:2181,host3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2'
engine = create_engine(conn_url)
with engine.connect() as conn:
df = pd.read_sql(query, conn)
I found kazoo module that is said to be Zookeeper implementation in Python, but when I tried the very first lines from Basic Usage and just 1 host:
from kazoo.client import KazooClient
zk = KazooClient(hosts = "host1:2181", read_only=True)
zk.start()
I got a lot of lines of Connection dropped: socket connection error
How can I correctly connect to multiple hosts in hive?
I am trying to use Python to connect to a SQL database by using Window authentication. I looked at some of the posts here (e.g., here), but the suggested methods didn't seem to work.
For example, I used the following code:
cnxn = pyodbc.connect(driver='{SQL Server Native Client 11.0}',
server='SERVERNAME',
database='DATABASENAME',
trusted_connection='yes')
But I got the following error:
Error: ('28000', "[28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]
Login failed for user 'DOMAIN\\username'. (18456) (SQLDriverConnect); [28000] [Microsoft]
[SQL Server Native Client 11.0][SQL Server]Login failed for user 'DOMAIN\\username'.
(18456)")
(Note that I replaced the actual domain name and user name with DOMAIN and username respectively, in the error message above.)
I also tried using my UID and PWD, which led to the same error.
Lastly, I tried to change the service account by following the suggestion from the link above, but on my computer, there was no Log On tab when I went to the Properties of services.msc.
I wonder what I did wrong and how I can fix the problem.
Connecting from a Windows machine:
With Microsoft's ODBC drivers for SQL Server, Trusted_connection=yes tells the driver to use "Windows Authentication" and your script will attempt to log in to the SQL Server using the Windows credentials of the user running the script. UID and PWD cannot be used to supply alternative Windows credentials in the connection string, so if you need to connect as some other Windows user you will need to use Windows' RUNAS command to run the Python script as that other user..
If you want to use "SQL Server Authentication" with a specific SQL Server login specified by UID and PWD then use Trusted_connection=no.
Connecting from a non-Windows machine:
If you need to connect from a non-Windows machine and the SQL Server is configured to only use "Windows authentication" then Microsoft's ODBC drivers for SQL Server will require you to use Kerberos. Alternatively, you can use FreeTDS ODBC, specifying UID, PWD, and DOMAIN in the connection string, provided that the SQL Server instance is configured to support the older NTLM authentication protocol.
I tried everything and this is what eventually worked for me:
import pyodbc
driver= '{SQL Server Native Client 11.0}'
cnxn = pyodbc.connect(
Trusted_Connection='Yes',
Driver='{ODBC Driver 11 for SQL Server}',
Server='MyServer,1433',
Database='MyDB'
)
Try this cxn string:
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=localhost;PORT=1433;DATABASE=testdb;UID=me;PWD=pass')
http://mkleehammer.github.io/pyodbc/
I had similar issue while connecting to the default database (MSSQLSERVER). If you are connecting to the default database, please remove the
database='DATABASENAME',
line from the connection parameters section and retry.
Cheers,
Deepak
The first option works if your credentials have been stored using the command prompt. The other option is giving the credentials (UId, Psw) in the connection.
The following worked for me:
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=yourServer;DATABASE=yourDatabase;UID=yourUsername;PWD=yourPassword')
import pyodbc #For python3 MSSQL
cnxn = pyodbc.connect("Driver={SQL Server};" #For Connection
"Server=192.168.0.***;"
"PORT=1433;"
"Database=***********;"
"UID=****;"
"PWD=********;")
cursor = cnxn.cursor() #Cursor Establishment
cursor.execute('select site_id from tableName') #Execute Query
rs = cursor.fetchall()
print(rs)
A slightly different use case than the OP, but for those interested it is possible to connect to a MS SQL Server database using Windows Authentication for a different user account than the one logged in.
This can be achieved using the python jaydebeapi module with the JDBC JTDS driver. See my answer here for details.
Note that you may need to change the authentication mechanism. For example, my database is using ADP. So my connection looks like this
pyodbc.connect(
Trusted_Connection='No',
Authentication='ActiveDirectoryPassword',
UID=username,
PWD=password,
Driver=driver,
Server=server,
Database=database)
Read more here
Trusted_connection=no did not helped me. When i removed entire line and added UID, PWD parameter it worked. My takeaway from this is remove
I am trying to connect a python terminal to a SQL Server database on AWS RDS.
I've looked through Microsoft's documentation on how to do it here, and I've seen aplethora of different questions asked here on Stack Overflow, however I think my unique problem is mapping the connection process to specific aspects of the RDS environment, and I'm unsure if my environment is setup correctly, and I can't find a question that maps directly to the issues I'm trying to research.
WHAT I HAVE RIGHT NOW:
import pyodbc as pydb
connection = pydb.connect('DRIVER={SQL Server};PORT=1433;SERVER=aws-database-endpoint;UID=instance-master-username;PWD=instance-password;DATABASE=db-instance-id')
print("Connecting....")
connection.close()
And getting the following traceback:
InterfaceError: ('28000', "[28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'admin'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC SQL Server Driver]Invalid connection string attribute (0); [28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'admin'. (18456); [28000] [Microsoft][ODBC SQL Server Driver]Invalid connection string attribute (0)")
To provide more detail, here's a picture of what my database setup currently looks like:
For port, and server, I'm getting the info from here:
For UID I'm going to RDS --> Databases, clicking on my instance and then going to Configuration, and I'm entering the Master Username, which is admin, as can be seen here:
And for my DATABASE value, I'm at the same spot, and using the database instance-id, which can be seen here:
I've created a rule that allows my IP address for incoming TCP traffic via VPC resource groups here:
What I've also tried:
using my account ID for the UID argument
using my AWS username for the UID argument
database is publicly accessible
I have tried logging in both as a root user, and as an IAM user with administrator privileges
I can't tell if I'm making a clerical mistake or if there is a larger admin issue that I'm not identifying.
Thank you.
use the sample connection below, note there is not port if its the default 1433
cnxn = pyodbc.connect('DRIVER={/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.3.so.1.1};SERVER='+server+';DATABASE='+database+';uid='+username+';pwd='+ password)
If you are running the python script from your local machine (wsl, linux box or ec2) make sure the msodbc drivers are correctly installed
You can also try this, it was easier than pyodbc for me:
import pymssql
connection={
'host': '',
'username': '',
'password': '',
'db': ''
}
con=pymssql.connect(connection['host'],connection['username'],connection['password'],connection['db'])
cursor=con.cursor()
Running into this problem myself I found that AWS RDS doesn't actually give you an option to name your database within the instance in SQL server like it does with PostreSQL when instantiating or afterwords and rather than giving it the name of the instance it leaves this field blank. You can confirm this by looking at the configuration tab within the instance summary in the AWS dashboard, under DB name there is simply a dash to indicate a blank field. I was able to solve the problem by just removing the database name field from my connection string. So the string the original example would look like this:
import pyodbc as pydb
connection = pydb.connect('DRIVER={SQL Server};PORT=1433;SERVER=aws-database-endpoint;UID=instance-master-username;PWD=instance-password;')
Try to bring each component to a variable
import pyodbc
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
#Sample select query
query = 'SELECT....'
df = pd.sql_read(query, cnxn)
print(df)
I would like to connect to an Oracle database with python through pyodbc. I have installed oracle driver and I tried the following script:
import pyodbc
connectString = """
DRIVER={Oracle in OraClient12Home1};
SERVER=some_oracle_db.com:1521;
SID=oracle_test;
UID=user_name;
PWD=user_pass
"""
cnxn = pyodbc.connect(connectString)
I got the following error message:
cnxn = pyodbc.connect(connectString)
Error: ('HY000', '[HY000] [Oracle][ODBC][Ora]ORA-12560: TNS:protocol adapter error\n (12560) (SQLDriverConnect)')
What's wrong here?
Why keyword DBQ works and SID/Service Name does not, see the section 21.4.1 Format of the Connection String in Oracle 12c documentation.
https://docs.oracle.com/database/121/ADFNS/adfns_odbc.htm#ADFNS1183/
or google keywords for odbc oracle 12c
Looks Like your missing a PORT
Try this way
NOTE:
Depending on your Server the syntax can be different this will work for Windows without DSN using an SQL Server Driver.
connectString = pyodbc.connect('DRIVER={SQL Server};SERVER=localhost;PORT=1433;DATABASE=testdb;UID=me;PWD=pass')
This is the connection, you still need a cursor and to use execute along with an SQL Statement..
You have to specify server or hostname (or IP address in connection string for your database server is running.
So close...
connectString = """
DRIVER={Oracle in OraClient12Home1};
SERVER=some_oracle_db.com:1521;
DBQ=oracle_test;
Uid=user_name;
Pwd=user_name
"""
I replaced SID with DBQ
Using SQLAlchemy, an Engine object is created like this:
from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost/mydb")
Accessing engine fails if the database specified in the argument to create_engine (in this case, mydb) does not exist. Is it possible to tell SQLAlchemy to create a new database if the specified database doesn't exist?
SQLAlchemy-Utils provides custom data types and various utility functions for SQLAlchemy. You can install the most recent official version using pip:
pip install sqlalchemy-utils
The database helpers include a create_database function:
from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database
engine = create_engine("postgres://localhost/mydb")
if not database_exists(engine.url):
create_database(engine.url)
print(database_exists(engine.url))
On postgres, three databases are normally present by default. If you are able to connect as a superuser (eg, the postgres role), then you can connect to the postgres or template1 databases. The default pg_hba.conf permits only the unix user named postgres to use the postgres role, so the simplest thing is to just become that user. At any rate, create an engine as usual with a user that has the permissions to create a database:
>>> engine = sqlalchemy.create_engine("postgres://postgres#/postgres")
You cannot use engine.execute() however, because postgres does not allow you to create databases inside transactions, and sqlalchemy always tries to run queries in a transaction. To get around this, get the underlying connection from the engine:
>>> conn = engine.connect()
But the connection will still be inside a transaction, so you have to end the open transaction with a commit:
>>> conn.execute("commit")
And you can then proceed to create the database using the proper PostgreSQL command for it.
>>> conn.execute("create database test")
>>> conn.close()
It's possible to avoid manual transaction management while creating database by providing isolation_level='AUTOCOMMIT' to create_engine function:
import sqlalchemy
with sqlalchemy.create_engine(
'postgresql:///postgres',
isolation_level='AUTOCOMMIT'
).connect() as connection:
connection.execute('CREATE DATABASE my_database')
Also if you are not sure that database doesn't exist there is a way to ignore database creation error due to existence by suppressing sqlalchemy.exc.ProgrammingError exception:
import contextlib
import sqlalchemy.exc
with contextlib.suppress(sqlalchemy.exc.ProgrammingError):
# creating database as above
Extending the accepted answer using with yields:
from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost")
NEW_DB_NAME = 'database_name'
with engine.connect() as conn:
conn.execute("commit")
# Do not substitute user-supplied database names here.
conn.execute(f"CREATE DATABASE {NEW_DB_NAME}")
Please note that I couldn't get the above suggestions with database_exists because whenever I check if the database exists using if not database_exists(engine.url): I get this error:
InterfaceError('(pyodbc.InterfaceError) (\'28000\', u\'[28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for
user \\'myUser\\'. (18456) (SQLDriverConnect); [28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open
database "MY_DATABASE" requested by the login. The login failed.
(4060); [28000] [Microsoft][SQL Server Native Client 11.0][SQL
Server]Login failed for user \\'myUser\\'. (18456); [28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open
database "MY_DATABASE" requested by the login. The login failed.
(4060)\')',)
Also contextlib/suppress was not working and I'm not using postgres so I ended up doing this to ignore the exception if the database happens to already exist with SQL Server:
import logging
import sqlalchemy
logging.basicConfig(filename='app.log', format='%(asctime)s-%(levelname)s-%(message)s', level=logging.DEBUG)
engine = create_engine('mssql+pyodbc://myUser:mypwd#localhost:1234/MY_DATABASE?driver=SQL+Server+Native+Client+11.0?trusted_connection=yes', isolation_level = "AUTOCOMMIT")
try:
engine.execute('CREATE DATABASE ' + a_database_name)
except Exception as db_exc:
logging.exception("Exception creating database: " + str(db_exc))
If someone like me don't want to take whole sqlalchemy_utils to your project just for database creation, you can use script like this. I've come with it, based on SingleNegationElimination's answer. I'm using pydantic here (it's FastAPI project) and my imported settings for reference, but you can easily change this:
from sqlalchemy import create_engine
from sqlalchemy.exc import OperationalError
from pydantic import PostgresDsn
from src.conf import settings
def build_db_connection_url(custom_db: Optional[str] = None):
db_name = f"/{settings.POSTGRES_DB or ''}" if custom_db is None else "/" + custom_db
return PostgresDsn.build(
scheme='postgresql+psycopg2',
user=settings.POSTGRES_USER,
password=settings.POSTGRES_PASSWORD,
host=settings.POSTGRES_HOST,
path=db_name,
)
def create_database(db_name: str):
try:
eng = create_engine(build_db_connection_url(custom_db=db_name))
conn = eng.connect()
conn.close()
except OperationalError as exc:
if "does not exist" in exc.__str__():
eng = create_engine(build_db_connection_url(custom_db="postgres"))
conn = eng.connect()
conn.execute("commit")
conn.execute(f"create database {db_name}")
conn.close()
print(f"Database {db_name} created")
else:
raise exc
eng.dispose()
create_database("test_database")