Develop Reference

pyodbc DSN - connection to SQL Server without UID and PWD in connection string

I am looking to establish a connection to MS SQL through a DSN using pyodbc. what I am seeing is that I cannot connect to database unless I specify the username (PID) and password (PWD) in the connection string like this:
conn_str = 'DSN=MYMSSQL;UID=sa;PWD=password'
so if I use PID and PWD it works but if I put the PID and PWD in my DSN configuration (MYMSSQL) and remove these two attributes from conn_str then it doesn't work, below is the DSN configuration:
[MYMSSQL]
Description = Test to SQLServer
Driver = FreeTDS
Servername = MYMSSQL
UID = sa
PWD = password
Database = tempdb
Observation from the pyodbc API docs, apprently no way to do it without UID and PWD
def connect(p_str, autocommit=False, ansi=False, timeout=0, **kwargs): # real signature unknown; restored from __doc__
"""
connect(str, autocommit=False, ansi=False, timeout=0, **kwargs) --> Connection
Accepts an ODBC connection string and returns a new Connection object.
**The connection string will be passed to SQLDriverConnect, so a DSN connection
can be created using:**
**cnxn = pyodbc.connect('DSN=DataSourceName;UID=user;PWD=password')**
To connect without requiring a DSN, specify the driver and connection
information:
DRIVER={SQL Server};SERVER=localhost;DATABASE=testdb;UID=user;PWD=password
Note the use of braces when a value contains spaces. Refer to SQLDriverConnect
documentation or the documentation of your ODBC driver for details.
The connection string can be passed as the string `str`, as a list of keywords,
or a combination of the two. Any keywords except autocommit, ansi, and timeout
(see below) are simply added to the connection string.
connect('server=localhost;user=me')
connect(server='localhost', user='me')
connect('server=localhost', user='me')
The DB API recommends the keywords 'user', 'password', and 'host', but these
are not valid ODBC keywords, so these will be converted to 'uid', 'pwd', and
'server'.
pass

The ODBC driver managers that I've dealt with (Windows' built-in DM, and unixODBC on Linux) silently ignore UID= and PWD= entries in "System DSN" and "User DSN" definitions. They do respect those entries in "File DSN" definitions, so you could create a file named "mymssql.dsn" in a secure location containing
[ODBC]
Description = Test to SQLServer
Driver = FreeTDS
Servername = MYMSSQL
UID = sa
PWD = password
Database = tempdb
and then use
conn_str = 'FILEDSN=/path/to/mymssql.dsn'

Connect to an URI in postgres

I'm guessing this is a pretty basic question, but I can't figure out why:
import psycopg2
psycopg2.connect("postgresql://postgres:postgres#localhost/postgres")
Is giving the following error:
psycopg2.OperationalError: missing "=" after
"postgresql://postgres:postgres#localhost/postgres" in connection info string
Any idea? According to the docs about connection strings I believe it should work, however it only does like this:
psycopg2.connect("host=localhost user=postgres password=postgres dbname=postgres")
I'm using the latest psycopg2 version on Python2.7.3 on Ubuntu12.04

I would use the urlparse module to parse the url and then use the result in the connection method. This way it's possible to overcome the psycop2 problem.
from urlparse import urlparse # for python 3+ use: from urllib.parse import urlparse
result = urlparse("postgresql://postgres:postgres#localhost/postgres")
username = result.username
password = result.password
database = result.path[1:]
hostname = result.hostname
port = result.port
connection = psycopg2.connect(
database = database,
user = username,
password = password,
host = hostname,
port = port
)

The connection string passed to psycopg2.connect is not parsed by psycopg2: it is passed verbatim to libpq. Support for connection URIs was added in PostgreSQL 9.2.

To update on this, Psycopg3 does actually include a way to parse a database connection URI.
Example:
import psycopg # must be psycopg 3
pg_uri = "postgres://jeff:hunter2#example.com/db"
conn_dict = psycopg.conninfo.conninfo_to_dict(pg_uri)
with psycopg.connect(**conn_dict) as conn:
...

Another option is using SQLAlchemy for this. It's not just ORM, it consists of two distinct components Core and ORM, and it can be used completely without using ORM layer.
SQLAlchemy provides such functionality out of the box by create_engine function. Moreover, via URI you can specify DBAPI driver or many various postgresql settings.
Some examples:
# default
engine = create_engine("postgresql://user:pass#localhost/mydatabase")
# psycopg2
engine = create_engine("postgresql+psycopg2://user:pass#localhost/mydatabase")
# pg8000
engine = create_engine("postgresql+pg8000://user:pass#localhost/mydatabase")
# psycopg3 (available only in SQLAlchemy 2.0, which is currently in beta)
engine = create_engine("postgresql+psycopg://user:pass#localhost/test")
And here is a fully working example:
import sqlalchemy as sa
# set connection URI here ↓
engine = sa.create_engine("postgresql://user:password#db_host/db_name")
ddl_script = sa.DDL("""
CREATE TABLE IF NOT EXISTS demo_table (
id serial PRIMARY KEY,
data TEXT NOT NULL
);
""")
with engine.begin() as conn:
# do DDL and insert data in a transaction
conn.execute(ddl_script)
conn.exec_driver_sql("INSERT INTO demo_table (data) VALUES (%s)",
[("test1",), ("test2",)])
conn.execute(sa.text("INSERT INTO demo_table (data) VALUES (:data)"),
[{"data": "test3"}, {"data": "test4"}])
with engine.connect() as conn:
cur = conn.exec_driver_sql("SELECT * FROM demo_table LIMIT 2")
for name in cur.fetchall():
print(name)
# you also can obtain raw DBAPI connection
rconn = engine.raw_connection()
SQLAlchemy provides many other benefits:
You can easily switch DBAPI implementations just by changing URI (psycopg2, psycopg2cffi, etc), or maybe even databases.
It implements connection pooling out of the box (both psycopg2 and psycopg3 has connection pooling, but API is different)
asyncio support via create_async_engine (psycopg3 also supports asyncio).

How to create db in MySQL with SQLAlchemy?

I need to create a db in MySQL using SQLAlchemy, I am able to connect to a db if it already exists, but I want to be able to create it if it does not exist. These are my tables:
#def __init__(self):
Base = declarative_base()
class utente(Base):
__tablename__="utente"
utente_id=Column(Integer,primary_key=True)
nome_utente=Column(Unicode(20))
ruolo=Column(String(10))
MetaData.create_all()
def __repr(self):
return "utente: {0}, {1}, id: {2}".format(self.ruolo,self.nome_utente,self.utente_id)
class dbmmas(Base):
__tablename__="dbmmas"
db_id=Column(Integer,primary_key=True,autoincrement=True)
nome_db=Column(String(10))
censimento=Column(Integer)
versione=Column(Integer)
ins_data=Column(DateTime)
mod_data=Column(DateTime)
ins_utente=Column(Integer)
mod_utente=Column(Integer)
MetaData.create_all()
def __repr(self):
return "dbmmas: {0}, censimento {1}, versione {2}".format(self.nome_db,self.censimento,self.versione)
class funzione(Base):
__tablename__="funzione"
funzione_id=Column(Integer,primary_key=True,autoincrement=True)
categoria=Column(String(10))
nome=Column(String(20))
def __repr__(self):
return "funzione:{0},categoria:{1},id:{2} ".format(self.nome,self.categoria,self.funzione_id)
class profilo(Base):
__tablename__="rel_utente_funzione"
utente_id=Column(Integer,primary_key=True)
funzione_id=Column(Integer,primary_key=True)
amministratore=Column(Integer)
MetaData.create_all()
def __repr(self):
l=lambda x: "amministratore" if x==1 else "generico"
return "profilo per utente_id:{0}, tipo: {1}, funzione_id: {2}".format(self.utente_id,l(self.amministratore),self.funzione_id)
class aree(Base):
__tablename__="rel_utente_zona"
UTB_id=Column(String(10), primary_key=True) # "in realta' si tratta della seatureSignature della feature sullo shapefile"
utente_id=Column(Integer, primary_key=True)
amministratore=Column(Integer)
MetaData.create_all()
def __repr(self):
l=lambda x: "amministratore" if x==1 else "generico"
return "zona: {0}, pe utente_id:{1}, {2}".format(self.UTB_id,self.utente_id,l(self.amministratore))
class rel_utente_dbmmas(Base):
__tablename__="rel_utente_dbmmas"
utente_id=Column(Integer,primary_key=True)
db_id=Column(Integer,primary_key=True)
amministratore=(Integer)
MetaData.create_all()
def __repr(self):
l=lambda x: "amministratore" if x==1 else "generico"
return "dbregistrato: {0} per l'utente{1} {2}".format(self.db_id,self.utente_id,l(self.amministratore))

To create a mysql database you just connect to the server an create the database:
import sqlalchemy
engine = sqlalchemy.create_engine('mysql://user:password#server') # connect to server
engine.execute("CREATE DATABASE dbname") #create db
engine.execute("USE dbname") # select new db
# use the new db
# continue with your work...
of course your user has to have the permission to create databases.

You can use SQLAlchemy-Utils for that.
pip install sqlalchemy-utils
Then you can do things like
from sqlalchemy_utils import create_database, database_exists
url = 'mysql://{0}:{1}#{2}:{3}'.format(user, pass, host, port)
if not database_exists(url):
create_database(url)
I found the answer here, it helped me a lot.

I don't know what the canonical way is, but here's a way to check to see if a database exists by checking against the list of databases, and to create it if it doesn't exist.
from sqlalchemy import create_engine
# This engine just used to query for list of databases
mysql_engine = create_engine('mysql://{0}:{1}#{2}:{3}'.format(user, pass, host, port))
# Query for existing databases
existing_databases = mysql_engine.execute("SHOW DATABASES;")
# Results are a list of single item tuples, so unpack each tuple
existing_databases = [d[0] for d in existing_databases]
# Create database if not exists
if database not in existing_databases:
mysql_engine.execute("CREATE DATABASE {0}".format(database))
print("Created database {0}".format(database))
# Go ahead and use this engine
db_engine = create_engine('mysql://{0}:{1}#{2}:{3}/{4}'.format(user, pass, host, port, db))
Here's an alternative method if you don't need to know if the database was created or not.
from sqlalchemy import create_engine
# This engine just used to query for list of databases
mysql_engine = create_engine('mysql://{0}:{1}#{2}:{3}'.format(user, pass, host, port))
# Query for existing databases
mysql_engine.execute("CREATE DATABASE IF NOT EXISTS {0} ".format(database))
# Go ahead and use this engine
db_engine = create_engine('mysql://{0}:{1}#{2}:{3}/{4}'.format(user, pass, host, port, db))

CREATE DATABASE IF NOT EXISTS dbName;

Would recommend using with:
from sqlalchemy import create_engine
username = ''
password = ''
host = 'localhost'
port = 3306
DB_NAME = 'db_name'
engine = create_engine(f"mysql://{username}:{password}#{host}:{port}")
with engine.connect() as conn:
# Do not substitute user-supplied database names here.
conn.execute(f"CREATE DATABASE IF NOT EXISTS {DB_NAME}")

The mysqlclient seems to be up to 10 times faster in benchmark tests than PyMySQL, see: What's the difference between MySQLdb, mysqlclient and MySQL connector/Python?.
Yet, why not use a Python-ready package for Python, at least, if it is not about every second of query time? PyMySQL is suggested by the following links, for example:
Using SQLAlchemy to access MySQL without frustrating library installation issues
How to connect MySQL database using Python+SQLAlchemy remotely?.
Python packages:
Install with pip, at best put in "requirements.txt":
PyMySQL
SQLAlchemy
Again, if it is about the best speed of the query, use mysqlclient package. Then you need to install an additional Linux package with sudo apt-get install libmysqlclient-dev.
import statements
Only one needed:
import sqlalchemy
Connection string (= db_url)
Connection string starting with {dialect/DBAPI}+{driver}:
db_url = mysql+pymysql://
where pymysql stands for the used Python package "PyMySQL" as the driver.
Again, if it is about the best speed of the query, use mysqlclient package. Then you need mysql+msqldb:// at this point.
For a remote connection, you need to add to the connection string:
host
user
password
database
port (the port only if it is not the standard 3306)
You can create your db_url with several methods. Do not write user and password and at best any other variable value directly in the string to avoid possible attacks:
sqlalchemy.engine.URL.create(), or with .url.URL, see an example at Connecting from Cloud Functions to Cloud SQL or an example which automatically adds ? suffixes, for example ?driver=SQL+Server, at the end of the string at Building a connection URL for mssql+pyodbc with sqlalchemy.engine.url.URL
f"""...{my_var}..."""
"""...{my_var}...""".format(my_var=xyz_var)
...
Example without the url helper of SQLAlchemy:
db_url = "{dialect}+{driver}://{user}:{password}#{host}:{port}/{database}".format(
or:
db_url = "{dialect}+{driver}://{user}:{password}#{host}/{database}?host={host}?port={port}".format(
dialect = 'mysql',
driver = 'pymysql',
username=db_user,
password=db_pass,
database=db_name,
host=db_host,
port=db_port
)
Other engine configurations
For other connection drivers, dialects and methods, see the SQLAlchemy 1.4 Documentation - Engine Configuration
Create the db if not exists
See How to create a new database using SQLAlchemy?.
engine = sqlalchemy.create_engine(db_url)
if not sqlalchemy.database_exists(engine.url):
create_database(engine.url)
with engine.connect() as conn:
conn.execute("commit")
conn.execute("create database test")

ssh first with mysqldb in python

I'm trying to connect to a MySQL database on a remote server using MySQLdb in python. The problem is that first I need to SSH into the host, and then from there, I need to connect to the MySQL server. The problem I'm having, though, is that MySQLdb does not seem to have a way of establishing an SSH connection before connecting to the SQL server. I've checked the documentation but have not had any luck.
This is how I'm connecting:
conn = MySQLdb.connect(host = 'mysqlhost.domain.com:3306', user = 'user', passwd = 'password', db = 'dbname')
But what I really need is something like this:
conn = MySQLdb.connect(sshhost = 'sshhost.domain.com', sshuser = 'sshusername', sshpasswd = 'sshpasswd', host = 'mysqlhost.domain.com:3306', user = 'user', passwd = 'password', db = 'dbname')
Which is of course just made up. Can anyone make any recommendations?

I prefer keeping the tunnel within the python code, I did hate to create tunnels manually, or separately, thanks to sshtunnel library its very simple to use.
Here is some simple sample that will work for what you want.
import MySQLdb
from sshtunnel import SSHTunnelForwarder
with SSHTunnelForwarder(
('sshhost.domain.com', 22),
ssh_password="sshpasswd",
ssh_username="sshusername",
remote_bind_address=('mysqlhost.domain.com', 3306)) as server:
conn = MySQLdb.connect(host='127.0.0.1',
port=server.local_bind_port,
user='user',
passwd='password',
db='dbname')

Setup an ssh tunnel before you use MySQLdb.connect. The tunnel will make it appear as though you have the mysql running locally, set it up something like this
ssh user#host.com -L 9990:localhost:3306
here your local port 9990 will bind to 3306 on the remote host, -L stands for local, then 9990:localhost:3306 means LOCALPORT:
conn = MySQLdb.connect(host = 'mysqlhost.domain.com:9990', user = 'user', passwd = 'password', db = 'dbname')
notice the 9990.
Add your public ssh key of user to the host.com so that you dont have to type the password each time you want to setup the tunnel (use public key authentication).
If you need to do this within python there is python-to-ssh binding libraries you could call from within python to setup the tunnel for you.

How to make mysql connection that requires CA-CERT with sqlalchemy or SQLObject

I would like to connect to a MySQL database that requires ca-cert. I can do it with MySQLdb like below:
MySQLdb.connect(host = self.host,
port = self.port,
unix_socket = self.unix_socket,
user = self.user,
passwd = self.passwd,
db = self.db,
ssl = { 'cert': self.sslcert,
'key': self.sslkey,
'ca': self.sslca }
How do I do the same think in SQLAlchemy or SQLObject?
Thanks,
peter

To use SSL certs with SQLAlchemy and MySQLdb, use the following python code:
db_connect_string='mysql://<user>:<pswd>#<db server>:3306/<database>'
ssl_args = {'ssl': {'cert':'/path/to/client-cert',
'key':'/path/to/client-key',
'ca':'/path/to/ca-cert'}}
create_engine(db_connect_string, connect_args=ssl_args)

create_engine() in SQLAlchemy has a connect_args parameter:
connect_args – a dictionary of options which will be passed directly
to the DBAPI’s connect() method as additional keyword arguments.

SQLObject (untested):
from sqlobject.mysql import MySQLConnection
connection = MySQLConnection(
db=self.db,
user=self.user,
password=self.password,
host=self.host,
ssl_key=self.sslkey,
ssl_cert=self.sslcert,
ssl_ca=self.sslca,
)

According to their docs, SQLAlchemy's create_engine function takes a db url with the following format: dialect[+driver]://user:password#host/dbname[?key=value..] meaning you could pass the ssl key, cert, and ca as key value pairs.