I want to connect to a remote PostgreSQL database through Python to do some basic data analysis. This database requires SSL (verify-ca), along with three files (which I have):
Server root certificate file
Client certificate file
Client key file
I have not been able to find a tutorial which describes how to make this connection with Python.
Any help is appreciated.
Use the psycopg2 module.
You will need to use the ssl options in your connection string, or add them as key word arguments:
import psycopg2
conn = psycopg2.connect(dbname='yourdb', user='dbuser', password='abcd1234', host='server', port='5432', sslmode='require')
In this case sslmode specifies that SSL is required.
To perform server certificate verification you can set sslmode to verify-full or verify-ca. You need to supply the path to the server certificate in sslrootcert. Also set the sslcert and sslkey values to your client certificate and key respectively.
It is explained in detail in the PostgreSQL Connection Strings documentation (see also Parameter Key Words) and in SSL Support.
You may also use an ssh tunnel with paramiko and sshtunnel:
import psycopg2
import paramiko
from sshtunnel import SSHTunnelForwarder
mypkey = paramiko.RSAKey.from_private_key_file('/path/to/private/key')
tunnel = SSHTunnelForwarder(
(host_ip, 22),
ssh_username=username,
ssh_pkey=mypkey,
remote_bind_address=('localhost', psql_port))
tunnel.start()
conn = psycopg2.connect(dbname='gisdata', user=psql_username, password=psql_password, host='127.0.0.1', port=tunnel.local_bind_port)
If you need to connect to your PostgresSQL database with an SSL certificate using psycopg2, you'll need to put your certificate SSL certificate in a subdirectory of your python program, and then you can reference the certificate in your connection string. I believe you could also set an environment variable as well, but in my example my SSL certificate will be in a subdirectory.
My python script is in a directory which looks like:
/Users/myusername/Desktop/MyCoolPythonProgram/test_database_connection.py
And my SSL certificate is in a directory which looks like:
/Users/myusername/Desktop/MyCoolPythonProgram/database/ssl_certificate/database/ssl_certificate/ca-certificate.crt
My HOSTNAME is a URL from DigitalOcean, but yours might be an IP Address instead.
This is what my test_database_connection.py script looks like:
import psycopg2
import os
POSTGRES_DATABASE_HOST_ADDRESS = "your-database-name-do-user-12345678-0.b.db.ondigitalocean.com"
POSTGRES_DATABASE_NAME = "defaultdb"
POSTGRES_USERNAME = "doadmin"
POSTGRES_PASSWORD = "$uperD00P3Rp#$$W0RDg0E$here"
# HOW TO (Relative Path Python): https://stackoverflow.com/questions/918154/relative-paths-in-python
path_to_current_directory = os.path.dirname(__file__)
relative_path_to_ssl_cert = 'database/ssl_certificate/ca-certificate.crt'
SSL_ROOT_CERT = os.path.join(path_to_current_directory , relative_path_to_ssl_cert )
POSTGRES_CONNECTION_PORT = "1234" # Set this to the correct port! Mine is provided by DigitalOcean and it's NOT 1234
db_info = "host='%s' dbname='%s' user='%s' password='%s' sslmode='require' sslrootcert='%s' port='%s'" % (POSTGRES_DATABASE_HOST_ADDRESS, POSTGRES_DATABASE_NAME, POSTGRES_USERNAME, POSTGRES_PASSWORD, SSL_ROOT_CERT, POSTGRES_CONNECTION_PORT)
postgres_connection = psycopg2.connect(db_info)
with postgres_connection:
with postgres_connection.cursor() as postgres_cursor:
sql = "SELECT * FROM your_table;"
postgres_cursor.execute(sql)
results = postgres_cursor.fetchall()
for row in results:
print("row in result")
print("Connection Success!")
# Close Database Cursor/Connection
postgres_cursor.close()
Adding this for completeness and because I couldn't find it anywhere else on SO. Like #mhawke says, you can use psycopg2, but you can also use any other Python database modules (ORMs, etc) that allow you to manually specify a database postgresql URI (postgresql://[user[:password]#][netloc][:port][/dbname][?param1=value1&...]) to connect to since the sslmode="require" parameter that psycopg2.connect uses to enforce ssl connections is just part of the postgresql:// URI that you use to connect to your database (see 33.1.2. Parameter Key Words). So, if you wanted to use sqlalchemy or another ORM instead of vanilla psycopg2, you can tack your desired sslmode onto the end of your database URI and connect that way.
import sqlalchemy
DATABASE_URI = "postgresql://postgres:postgres#localhost:5432/dbname"
# sqlalchemy 1.4+ uses postgresql:// instead of postgres://
ssl_mode = "?sslmode=require"
DATABASE_URI += ssl_mode
engine = sqlalchemy.create_engine(URI)
Session = sqlalchemy.orm.sessionmaker(bind=engine)
There's a nifty figure (Table 33.1) in the postgres documentation on SSL Support that breaks down the different options you can supply. If you want to use any of the fancier options that require you to specify a path to a specific certificate, you can drop it in with a format string.
Related
Using python 3.10.10 on Windows 10 I am trying to connect to a mongo database via ssh ideally. On the command line I just do
ssh myuser#111.222.333.444
mongo
and I can query the mongo DB. With the following python code
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure
HOST = "111.222.333.444"
USER = "myuser"
class Mongo:
def __init__(self):
self.host = HOST
self.user = USER
self.uri = f"mongodb://{self.user}#{self.host}"
def connection(self):
try:
client = MongoClient(self.uri)
client.server_info()
print('Connection Established')
except ConnectionFailure as err:
raise(err)
return client
mongo = Mongo()
mongo.connection()
however I get an error
pymongo.errors.ConfigurationError: A password is required.
But as I am able to just login via ssh using my public key I do not require a password. How can this be solved in python?
I also tried to run a command on the command line using ssh alone like
ssh myuser#111.222.333.444 "mongo;use mydb; show collections"
but this does not work like that either.
You do two different things. In the first command you connect via ssh (using port 22) to the remote server. On the remote server you start the mongo shell. In the second command, you try to connect directly to the mongod server (default port 27017).
In your case myuser is the user on remote server operating system, not the user on the MongoDB.
You can (almost) always connect to a MongoDB without username/password, however when you provide a username then you also need a password. Try
self.uri = f"mongodb://{self.host}"
It is not fully clear what you try to achieve. You can configure MongoDB to logon with x509 certificate instead of username/password, see Use x.509 Certificates to Authenticate Clients. These connections are also encrypted via TLS/SSL.
Or are you looking to configure a SSH-Tunnel? See https://serverfault.com/questions/597765/how-to-connect-to-mongodb-server-via-ssh-tunnel
Here is the solution that I found in the end, as simple as possible, and it can be run from within python, and without any special module to install, from a windows powershell:
import json
import subprocess
cmd_mongo = json.dumps('db.units.find({"UnitId": "971201065"})')
cmd_host = json.dumps(f"mongo mydb --eval {cmd_mongo}")
cmd_local = f"ssh {USER}#{HOST} \"{cmd_host}\""
output = subprocess.check_output(cmd_local, shell=True)
print(output)
My app is deployed in GCP, I'm trying to make a connection to DB using psycopg2. The SSL certificates and key are not stored as files, so I'll be getting them as strings.
When I try to make a connection by passing the filepath for these certificate pem files, it works.
psycopg2.connect(host='hostname',port=1234, connect_timeout=100, database='db', user='user', password='pwd',
sslrootcert="server-cert.pem",
sslcert="client-cert.pem",
sslkey="key.pem")
But when I pass certificates and key as strings, it doesn't work. It gives an error
FATAL: connection requires a valid client certificate\nconnection to
server at "hostname", port 1234 failed
SERVER_CERT = """-----BEGIN CERTIFICATE-----\nxxxxxx\n-----END CERTIFICATE-----"""
CLIENT_CERT = """-----BEGIN CERTIFICATE-----\nxxxxxx\n-----END CERTIFICATE-----"""
KEY = """-----BEGIN RSA PRIVATE KEY-----\nxxxxn-----END RSA PRIVATE KEY-----"""
psycopg2.connect(host='hostname',port=1234, connect_timeout=100, database='db', user='user', password='pwd',
sslrootcert=SERVER_CERT,
sslcert=CLIENT_CERT,
sslkey=KEY)
I also tried using ssl.DER_cert_to_PEM_cert(CERT) and RSA.importKey(KEY), but it still fails.
Is there a way to pass string instead of files? Thanks.
I was facing the same situation a couple of hours ago. What I did to resolve this is creating the files in python with the value of the variable:
cert = """cert"""
file = open("cert.txt","w")
file.write(cert)
file.close()
And then, just pass the path to the psycopg2 connection.
I am using cx_oracle module with python 3.7 version and I need to check whether the connection is encrypted. If not I need to set the ssl as true in order to make it encrypted.
Here is my piece of code to make the connection:
import cx_Oracle
dsn = cx_Oracle.makedsn(host='127.0.0.1', port=1521, sid='your_sid')
conn = cx_Oracle.connect(user='your_username', password='your_password', dsn=dsn)
conn.close()
As you are thinking in enabling security to your connection, your first step should be to use a wallet, even before considering using ssl , and avoid using passwords. It does not matter how encrypted is your network traffic, if your passwords are visible in your Python programs. I know it is not part of the question itself, but it is a very good practice and available for cx_Oracle.
One example ( My Python programs runs in a Linux client machine which connects to an Oracle Database in Linux too using ssl )
Client Side
1.Create the wallet
mkstore -wrl "/home/myuser/wallet_directory" -create
2.Create the credential
mkstore -wrl "/home/myuser/wallet_directory" -createCredential mynetalias myuser myuserpw
Where mynetalias is an alias for my tns string connection which I will store on my tnsnames.ora file. In my example I will use the same directory where I created the wallet.
3.Create the tnsnames.ora and add the same alias used in the wallet
mynetalias =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbhost.example.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orclpdb1)
)
)
4.Create the sqlnet.ora file
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /home/myuser/wallet_dir)
)
)
SQLNET.WALLET_OVERRIDE = TRUE
5.Add your TNS_ADMIN environment variable to your bash profile.
cd
echo "export TNS_ADMIN=/home/myuser/wallet_directory" >> .bashrc
If you definitely know that the database server enforces integrity and encryption, then you do not need to configure anything in the client side. However you can also, or alternatively, do so depending on your business needs. Add the following lines to the sqlnet.ora file where the wallet is located
SQLNET.CRYPTO_CHECKSUM_CLIENT = required
SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT = (SHA512)
SQLNET.ENCRYPTION_CLIENT = required
SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256)
Database Side
In order to setup SSL and encryption we need to add these values to the Database sqlnet.ora file. Review your requirements and discuss the right security algorithms. In my case my database accepts connection either way ( with or without encryption ).
SQLNET.CRYPTO_CHECKSUM_SERVER = accepted
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER = (SHA512)
SQLNET.ENCRYPTION_SERVER = accepted
SQLNET.ENCRYPTION_TYPES_SERVER = (AES256)
You might want to review these parameters here:
SQLNET Parameters
How to connect
Normal
connection = cx_Oracle.connect(dsn="mynetalias")
Pool
pool = cx_Oracle.SessionPool(externalauth=True, homogeneous=False,
dsn="mynetalias")
pool.acquire()
Remember that dsn must match exactly the alias used in your tnsnames.ora configured before.
Use the information provided by the view V$SESSION_CONNECT_INFO to assure your connection is encrypted ( field network_service_banner)
we can use python-oracledb driver which is the major version successor to cx_Oracle 8.3. https://python-oracledb.readthedocs.io/en/latest/user_guide/introduction.html
reference to the code: https://github.com/oracle/python-oracledb/discussions/34
I tried the exact same code as found here...
https://github.com/TomMalkin/SimQLe
I am not sure how to connect to mysql database.
from simqle import ConnectionManager
cm = ConnectionManager("connections.yaml")
sql = "SELECT name, age FROM people WHERE category = :category"
params = {"category": 5}
result = cm.recordset(con_name="my-database", sql=sql, params=params)
Getting an error:
UnknownConnectionError: Unknown connection my-database
This is how I can connect to mysql database from command prompt.
mysql -h 172.31.84.39 -udba -pXXXX -P 3392
How do I write the connection string?
I usually use sqlalchemy to connect mysql database. I have readed the document of SimQLe which you are using. In SimQLe document,
cm = ConnectionManager("connections.yaml")
is the way to connect to database and you should put your login param in this yaml file called connections.yaml.
Here is the offical document simple:
https://github.com/TomMalkin/SimQLe#the-connectionsyaml-file
connections:
# The name of the connection - this is what will be used in your project
# to reference this connection.
- name: my-sql-server-database
driver: mssql+pyodbc:///?odbc_connect=
connection: DRIVER={SQL Server};UID=<username>;PWD=<password>;SERVER=<my-server>
# some odbc connections require urls to be escaped, this is managed by
# setting url_escaped = true:
url_escape: true
# File based databases like sqlite are slightly different - the driver
# is very simple.
- name: my-sqlite-database
driver: sqlite:///
# put a leading '/' before the connection for an absolute path, or omit
# if it's relative to the project path
connection: databases/my-database.db
# This connection will be used if no name is given if the default
# parameter is used:
default: true
Maybe you should change some params in here:
driver: mssql+pyodbc:///?odbc_connect=
connection: DRIVER={SQL Server};UID=<username>;PWD=<password>;SERVER=<my-server>
And from the document, it says that SimQle is built on SQLAlchemy,
SimQLe is built on top of the fantastic SQLAlchemy library.
maybe you can use the SQLAlchemy's login params to connect the database in SimQLe. Such like this:
mysql+pymysql://<username>:<password>#<host>/<dbname>[?<options>]
Changed to:
driver: mysql+pymysql://
connection: <username>:<password>#<host>/<dbname>[?<options>]
Offical documents:
https://simqle.readthedocs.io/en/latest/
https://docs.sqlalchemy.org/en/14/dialects/mysql.html#module-sqlalchemy.dialects.mysql.pymysql
https://docs.sqlalchemy.org/en/14/dialects/mssql.html#module-sqlalchemy.dialects.mssql.pyodbc
I have a Django app below that uses a proxy to connect to an external Postgres database. I had to replace another package with psycopg2 and it works fine locally, but doesn't work when I move onto our production server which is a Heroku app using QuotaguardStatic for proxy purposes. I'm not sure what's wrong here
For some reason, the psycopg2.connect part returns an error with a different IP address. Is it not inheriting the proxy set in the context manager? What would be
from apps.proxy.socks import Socks5Proxy
import requests
PROXY_URL = os.environ['QUOTAGUARDSTATIC_URL']
with Socks5Proxy(url=PROXY_URL) as p:
public_ip = requests.get("http://wtfismyip.com/text").text
print(public_ip) # prints the expected IP address
print('end')
try:
connection = psycopg2.connect(user=EXTERNAL_DB_USERNAME,
password=EXTERNAL_DB_PASSWORD,
host=EXTERNAL_DB_HOSTNAME,
port=EXTERNAL_DB_PORT,
database=EXTERNAL_DB_DATABASE,
cursor_factory=RealDictCursor # To access query results like a dictionary
) # , ssl_context=True
except psycopg2.DatabaseError as e:
logger.error('Unable to connect to Illuminate database')
raise e
Error is:
psycopg2.OperationalError: FATAL: no pg_hba.conf entry for host "12.345.678.910", user "username", database "databasename", SSL on
Basically, the IP address 12.345.678.910 does not match what was printed at the beginning of the context manager where the proxy is set. Do I need to set a proxy another method so that the psycopg2 connection uses it?