Develop Reference

How to connect to a mongo DB via SSH using Python?

Using python 3.10.10 on Windows 10 I am trying to connect to a mongo database via ssh ideally. On the command line I just do
ssh myuser#111.222.333.444
mongo
and I can query the mongo DB. With the following python code
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure
HOST = "111.222.333.444"
USER = "myuser"
class Mongo:
def __init__(self):
self.host = HOST
self.user = USER
self.uri = f"mongodb://{self.user}#{self.host}"
def connection(self):
try:
client = MongoClient(self.uri)
client.server_info()
print('Connection Established')
except ConnectionFailure as err:
raise(err)
return client
mongo = Mongo()
mongo.connection()
however I get an error
pymongo.errors.ConfigurationError: A password is required.
But as I am able to just login via ssh using my public key I do not require a password. How can this be solved in python?
I also tried to run a command on the command line using ssh alone like
ssh myuser#111.222.333.444 "mongo;use mydb; show collections"
but this does not work like that either.

You do two different things. In the first command you connect via ssh (using port 22) to the remote server. On the remote server you start the mongo shell. In the second command, you try to connect directly to the mongod server (default port 27017).
In your case myuser is the user on remote server operating system, not the user on the MongoDB.
You can (almost) always connect to a MongoDB without username/password, however when you provide a username then you also need a password. Try
self.uri = f"mongodb://{self.host}"
It is not fully clear what you try to achieve. You can configure MongoDB to logon with x509 certificate instead of username/password, see Use x.509 Certificates to Authenticate Clients. These connections are also encrypted via TLS/SSL.
Or are you looking to configure a SSH-Tunnel? See https://serverfault.com/questions/597765/how-to-connect-to-mongodb-server-via-ssh-tunnel

Here is the solution that I found in the end, as simple as possible, and it can be run from within python, and without any special module to install, from a windows powershell:
import json
import subprocess
cmd_mongo = json.dumps('db.units.find({"UnitId": "971201065"})')
cmd_host = json.dumps(f"mongo mydb --eval {cmd_mongo}")
cmd_local = f"ssh {USER}#{HOST} \"{cmd_host}\""
output = subprocess.check_output(cmd_local, shell=True)
print(output)

ValueError: No password or public key available

I'm trying to connect to a remote MySQL database through an SSH Tunnel and deploying my code to Streamlit. When I try to do it, I get this error:
File "/home/appuser/venv/lib/python3.9/site-packages/sshtunnel.py", line 966, in __init__
(self.ssh_password, self.ssh_pkeys) = self._consolidate_auth(
File "/home/appuser/venv/lib/python3.9/site-packages/sshtunnel.py", line 1169, in _consolidate_auth
raise ValueError('No password or public key available!')
ValueError: No password or public key available!
I've tried a lot of things, from updating my SSH keys to my server and github to changing my code.
The code I have for the SSH - MySQL section looks like this:
import MySQLdb as db
from sshtunnel import SSHTunnelForwarder
def query(q):
with SSHTunnelForwarder(
ssh_address_or_host=("host_ip"),
ssh_username=("host_username"),
ssh_pkey=("path_to_private_sshkey"),
remote_bind_address=("private_host_ip", "host_port")
) as server:
conn = db.connect(
host="localhost",
port=server.local_bind_port,
user="db_username",
passwd="db_password",
db="db_database"
)
return pd.read_sql_query(q, conn)
I appreciate any help you can give me.

conn = db.connect(host="localhost"),
port=server.local_bind_port,
user=("db_username"),
passwd=("db_password"),
db=("db_database")
Because you have a closing parentheses on the first line, only the host argument is being passed to the db.connect() function. And so the function is complaining that it doesn't have a password, username, etc.
The other lines are creating plain local variables.

How to use Python SSHTunnle to forward multiple ports

I need to forward to multiple ports which are sits behind a server
server1(22) -> Server2(mysql, 3360) = local 3360
-> Server3(http, 8080) = local 8080
-> Server4(oracle,1234) = local 1234
I can only access Server2,3, and 4 via server1.
I am using Python ssltunnel package https://pypi.org/project/sshtunnel/
In example1&2, I can only specify one remote&local bind address.
Not sure how to connect multiple servers(2,3,4)
Example1
from sshtunnel import SSHTunnelForwarder
server = SSHTunnelForwarder(
'pahaz.urfuclub.ru',
ssh_username="pahaz",
ssh_password="secret",
remote_bind_address=('127.0.0.1', 8080)
)
server.start()
print(server.local_bind_port) # show assigned local port
# work with `SECRET SERVICE` through `server.local_bind_port`.
server.stop()
Example 2
import paramiko
import sshtunnel
with sshtunnel.open_tunnel(
(REMOTE_SERVER_IP, 443),
ssh_username="",
ssh_pkey="/var/ssh/rsa_key",
ssh_private_key_password="secret",
remote_bind_address=(PRIVATE_SERVER_IP, 22),
local_bind_address=('0.0.0.0', 10022)
) as tunnel:
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('127.0.0.1', 10022)
# do some operations with client session
client.close()
print('FINISH!')
I could use any other Python package that can do the job.

Both examples can be modified slightly to work the way you want.
There is the singular versions of bindings (local_bind_address & remote_bind_address) and the plural versions of bindings (local_bind_addresses & remote_bind_addresses.
The singular verisons expects a tuple containing variables for the connections, while the plural versions expects a list of one or more tuple(s).
Here is a modified version of your example 2:
import paramiko
import sshtunnel
tunnels = [("172.16.0.1", 80),
("172.16.0.2", 22)]
localPorts = [("127.0.0.1", 1180),
("127.0.0.1", 10022)]
with sshtunnel.open_tunnel(
(REMOTE_SERVER_IP, 22),
ssh_username="",
ssh_pkey="/var/ssh/rsa_key",
ssh_private_key_password="secret",
remote_bind_addresses=tunnels,
local_bind_addresses=localPorts
) as tunnel:
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('127.0.0.1', 10022)
# do some operations with client session
client.close()
If the lengths of the lists are the same length, then the IP-addresses / ports will correspond with each other.
In my example above, the following is happening:
Connection: 172.16.0.1 Port: 80, Is tunneled via: 127.0.0.1 Port: 1180
Connection: 172.16.0.2 Port: 22, Is tunneled via: 127.0.0.1 Port:
10022

I am adding this solution for multiple hops on multiple ports:
I have this setup:
the goal is to access the database by calling my machine on port 33306. This isn't possible because only gateway2 is allowed to speak to the database. We cant access gateway2 because only gateway1 is allowed to speak to it.
the following is the corresponding ssh .config file:
Host gateway1
HostName gtw1_IP_address
User gtw1_user
IdentityFile "path_to_gtw1_ssh_key"
IdentitiesOnly True
Host gateway2
User gtw2_user
Hostname gtw2_IP_address
IdentityFile "path_to_gtw2_ssh_key"
IdentitiesOnly True
# mysql
LocalForward 127.0.0.1:33306 127.0.0.1:3306
ProxyCommand ssh -W %h:%p switch-cede
this is how I reproduce it in python:
from sqlalchemy import create_engine
import config
import pandas as pd
import sshtunnel
from paramiko import SSHClient
with sshtunnel.open_tunnel(
ssh_username='gtw1_user',
ssh_address_or_host=('gtw1_IP_address', 22),
remote_bind_addresses=[('gtw2_IP_address', 22), ('gtw2_IP_address', 33306)],
local_bind_addresses=[('0.0.0.0', 22), ('0.0.0.0', 33306)], #this line is optional
ssh_pkey=path_to_gtw1_ssh_key,
) as tunnel1: # tunnel1 is the tunnel between myMachine and gateway1 I believe
print(tunnel1.local_bind_ports)
with sshtunnel.open_tunnel(
ssh_address_or_host=('localhost', tunnel1.local_bind_ports[0]),
remote_bind_addresses=[('127.0.0.1', 22),('127.0.0.1', 3306)],
local_bind_addresses=[('0.0.0.0', 22), ('127.0.0.1', 33306)],
ssh_username='gtw2_user',
ssh_pkey=path_to_gtw2_ssh_key,
) as tunnel2: # tunnel2 is the tunnel between gtw1 and gtw2 I believe
print(tunnel2.local_bind_ports)
db = create_engine(
f"mysql+pymysql://{config.USER}:{config.PASSWORD}#{config.HOST}:{config.PORT}/{config.DATABASE}")
print(db)
query = "SELECT * FROM randomTable LIMIT 10;"
df = pd.read_sql(query, db)
print(df)
# note that config is a file holding the credentials to connect to the database

How to connect to a remote PostgreSQL database through SSL with Python

I want to connect to a remote PostgreSQL database through Python to do some basic data analysis. This database requires SSL (verify-ca), along with three files (which I have):
Server root certificate file
Client certificate file
Client key file
I have not been able to find a tutorial which describes how to make this connection with Python.
Any help is appreciated.

Use the psycopg2 module.
You will need to use the ssl options in your connection string, or add them as key word arguments:
import psycopg2
conn = psycopg2.connect(dbname='yourdb', user='dbuser', password='abcd1234', host='server', port='5432', sslmode='require')
In this case sslmode specifies that SSL is required.
To perform server certificate verification you can set sslmode to verify-full or verify-ca. You need to supply the path to the server certificate in sslrootcert. Also set the sslcert and sslkey values to your client certificate and key respectively.
It is explained in detail in the PostgreSQL Connection Strings documentation (see also Parameter Key Words) and in SSL Support.

You may also use an ssh tunnel with paramiko and sshtunnel:
import psycopg2
import paramiko
from sshtunnel import SSHTunnelForwarder
mypkey = paramiko.RSAKey.from_private_key_file('/path/to/private/key')
tunnel = SSHTunnelForwarder(
(host_ip, 22),
ssh_username=username,
ssh_pkey=mypkey,
remote_bind_address=('localhost', psql_port))
tunnel.start()
conn = psycopg2.connect(dbname='gisdata', user=psql_username, password=psql_password, host='127.0.0.1', port=tunnel.local_bind_port)

If you need to connect to your PostgresSQL database with an SSL certificate using psycopg2, you'll need to put your certificate SSL certificate in a subdirectory of your python program, and then you can reference the certificate in your connection string. I believe you could also set an environment variable as well, but in my example my SSL certificate will be in a subdirectory.
My python script is in a directory which looks like:
/Users/myusername/Desktop/MyCoolPythonProgram/test_database_connection.py
And my SSL certificate is in a directory which looks like:
/Users/myusername/Desktop/MyCoolPythonProgram/database/ssl_certificate/database/ssl_certificate/ca-certificate.crt
My HOSTNAME is a URL from DigitalOcean, but yours might be an IP Address instead.
This is what my test_database_connection.py script looks like:
import psycopg2
import os
POSTGRES_DATABASE_HOST_ADDRESS = "your-database-name-do-user-12345678-0.b.db.ondigitalocean.com"
POSTGRES_DATABASE_NAME = "defaultdb"
POSTGRES_USERNAME = "doadmin"
POSTGRES_PASSWORD = "$uperD00P3Rp#$$W0RDg0E$here"
# HOW TO (Relative Path Python): https://stackoverflow.com/questions/918154/relative-paths-in-python
path_to_current_directory = os.path.dirname(__file__)
relative_path_to_ssl_cert = 'database/ssl_certificate/ca-certificate.crt'
SSL_ROOT_CERT = os.path.join(path_to_current_directory , relative_path_to_ssl_cert )
POSTGRES_CONNECTION_PORT = "1234" # Set this to the correct port! Mine is provided by DigitalOcean and it's NOT 1234
db_info = "host='%s' dbname='%s' user='%s' password='%s' sslmode='require' sslrootcert='%s' port='%s'" % (POSTGRES_DATABASE_HOST_ADDRESS, POSTGRES_DATABASE_NAME, POSTGRES_USERNAME, POSTGRES_PASSWORD, SSL_ROOT_CERT, POSTGRES_CONNECTION_PORT)
postgres_connection = psycopg2.connect(db_info)
with postgres_connection:
with postgres_connection.cursor() as postgres_cursor:
sql = "SELECT * FROM your_table;"
postgres_cursor.execute(sql)
results = postgres_cursor.fetchall()
for row in results:
print("row in result")
print("Connection Success!")
# Close Database Cursor/Connection
postgres_cursor.close()

Adding this for completeness and because I couldn't find it anywhere else on SO. Like #mhawke says, you can use psycopg2, but you can also use any other Python database modules (ORMs, etc) that allow you to manually specify a database postgresql URI (postgresql://[user[:password]#][netloc][:port][/dbname][?param1=value1&...]) to connect to since the sslmode="require" parameter that psycopg2.connect uses to enforce ssl connections is just part of the postgresql:// URI that you use to connect to your database (see 33.1.2. Parameter Key Words). So, if you wanted to use sqlalchemy or another ORM instead of vanilla psycopg2, you can tack your desired sslmode onto the end of your database URI and connect that way.
import sqlalchemy
DATABASE_URI = "postgresql://postgres:postgres#localhost:5432/dbname"
# sqlalchemy 1.4+ uses postgresql:// instead of postgres://
ssl_mode = "?sslmode=require"
DATABASE_URI += ssl_mode
engine = sqlalchemy.create_engine(URI)
Session = sqlalchemy.orm.sessionmaker(bind=engine)
There's a nifty figure (Table 33.1) in the postgres documentation on SSL Support that breaks down the different options you can supply. If you want to use any of the fancier options that require you to specify a path to a specific certificate, you can drop it in with a format string.

ssh first with mysqldb in python

I'm trying to connect to a MySQL database on a remote server using MySQLdb in python. The problem is that first I need to SSH into the host, and then from there, I need to connect to the MySQL server. The problem I'm having, though, is that MySQLdb does not seem to have a way of establishing an SSH connection before connecting to the SQL server. I've checked the documentation but have not had any luck.
This is how I'm connecting:
conn = MySQLdb.connect(host = 'mysqlhost.domain.com:3306', user = 'user', passwd = 'password', db = 'dbname')
But what I really need is something like this:
conn = MySQLdb.connect(sshhost = 'sshhost.domain.com', sshuser = 'sshusername', sshpasswd = 'sshpasswd', host = 'mysqlhost.domain.com:3306', user = 'user', passwd = 'password', db = 'dbname')
Which is of course just made up. Can anyone make any recommendations?

I prefer keeping the tunnel within the python code, I did hate to create tunnels manually, or separately, thanks to sshtunnel library its very simple to use.
Here is some simple sample that will work for what you want.
import MySQLdb
from sshtunnel import SSHTunnelForwarder
with SSHTunnelForwarder(
('sshhost.domain.com', 22),
ssh_password="sshpasswd",
ssh_username="sshusername",
remote_bind_address=('mysqlhost.domain.com', 3306)) as server:
conn = MySQLdb.connect(host='127.0.0.1',
port=server.local_bind_port,
user='user',
passwd='password',
db='dbname')

Setup an ssh tunnel before you use MySQLdb.connect. The tunnel will make it appear as though you have the mysql running locally, set it up something like this
ssh user#host.com -L 9990:localhost:3306
here your local port 9990 will bind to 3306 on the remote host, -L stands for local, then 9990:localhost:3306 means LOCALPORT:
conn = MySQLdb.connect(host = 'mysqlhost.domain.com:9990', user = 'user', passwd = 'password', db = 'dbname')
notice the 9990.
Add your public ssh key of user to the host.com so that you dont have to type the password each time you want to setup the tunnel (use public key authentication).
If you need to do this within python there is python-to-ssh binding libraries you could call from within python to setup the tunnel for you.