Develop Reference

Pandas read from remote Postgresql with SSH tunnel and sqlalchemy

I can read from my local psql instance like this:
engine = create_engine('postgresql://postgres:postgres#localhost/db_name')
df = pd.read_sql("select * from table_name;", engine)
I have a remote postgresql sever which I successfully accessed with ssh tunneling both in PgAdmin4 and pycharm. I use public key file to login into remote server. Now, my question is how do I access that database with pandas. I tried:
engine = create_engine('postgresql://username:password#localhost/db_name')
Here, username and password are of remote database. I get sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user. However, with the same username and password I can access the table in PgAdmin.
From what I read, because of ssh tunneling I have to use localhost and not the remote server address, right? In pgAdmin I can see that the server is running. So, my question is how do I read the table from remote postgresql database with ssh tunneling? In examples I have seen people using different port (different than 5432) but for me the setup only works if I use port 5432. I have disconnected all other servers to avoid the port conflict but I get the same error.

The tunnel created by pgAdmin4 is intended for its own use. It does not arrange for it to listen on 5432, it picks some arbitrary high numbered port and doesn't advertise what port that is. While you can discover what port it is listening on using system tools (like netstat) and then connect to it, you would probably be better served by finding some other way to set up your tunnel. There are python libraries that can help with that.
As for why you can connect to 5432 at all, clearly there is something listening there which is either PostgreSQL or pretending to be PostgreSQL, but it doesn't seem to be the one you intend. You can use netstat -ao to find the pid for it and then look up based on that.

Open Cassandra for remote connection only for single IP, and connect using python

I'm trying to write a script to connect to the remote Cassandra DB in python.
I'm using flask_cqlalchemy. Cassandra is set up on a remote server with SSH access and while connecting to the Cassandra, it's giving:
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'IP:9042': ConnectionRefusedError(61, "Tried connecting to [('IP', 9042)]. Last error: Connection refused")})
And 9042 port is open in the firewall.
I tried using "TablePlus tool" and it's connecting perfectly.

Your Cassandra node is configured to accept connections from localhost - to allow remote access you need to set rpc_address to the IP address of the host, but you need to be careful with it, as then anybody can access your Cassandra node (especially if it's in the cloud).
Plus, if you configure only rpc_address, then it won't accept connections on localhost - in this case you can set rpc_address to 0.0.0.0, and broadcast_rpc_address to node's IP.

Connecting through Python to a MySQL database hosted on a remote server with no direct access

I wish to connect to a database server through my local machine at work, but I do not have direct access to the database server (due to security reasons). The database server is accessible through another intermediary server which I can connect to.
I understand I can connect to the database if I run my script on the intermediary server, but is there any way through which I can connect to the database server directly through my local machine?
I am trying to do this in a Python script as I wish to read the data into a pandas dataframe (I can do this part once I can set up the connection).

If you have SSH access to that intermediary server you can connect via an SSH tunnel. This post describes how to do that: Enable Python to Connect to MySQL via SSH Tunnelling

How to secure client connections to an HBase Thrift Server?

Anyone who knows the port and host of a HBase Thrift server, and who has access to the network, can access HBase. This is a security risk. How can the client access to the HBase Thrift server be made secure?

You could secure HBase Thrift server setting up authentication via kerberos and then setting this property in hbase-site.xml
<name>hbase.thrift.security.qop</name>
<value>auth</value>
http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hbase_authentication.html

My sysadmin told me that in theory he could install an HBase Thrift Server on one of the Hadoop edge nodes that are blocked off, and only open the port to my server via ACLs. He however has no intention of doing this (and I do not either). As this is not a suitable answer I'll leave the question open.

Python connect to ssh mysql server

I'm writing a small python program locally as i don't have root access on the server. It basically does a lot of mysql queries using python MySQLdb module.
The thing is I cant use MySQLdb with the server, as the mysql server is hosted locally and I need to ssh into the server and then use mysql from there.
Is there any module available where I can connect to a mysql database via SSH.
At the moment I can connect to the mysql instance using SSH credentials (IP, User, Pass)
I also have the user/pass for the mysql instance and I'm pretty sure it runs on 127.0.0.1/localhost.

If you're set on Python, I would use paramiko:
https://github.com/paramiko/paramiko
This seems to be one of the most widely used Python SSH libraries. There are some other StackOverflow questions that address how to do this.
How to open an SSH tunnel using python?
SSH Tunnel for Python MySQLdb connection
The main idea is to create a tunnel with paramiko and then connect to the localhost port through which you are tunneling traffic to the remote server using the Python library MySQLdb.