Develop Reference

Python clickhouse_driver through ssh connection error

struggling for a week already with my issue, hope you'll be able to explain to me what is wrong here.
I'm trying to connect Clickhouse DB I have on the remote server, which is on the absolutely default settings. So I'm connecting to a remote server and creating a tunnel on my machine.
import sshtunnel as sshtunnel
from clickhouse_driver import connect
server = sshtunnel.SSHTunnelForwarder(
('host', 22),
ssh_username='username',
ssh_pkey="username.openssh",
ssh_private_key_password="password",
remote_bind_address=('localhost', 8123),
local_bind_address=('localhost', 555)
)
server.start()
conn = connect(host='localhost', port=555, database='ertb')
cursor = conn.cursor()
cursor.execute('SHOW TABLES')
cursor.fetchall()
server.stop()
and I receive this error
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/alerts/ssh_coonection.py", line 42, in <module>
cursor.execute('SHOW TABLES')
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\dbapi\cursor.py", line 102, in execute
**execute_kwargs
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\client.py", line 205, in execute
self.connection.force_connect()
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\connection.py", line 180, in force_connect
self.connect()
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\connection.py", line 256, in connect
return self._init_connection(host, port)
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\connection.py", line 237, in _init_connection
self.send_hello()
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\connection.py", line 325, in send_hello
write_binary_str(self.user, self.fout)
File "C:\Users\user\PycharmProjects\alerts\venv\lib\site-packages\clickhouse_driver\writer.py", line 19, in write_binary_str
text = text.encode('utf-8')
AttributeError: 'NoneType' object has no attribute 'encode'
I really tried to understand from where this NoneType object appears, but a bit stuck in code.

There are two issues:
is not passed user and password that leads to the error 'NoneType' object has no attribute 'encode'
used the wrong port to access to CH (clickhouse-driver is designed to communicate with ClickHouse server over native protocol (TCP) that by default use port 9000 for non-secure communication (see Issue connecting to Dockerized Clickhouse Server with Python driver for details))
import sshtunnel as sshtunnel
from clickhouse_driver import connect
with sshtunnel.SSHTunnelForwarder(
('localhost', 22),
ssh_username="root",
ssh_password="root",
remote_bind_address=('localhost', 9000)) as server:
local_port = server.local_bind_port
print(local_port)
conn = connect(f'clickhouse://default:#localhost:{local_port}/ertb')
#conn = connect(host='localhost', port=local_port, database='ertb', user='default', password='')
cursor = conn.cursor()
cursor.execute('SHOW TABLES')
print(cursor.fetchall())

psycopg2.OperationalError: FATAL: password authentication

I am new to python and postgresql. This is the first time i was trying to connect python to postgresql.But i am getting this error:
`Traceback (most recent call last):
File "/home/vishal/Courses/database/app.py", line 5, in <module>
user.save_to_db()
File "/home/vishal/Courses/database/user.py", line 13, in save_to_db
connection = psycopg2.connect(user='postgres', password='1234', database='learning', host='localhost')
File "/home/vishal/Courses/database/venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: password authentication failed for user "postgres"
FATAL: password authentication failed for user "postgres"`
Even i tried changing the password and creating new user with different password the result is same.
Where am i going wrong?

Not able to connect to remote database through ssh tunneling in python

I'm trying to connect to a remote database through ssh tunneling. But I am getting an error when I try to connect. Below is my code and the error
def query(q):
with SSHTunnelForwarder(
("bastion.example.co", 22),
ssh_username="ubuntu",
ssh_pkey="~/.ssh/id_rsa.pub",
remote_bind_address=("127.0.0.1", 3306)
) as server:
conn = mysql.connect(host="xyz.rds.amazonaws.com",
port=server.local_bind_port,
user="prod_db",
passwd="123abc",
db="prod_db")
return pd.read_sql_query(q, conn)
print(query('select * from abc_table'))
Error:
Traceback (most recent call last):
File "conn_mysql.py", line 20, in <module>
print(query('select * from abc_table'))
File "conn_mysql.py", line 11, in query
remote_bind_address=("127.0.0.1", 3306)
File "/Users/mohnishmallya/PycharmProjects/3Bvs2A/venv/lib/python3.7/site-packages/sshtunnel.py", line 904, in __init__
logger=self.logger
File "/Users/mohnishmallya/PycharmProjects/3Bvs2A/venv/lib/python3.7/site-packages/sshtunnel.py", line 1095, in _consolidate_auth
raise ValueError('No password or public key available!')
ValueError: No password or public key available!
How do I solve this?

Use ssh_pkey parameter of SSHTunnelForwarder constructor to provide the path to your private key file (not the public key .pub).

Peewee - Can't connect to MySQL server on host

I'm developing a Flask based python app using the peewee ORM. I was initially connecting to the database that was being stored locally on my machine and I'm now trying to transition to connecting to the db remotely. I've set up the database in phpmyadmin via my server's cpanel section.
The Issue
I've set up my IP address to be able to remotely access my databases but I am getting the following error when I attempt to connect to the database:
Traceback (most recent call last):
File "app.py", line 294, in <module>
models.initialize()
File "/Users/wyssuser/Desktop/dscraper/models.py", line 145, in initialize
DATABASE.connect()
File "/Library/Python/2.7/site-packages/peewee.py", line 2767, in connect
self.__local.closed = False
File "/Library/Python/2.7/site-packages/peewee.py", line 2688, in __exit__
reraise(new_type, new_type(*exc_value.args), traceback)
File "/Library/Python/2.7/site-packages/peewee.py", line 2766, in connect
**self.connect_kwargs)
File "/Library/Python/2.7/site-packages/peewee.py", line 3209, in _connect
return mysql.connect(db=database, **conn_kwargs)
File "/Library/Python/2.7/site-packages/pymysql/__init__.py", line 88, in Connect
return Connection(*args, **kwargs)
File "/Library/Python/2.7/site-packages/pymysql/connections.py", line 644, in __init__
self._connect()
File "/Library/Python/2.7/site-packages/pymysql/connections.py", line 869, in _connect
raise exc
peewee.OperationalError: (2003, "Can't connect to MySQL server on '142.157.25.22' ([Errno 61] Connection refused)")
This is the portion of my code that references the database connection:
app.py
if __name__ == '__main__':
models.initialize()
app.run(debug=DEBUG, port=PORT, host=HOST)
config.py
DATABASE = {
'db': 'my_dbname',
'host': '142.157.25.22',
'port': 3306,
'user': 'my_username',
'passwd': 'my_pswd',
}
models.py
from peewee import *
import config
DATABASE = MySQLDatabase(config.DATABASE['db'], host=config.DATABASE['host'], port=config.DATABASE['port'], user=config.DATABASE['user'], passwd=config.DATABASE['passwd'])
...all of my models related code
def initialize():
print 'starting db connection'
DATABASE.connect()
print 'connected'
DATABASE.create_tables([Batch, Company, User, Post],safe=True)
DATABASE.close()
I've also tried connecting to 'localhost' as the host but that doesn't seem to work here, is there a different host I should be connecting to?

Solution is bad default port:
the example from peewee doc is
# Connect to a MySQL database on network.
mysql_db = MySQLDatabase('my_app', user='app', password='db_password',
host='10.1.0.8', port=3316)
but defaul port is 3306

Connection appears accepted server-side but appears to be refused in my python script

I have a python script I am running to add a document to a collection for a remote mongodb database. The file is as follows:
from pymongo import Connection
ip = 'x.x.x.x' #edited out
conn = Connection()
db = conn['netmon']
users = db.users
print 'number of users: ' + str(users.count())
if users.count() == 0:
print 'Please create a new account.'
t_user = raw_input('Username:')
users.insert({'username':unicode(t_user)})
conn.close()
When I run this script, I have some interesting behavior. In the server logs, it appears that the connection is accepted, the query is run, and then the connection is closed.
# Mongodb logs
14:27:18 [initandlisten] connection accepted from 108.93.46.75:39558 #1
14:27:18 [conn1] run command admin.$cmd { ismaster: 1 }
14:27:18 [conn1] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:71 0ms
14:27:18 [conn1] run command netmon.$cmd { count: "users", fields: null, query: {} }
14:27:18 [conn1] Accessing: netmon for the first time
14:27:18 [conn1] command netmon.$cmd command: { count: "users", fields: null, query: {} } ntoreturn:1 reslen:58 10ms
14:27:18 [conn1] end connection 108.93.46.75:39558
However, this is what I get from the python script:
# Python error
> python testmongo.py
Traceback (most recent call last):
File "testmongo.py", line 7, in <module>
conn = Connection()
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 290, in __init__
self.__find_node()
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 586, in __find_node
raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: could not connect to localhost:27017: [Errno 111] Connection refused
charles#charles-mbp:~/Desktop$ python testmongo.py
Traceback (most recent call last):
File "testmongo.py", line 7, in <module>
conn = Connection()
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 290, in __init__
self.__find_node()
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 586, in __find_node
raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: could not connect to localhost:27017: [Errno 111] Connection refused
charles#charles-mbp:~/Desktop$ python testmongo.py
Traceback (most recent call last):
File "testmongo.py", line 7, in <module>
conn = Connection()
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 290, in __init__
self.__find_node()
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.2.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 586, in __find_node
raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: could not connect to localhost:27017: [Errno 111] Connection refused
Some useful information:
The remote server is a VPS running on OpenVZ. I know there are compatibility problems related to count() and iterators, but I (assumingly) fixed them by using the --smallfiles option, as well as limiting my nssize to 100. My firewall on both sides are completely open.
I've also noticed that in the errors, autoreconnect is trying to connect to localhost, even though I specified a different IP address.

You need to pass the IP - conn = Connection(ip)