My current connection configuration is as follow, this is for redshift db
con=('postgresql://username:password#hostname:port/databasename')
server = SSHTunnelForwarder(
('ssh host', 22),
ssh_username="-",
ssh_password="-",
remote_bind_address=('db host', port)
)
server.start()
local_port = str(server.local_bind_port)
engine = sa.create_engine(con)
######## Reaches here then times out when reading the table
df_read = pd.read_sql_table('tablename',engine)
However, the Redshift database also has an SSH, which might be affecting the connection? It creates the engine but when reading the SQL in pd.read_sql_query I reach this error.
(psycopg2.OperationalError) could not connect to server: Connection timed out (0x0000274C/10060)
Is the server running on host "xxx" (xxx) and accepting
TCP/IP connections on port xxx?
(Background on this error at: http://sqlalche.me/e/e3q8)
You cannot SSH into a Redshift cluster but you can use SSL to secure the connection.
Amazon Redshift security overview
Configuring security options for connections
Related
I want to create a temporary MySQL database for unit testing in Python. I decided to go with docker as the tests will be run on different machines.
As a first thing I run the container:
client = docker.from_env()
container = client.containers.run("ubuntu/mysql", detach=True, ports={3306:3307},environment={
'MYSQL_USER':'mysqluser',
'MYSQL_PASSWORD':'password',
'MYSQL_DATABASE':'mydb',
})
print('Container Started : {}'.format(container.status))
And then I try to connect using SQLAlchemy:
engine = sa.create_engine(f'mysql+pymysql://mysqluser:password#localhost:3307/mydb')
engine.connect()
which returns:
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
EDIT
After putting sleep(10) after container.run, the response is different:
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([Errno 111] Connection refused)")
Do you know what I'm doing wrong?
What I want to do is to connect to MySQL database on port 3307 (as one server already runs on the host). Then I want to create and populate a table.
EDIT2
Switching ports to {3307:3306} returns error:
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/containers/67a04b384bf3d39407881d4dfc73961932a3ce72fe8dc82c3cd6aedd57da2821/start: Internal Server Error ("driver failed programming external connectivity on endpoint hopeful_leakey (97840eb42d2c09b5e9754b634481d8e048c57ba2de8ffc10b34261de2f76600b): Error starting userland proxy: listen tcp4 0.0.0.0:3306: bind: address already in use")
I have a postgresql database hosted on my ec2 instance. I am trying to connect to it using Python from my local computer.
import psycopg2
try:
connection = psycopg2.connect(user="postgres",
password="<password>",
host="ec2-***-***-***-***.***-***-1.compute.amazonaws.com",
port="5432",
database="<db_name>")
cursor = connection.cursor()
cursor.execute(f"SELECT * FROM <tablename>;")
record = cursor.fetchall()
print(record, "\n")
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL", error)
finally:
try:
if connection:
cursor.close()
connection.close()
print("PostgreSQL connection is closed")
except:
print("no work")
But I get
Error while connecting to PostgreSQL could not connect to server: Connection timed out (0x0000274C/10060)
Is the server running on host "ec2-***-***-***-***.***-***-1.compute.amazonaws.com" (**.**.**.***) and accepting
TCP/IP connections on port 5432?
no work
My pg_hba.conf file looks like this
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 md5
host all all 0.0.0.0/0 md5
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres peer
#host replication postgres 127.0.0.1/32 ident
#host replication postgres ::1/128 ident
and my postgresql.conf file looks like
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------
# - Connection Settings -
listen_addresses = '*' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5432 # (change requires restart)
I have an ec2 Security Group:
What am I doing wrong? I am a complete newbie, any help appreciated!
Connection timeout usually indicates that you cannot reach the host or are being blocked by a firewall.
First try to ping the host from your machine. If it pings all right, then there is probably a firewall between you and your EC2 instance. Your EC2 security screenshot looks right to me.
Are you behind a firewall that might be blocking outbound sessions from your local computer to the Internet?
After some troubleshooting over chat, we found that the AWS Security Group allowing TCP/5432 inbound was not assigned to the EC2 instance.
There are many possible causes, so I would start with a trusted DB client like DBeaver and attempt to make the connection from your local machine to rule out python issues.
Depending on your setup, you may have a second incoming firewall (iptables, etc) running inside your ec2 instance that needs to be configured or disabled.
Log into the ec2 console and see if you can connect to the server. Load a db client like pgsql inside ec2 and attempt to connect to the server with localhost:5432 as the target.
You may need to alter pg_hba.conf so that the server will generate log files, but that can tell you if the server is being reached, and what the problem is.
I'm connecting successfully to my sqlexpress instance from sqlalchemy with the connection string:
connection_str = 'mssql+pyodbc://mylaptop\sqlexpress/master?driver=SQL+Server+Native+Client+11.0&trusted_connection=yes'
How would i connect to a sql instance on a non default port, is it possible?
connection_str = 'mssql+pyodbc://mylaptop\sqlexpress:1433/master?driver=SQL+Server+Native+Client+11.0&trusted_connection=yes'
is what i would expect the connection string to be but this returns the error:
(pyodbc.OperationalError) ('08001', '[08001] [Microsoft][SQL Server Native Client 11.0]TCP Provider: No connection could be made because the target machine actively refused it.\r\n (10061) (SQLDriverConnect); [08001] [Microsoft][SQL Server Native Client 11.0]Login timeout expired (0); [08001] [Microsoft][SQL Server Native Client 11.0]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (10061)
I've tried a number of combinations with the port but don't see a legal combination?
In a SQLAlchemy connection URI you can specify a SQL Server instance in one of two ways
server_name\instance_name, or
server_name:port
You should only be using one or the other. If you specify the instance_name then the port number is provided by the SQL Browser service on server_name. If you specify the port then the actual instance_name doesn't matter because the port number is already known.
I have problem to connect to my postgreSQL database.
I have databasename, password, hostname, port and I use this:
conn_string = "host='localhost' dbname='my_database' user='postgres' password='secret'"
But I got error like this:
Is the server running on host "...." and accepting TCP/IP connections on port 5432
I don't know if I used correctly host, I insert the value of hostname.
What is the difference between hostname and host? Anyone could help me?
psycopg2.connect(dbname=dbname, user=user, password=password, host=postgres_address, port=postgres_port)
This is working example to connect, you must define early dbname, user, password, postgres_address. If you have connection error, you can use ping for testing connection and telnet for testing openning port. Or you can use Beaver for test connection to postgres server.
Most likely your database has a firewall, be sure to whitelist the IP you are trying to connect from.
Difference of host and hostname
The difference of host and hostname really depends on the context. In your context of psycopg2 and PostgreSQL connection, the host normally means the IP address of the PostgreSQL server or the resolvable name of the PostgreSQL server such as DNS name if it has. If you are running linux server, the output of command hostname is unlikely to work in your case.
psycopg2 connection
Your connection string looks OK. But I will suggest you to use below connection format:
import psycopg2
try:
connection = psycopg2.connect(user = "sysadmin",
password = "pynative##29",
host = "127.0.0.1",
port = "5432",
database = "postgres_db")
except (Exception, psycopg2.Error) as error :
print ("Error while connecting to PostgreSQL", error)
You should use the IP address as the host value.
Troubleshooting
In the case of connection error, you should use other tools to test the connection of PostgreSQL server such as psql, pgAdmin4 or DBeaver.
You can also use telnet or netcat tools to test the network connection of PostgreSQL server, such as
telnet PostgreSQL_ip_address 5432
nc -v PostgreSQL_ip_address 5432
I'm tring to understand what I should do in order to connect to a remote MySQL server using SSH.
from sshtunnel import SSHTunnelForwarder
with SSHTunnelForwarder(
('192.168.0.60', 22),
ssh_password="MySSHPASSWORD",
ssh_username="MUSSHUserName",
remote_bind_address=('192.168.0.30', 3306)) as server:
conn = MySQLdb.connect(user='UserNameOnSQLSERVER', passwd='PasswordOnSQLSERVER', host='192.168.0.60', db='myDB',
port=server.local_bind_port)
I've been able to connect to this server with the above configuration throw MySQL Workbench and Run queries and everything worked great
The problem is that when I'm trying to connect this DB by the above code I keep getting error:
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on '192.168.0.60' (10060)")
I also tried to change the ip in MySQLdb.connect to be
conn = MySQLdb.connect(user='kiboUser', passwd='Lb24#db30#kL68', host='192.168.0.60', db='dbKiboDashboard',
port=server.local_bind_port)
and I got the following error:
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on '192.168.0.30' (10060)")
Can please someone help me understand what im doing wrong
Thanks
remote_bind_address is the one you are tunneling into. Make sure to verify the server's TCP port that is listening.
To verify this, you could try the command
netstat
or
netstat -tln
This should work:
with SSHTunnelForwarder(
('ServerIP', sshport), #The server's IP and SSH port
ssh_password="MySSHPASSWORD",
ssh_username="MUSSHUserName",
remote_bind_address=('127.0.0.1', 3306)) as server: #The server's TCP IP and port, you're tunneling
conn = MySQLdb.connect(user='UserNameOnSQLSERVER', passwd='PasswordOnSQLSERVER', host='127.0.0.1',
port=server.local_bind_port) #Your TCP IP and port (your end of the tunnel)