I'm trying to connect to Cloud SQL from a Python application (using PyMySQL 0.7.9) running on top of Google App Engine.
My connection string looks like this (credentials are fake of course):
pymysql.connect(unix_socket='/cloudsql/gae_project_name:cloudsql_instance_name',
user='user', password='', db='database_name')
The error message I receive is:
OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 97] Address family not supported by protocol)")
It's like PyMySQL doesn't recognize that I'm trying to connect through a Unix socket and tries the default value for the host argument instead (which I presume is localhost)
I am able to connect with MySQLdb with the same connection string.
Why don't use MySQLdb instead then ?
I just had the same problem deploying a Flask application with PyMySQL, I tried a lot of fixes without success. My workaround was to use MySQLDb instead aha..!
Apparently, PyMySQL is not currently supported on the Google App Engine Standard environment, which only runs Python 2.7 (as of June 2018). This is from the maintainers of the GCP python project:
I can confirm that pymysql is not supported in the python27 runtime. However, for most use cases, it's possible to use pymysql locally and mysqldb in production by using a try: / except ImportError: to import one or the other conditionally. As they share the same interface, you can use import as to make the two different libraries share the same name for ease of use in your code.
See this Github thread for details
I got it to connect using a Python3 Flex App Engine and PyMySQL.
Here's what my app.yaml looks like:
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT api:app --timeout 180
runtime_config:
python_version: 3
env_variables:
SQLALCHEMY_DATABASE_URI: >-
mysql+pymysql://user:password#/database?unix_socket=/cloudsql/project-name:us-central1:instance-name
beta_settings:
cloud_sql_instances: us-central1:instance-name
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/python/configuring-your-app-with-app-yaml
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
Make sure to replace the user, password, database, and instance connection in both the env_variables and the beta_settings.
Here's my python:
import pymysql.cursors
import pymysql
connection = pymysql.connect(unix_socket='/cloudsql/project-name:us-central1:instance-name',
user='user',
password='password',
db='database',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
Related
I'm using Macbook
Psycopg2 works well when connecting the localhost db (PostgreSQL on Mac). The error was raised when I tried to connect PostgreSQL db on a Windows10.
the following code is what I have for connection, the host is just the IP of the windows10
db= psycopg2.connect(database='dbname',user='username',password="secret",host="192.168.3.9",port="5432")
Errors:
File "path/to/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: unsupported frontend protocol 1234.5679: server supports 2.0 to 3.0
Is this because of system compatibility or something else? I've tried other Windows machine and I got no luck to make it work.
However, I was able to connect PostgreSQL on windows while I using Node.JS module pg
1234.5679 is the special code sent by the client to request SSL-encrypted database connections, and support for that has been in PostgreSQL since commit e0e7daef6da in 1999. But your PostgreSQL cannot be that old, because support for protocol version 3.0 was not added before 2003.
Actually, from studying src/backend/postmaster/postmaster.c and reading the mailing list, this is a bug on the PostgreSQL server:
The client must be configured to try GSS authentication, and when the server rejects, it wants to negotiate an SSL connections, but the server doesn't expect that at this point; hence the error.
See the discussion here. The bug has been fixed with release 12.3.
As a workaround, disable either GSS authentication or SSL negotiation on the client.
In psycopg2, disabling SSL is done by using sslmode="disable" in the connection string, and disabling GSS is done with gssencmode="disable". See the documentation for details.
Adding ?gssencmode=disable to the connection string worked for me:
import pyodbc
from sqlalchemy import create_engine
engine = create_engine(f'postgresql://{user}:{password}#localhost:5432/database_name?gssencmode=disable')
Getting a similar error working with Laravel and Postgres. Solved it by putting this in my .env file: PGGSSENCMODE=disable
If you installed this Psycopg2 module through conda command, then uninstall that module and install using pip command.
Command:
pip install Psycopg2
The above command may resolve your issue. I resolved through this step
In order to have a SSL connection working and using certificates, you can disable GSS connection mode to avoid the client to attempt to connect with GSS and directly try a SSL connection (Postgres version 12 < 12.3, I had no such issues with a test on version 11).
Below an example with the verify-ca option, providing filenames for certificates and key (example names from GCP Cloud SQL, running Postgres 12.1 when posted).
from sqlalchemy import create_engine
db_user = 'user'
db_pwd = 'secret'
db_host = 'hostname'
db_port = '5432'
db_name = 'test'
cnn = f'postgresql://{db_user}:{db_pwd}#{db_host}:{db_port}/{db_name}'
ssl_args = {
'gssencmode': 'disable',
'sslmode': 'verify-ca',
'sslrootcert': 'server-ca.pem',
'sslcert': 'client-cert.pem',
'sslkey': 'client-key.pem',
}
engine = create_engine(cnn, connect_args=ssl_args)
This engine can then be used with pandas for example:
df.to_sql('my_table', con=engine)
Using PostgresSQL 13.0 I had the same problem, displaying the error messages:
unsupported frontend protocol 255.255: server supports 2.0
unsupported frontend protocol 0.0: server supports 2.0 to
according to this site it postgresql.org deals with the SSL / GSS Protocol Negotiation Problem. Which should already be resolved in Postgres version 12.0
I was able to solve it following the guidelines contained in these sites:
highgo.ca
In the postgres terminal, I executed the command below setting the environment variable gssencmode = disable and the problem was solved:
psql -h localhost -U postgres -d "dbname=belez gssencmode=disable";
I'm using Macbook
Psycopg2 works well when connecting the localhost db (PostgreSQL on Mac). The error was raised when I tried to connect PostgreSQL db on a Windows10.
the following code is what I have for connection, the host is just the IP of the windows10
db= psycopg2.connect(database='dbname',user='username',password="secret",host="192.168.3.9",port="5432")
Errors:
File "path/to/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: unsupported frontend protocol 1234.5679: server supports 2.0 to 3.0
Is this because of system compatibility or something else? I've tried other Windows machine and I got no luck to make it work.
However, I was able to connect PostgreSQL on windows while I using Node.JS module pg
1234.5679 is the special code sent by the client to request SSL-encrypted database connections, and support for that has been in PostgreSQL since commit e0e7daef6da in 1999. But your PostgreSQL cannot be that old, because support for protocol version 3.0 was not added before 2003.
Actually, from studying src/backend/postmaster/postmaster.c and reading the mailing list, this is a bug on the PostgreSQL server:
The client must be configured to try GSS authentication, and when the server rejects, it wants to negotiate an SSL connections, but the server doesn't expect that at this point; hence the error.
See the discussion here. The bug has been fixed with release 12.3.
As a workaround, disable either GSS authentication or SSL negotiation on the client.
In psycopg2, disabling SSL is done by using sslmode="disable" in the connection string, and disabling GSS is done with gssencmode="disable". See the documentation for details.
Adding ?gssencmode=disable to the connection string worked for me:
import pyodbc
from sqlalchemy import create_engine
engine = create_engine(f'postgresql://{user}:{password}#localhost:5432/database_name?gssencmode=disable')
Getting a similar error working with Laravel and Postgres. Solved it by putting this in my .env file: PGGSSENCMODE=disable
If you installed this Psycopg2 module through conda command, then uninstall that module and install using pip command.
Command:
pip install Psycopg2
The above command may resolve your issue. I resolved through this step
In order to have a SSL connection working and using certificates, you can disable GSS connection mode to avoid the client to attempt to connect with GSS and directly try a SSL connection (Postgres version 12 < 12.3, I had no such issues with a test on version 11).
Below an example with the verify-ca option, providing filenames for certificates and key (example names from GCP Cloud SQL, running Postgres 12.1 when posted).
from sqlalchemy import create_engine
db_user = 'user'
db_pwd = 'secret'
db_host = 'hostname'
db_port = '5432'
db_name = 'test'
cnn = f'postgresql://{db_user}:{db_pwd}#{db_host}:{db_port}/{db_name}'
ssl_args = {
'gssencmode': 'disable',
'sslmode': 'verify-ca',
'sslrootcert': 'server-ca.pem',
'sslcert': 'client-cert.pem',
'sslkey': 'client-key.pem',
}
engine = create_engine(cnn, connect_args=ssl_args)
This engine can then be used with pandas for example:
df.to_sql('my_table', con=engine)
Using PostgresSQL 13.0 I had the same problem, displaying the error messages:
unsupported frontend protocol 255.255: server supports 2.0
unsupported frontend protocol 0.0: server supports 2.0 to
according to this site it postgresql.org deals with the SSL / GSS Protocol Negotiation Problem. Which should already be resolved in Postgres version 12.0
I was able to solve it following the guidelines contained in these sites:
highgo.ca
In the postgres terminal, I executed the command below setting the environment variable gssencmode = disable and the problem was solved:
psql -h localhost -U postgres -d "dbname=belez gssencmode=disable";
I'm using Macbook
Psycopg2 works well when connecting the localhost db (PostgreSQL on Mac). The error was raised when I tried to connect PostgreSQL db on a Windows10.
the following code is what I have for connection, the host is just the IP of the windows10
db= psycopg2.connect(database='dbname',user='username',password="secret",host="192.168.3.9",port="5432")
Errors:
File "path/to/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: unsupported frontend protocol 1234.5679: server supports 2.0 to 3.0
Is this because of system compatibility or something else? I've tried other Windows machine and I got no luck to make it work.
However, I was able to connect PostgreSQL on windows while I using Node.JS module pg
1234.5679 is the special code sent by the client to request SSL-encrypted database connections, and support for that has been in PostgreSQL since commit e0e7daef6da in 1999. But your PostgreSQL cannot be that old, because support for protocol version 3.0 was not added before 2003.
Actually, from studying src/backend/postmaster/postmaster.c and reading the mailing list, this is a bug on the PostgreSQL server:
The client must be configured to try GSS authentication, and when the server rejects, it wants to negotiate an SSL connections, but the server doesn't expect that at this point; hence the error.
See the discussion here. The bug has been fixed with release 12.3.
As a workaround, disable either GSS authentication or SSL negotiation on the client.
In psycopg2, disabling SSL is done by using sslmode="disable" in the connection string, and disabling GSS is done with gssencmode="disable". See the documentation for details.
Adding ?gssencmode=disable to the connection string worked for me:
import pyodbc
from sqlalchemy import create_engine
engine = create_engine(f'postgresql://{user}:{password}#localhost:5432/database_name?gssencmode=disable')
Getting a similar error working with Laravel and Postgres. Solved it by putting this in my .env file: PGGSSENCMODE=disable
If you installed this Psycopg2 module through conda command, then uninstall that module and install using pip command.
Command:
pip install Psycopg2
The above command may resolve your issue. I resolved through this step
In order to have a SSL connection working and using certificates, you can disable GSS connection mode to avoid the client to attempt to connect with GSS and directly try a SSL connection (Postgres version 12 < 12.3, I had no such issues with a test on version 11).
Below an example with the verify-ca option, providing filenames for certificates and key (example names from GCP Cloud SQL, running Postgres 12.1 when posted).
from sqlalchemy import create_engine
db_user = 'user'
db_pwd = 'secret'
db_host = 'hostname'
db_port = '5432'
db_name = 'test'
cnn = f'postgresql://{db_user}:{db_pwd}#{db_host}:{db_port}/{db_name}'
ssl_args = {
'gssencmode': 'disable',
'sslmode': 'verify-ca',
'sslrootcert': 'server-ca.pem',
'sslcert': 'client-cert.pem',
'sslkey': 'client-key.pem',
}
engine = create_engine(cnn, connect_args=ssl_args)
This engine can then be used with pandas for example:
df.to_sql('my_table', con=engine)
Using PostgresSQL 13.0 I had the same problem, displaying the error messages:
unsupported frontend protocol 255.255: server supports 2.0
unsupported frontend protocol 0.0: server supports 2.0 to
according to this site it postgresql.org deals with the SSL / GSS Protocol Negotiation Problem. Which should already be resolved in Postgres version 12.0
I was able to solve it following the guidelines contained in these sites:
highgo.ca
In the postgres terminal, I executed the command below setting the environment variable gssencmode = disable and the problem was solved:
psql -h localhost -U postgres -d "dbname=belez gssencmode=disable";
My goal is to take a working Python 2.7 project (MySQL + MS Word files) to work at GCP.
I realize that I need
App Engine - where the app will be running (scaling, etc).
Cloud SQL working as MySQL db.
For that I've followed that Cloud SQL for MySQL tut and
Cloud SQL instance is created with root user.
Both App Engine app and Cloud SQL instance are in the same project.
Cloud Storage
The SQL second generation instance is successfully created and a root user is set.
How I run or deploy
I use Cloud Shell to test the app - dev_appserver.py $PWD and deploy the app from Cloud Shell - gcloud app deploy. It works at appspot.com till I try to use MySQL connection in it.
MySQL connection
The MySQL connection code is taken from here:
import MySQLdb
import webapp2
CLOUDSQL_CONNECTION_NAME = os.environ.get('CLOUDSQL_CONNECTION_NAME')
CLOUDSQL_USER = os.environ.get('CLOUDSQL_USER')
CLOUDSQL_PASSWORD = os.environ.get('CLOUDSQL_PASSWORD')
DB_NAME='test-db'
def connect_to_cloudsql():
# When deployed to App Engine, the `SERVER_SOFTWARE` environment variable
# will be set to 'Google App Engine/version'.
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
# Connect using the unix socket located at
# /cloudsql/cloudsql-connection-name.
cloudsql_unix_socket = os.path.join(
'/cloudsql', CLOUDSQL_CONNECTION_NAME)
db = MySQLdb.connect(
unix_socket=cloudsql_unix_socket,
user=CLOUDSQL_USER,
passwd=CLOUDSQL_PASSWORD)
# If the unix socket is unavailable, then try to connect using TCP. This
# will work if you're running a local MySQL server or using the Cloud SQL
# proxy, for example:
#
# $ cloud_sql_proxy -instances=your-connection-name=tcp:3306
#
else:
db = MySQLdb.connect(
host='127.0.0.1', user=CLOUDSQL_USER, passwd=CLOUDSQL_PASSWORD, db=DB_NAME)
return db
db = connect_to_cloudsql()
Variables are set in app.yaml:
runtime: python27
api_version: 1
threadsafe: true
env_variables:
CLOUDSQL_CONNECTION_NAME: coral-heuristic-215610:us-central1:db-basic-1
CLOUDSQL_USER: root
CLOUDSQL_PASSWORD: xxxxx
When app is run in test mode thru dev_appserver.py $PWD and I choose to use MySQL connection I got an error:
ERROR 2018-09-13 08:37:42,492 wsgi.py:263]
Traceback (most recent call last):
...
File "/home/.../mysqldb.py", line 35, in connect_to_cloudsql
host='127.0.0.1', user=CLOUDSQL_USER, passwd=CLOUDSQL_PASSWORD)
File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (2003, 'Can\'t connect to MySQL server on \'127.0.0.1\' (111 "Connection refused")')
Cloud SQL Proxy
I've downloaded and run the Cloud Proxy for Win-64 (https://dl.google.com/cloudsql/cloud_sql_proxy_x64.exe ) yet still the problem persists... Seems that proxy background app is only for connection to Cloud SQL from my local machine.
You do not need to use the proxy or configure SSL to connect to Cloud SQL from the App Engine standard or flexible environment. (source)
Why is the connection refused?
Should I use rather first generation Cloud sql instance to simplify connection from App Engine?
Update 1
I edit code at the Cloud Console and so far Cloud Console works good.
Update 2
I've succeded to conenect to the sql instance thru Cloud Shell:
(coral-heuristic-215610)$ gcloud sql connect db-basic-1 --user=root
Whitelisting your IP for incoming connection for 5 minutes...done.
Connecting to database with SQL user [root].Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 48841
Server version: 5.7.14-google-log (Google)
MySQL [(none)]>
Update 3
The comment on a similar issue concerns the regions where Cloud SQL instance and App Engine app should be, that is in the same region.
In my case I've checked:
Cloud SQL instance to connect to: us-central1-a
App Engine app: us-central
Are these of one region? - turned out these of one region.
Update 4
I could have figured out to open db connection:
DB connection: <_mysql.connection open to '127.0.0.1' at 7f628c02bc00>
But this seems happened only after I've opened another Cloud Shell instance with the same project (coral-heuristic-215610). At that instance I've started connection to SQL instance and it was successful:
(coral-heuristic-215610)$ gcloud sql connect db-basic-1 --user=root
Whitelisting your IP for incoming connection for 5 minutes...done.
Connecting to database with SQL user [root].Enter password:
I guess that the first cloud shell instance started to connect to db because the second instance has white-listed my IP, isn't it?
The GAE app and Google Cloud SQL instance have to be deployed in the same region if you’re using MySQL First Generation, otherwise, I verified that they can be in different regions as long as you’re using MySQL Second Generation.
I had trouble understanding where are you trying to connect from. I assume you want to connect from the Google Cloud Shell using the proxy and the Cloud SDK Credentials. According to the documentation regarding Cloud SQL Proxy:
The Cloud SQL Proxy provides secure access to your Cloud SQL Second
Generation instances without having to whitelist IP addresses or
configure SSL.
The Cloud SQL Proxy works by having a local client, called the proxy,
running in the local environment. Your application communicates with
the proxy with the standard database protocol used by your database.
The proxy uses a secure tunnel to communicate with its companion
process running on the server.
Remember, since you’re not deploying the application, it is not using the environment variables you have established in the app.yaml. Therefore you have to export and set them yourself in the local machine:
export CLOUDSQL_CONNECTION_NAME=your-connection-name
export CLOUDSQL_USER=root
export CLOUDSQL_PASSWORD=xxxxx
Verify that they are set by doing e.g echo $CLOUDSQL_CONNECTION_NAME. When you deploy your app with gcloud app deploy, this is not necessary since GAE sets the whatever environment variables are specified in the app.yaml.
The proxy has to launched before trying to establish a connection following these steps:
Download the proxy with:
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
Give it execution permission with:
chmod +x cloud_sql_proxy
Start the proxy replacing <INSTANCE_CONNECTION_NAME> with your Cloud SQL instance connection name:
./cloud_sql_proxy -instances=<INSTANCE_CONNECTION_NAME>=tcp:3306
You should see at the end something similar to this:
2018/11/09 13:24:32 Rlimits for file descriptors set to {&{8500 1048576}}
2018/11/09 13:24:35 Listening on 127.0.0.1:3306 for my-project:cloud-sql-region:cloud-sql-name
2018/11/09 13:24:35 Ready for new connections
At this point, is when you can connect to the proxy running locally from the Google Cloud Shell instance, which in turn will connect you to the Cloud SQL instance.
Open another Cloud Shell session (or tab) and launch your code with python myapp.py. You will be connected to the proxy running locally. You can also test the connection by running mysql -h 127.0.0.1 --user=root -p.
I solved the connection refused problem by adding to the app.yaml the following:
beta_settings:
cloud_sql_instances: "<CONNECTION_NAME>"
I have a Django app (with a postgresql backend) which I used to test on a local development server, and then simply push to Heroku (my service of choice). I had a Procfile that told Heroku dynos what processes to spin up, and then never worried about anything else.
I'm now migrating to Azure, where I'm setting up my own VM (v1) to host my app + postgres db. Now I need to set up my own webserver as well, which I unfortunately have thin experience of. So can someone guide me how to set up my own webserver? My ultimate goal is to set up Gunicorn with NginX as a reverse proxy. The first step, though, is to set up just Gunicorn for starters, and start seeing some HTTP traffic. How do I do that?
Here's my directory structure:
app/
__init__.py
models.py
tests.py
views.py
project/
__init__.py
wsgi.py
urls.py
settings.py
celery.py
static/
templates/
middleware/
The contents of my wsgi.py file are:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())
I tried the following:
gunicorn -b 0.0.0.0:8080 project.wsgi:application
It fires up; I see the following:
I likewise added endpoints in my Azure VM like so:
I likewise added endpoints for port 5432. Finally, when I run this I get the error:
Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5432?. Can you point out what I'm doing wrong?
According your description, it seems a PostgreSQL configuration issue. As I have a quick test on ubuntu + PostgreSQL + python27 + psycopg2 to test the connection to local PostgreSQL on Azure VM. However, I cannot reproduce your issue, it works fine on my side. Here are my steps of installing PostgreSQL service and test on python, for your information:
1, remote to Azure VM, and install PostgreSQL via sudo apt-get update, sudo apt-get install PostgreSQL postgresql-contrib
2, create a new PostgreSQL user: psql , ` create user someuser password 'somepassword';
3, install PostgreSQL service which is required by psycopg2 : sudo apt-get install postgresql-server-dev-9.4
4, install python package psycopg2: sudo apt-get install python-pip and sudo apt-get install python-psycopg2
5, And test in python code:
import psycopg2
try:
conn = psycopg2.connect("dbname='postgres' user=' someuser' host='localhost' password='somepassword'")
cur = conn.cursor()
cur.execute("""SELECT 1=1""")
rows = cur.fetchall()
for row in rows:
print " ", row[0]
except:
print "I am unable to connect to the database"
And specific to your issue, I think you can quick check the following points for troubleshooting:
1, make sure your PostgreSQL service is running, because of here is an issue with the same error message with you Posgresql connection through port 5432.
2, make sure the host setting of PostgreSQL in your Django app is “localhost”, and you can easy test in psql –h localhost -Usomeuser to login on PostgreSQL. If you occur the issue, you can try to configure your PostgreSQL server shown in Postgresql and TCP/IP connections on port 5432.