My goal is to take a working Python 2.7 project (MySQL + MS Word files) to work at GCP.
I realize that I need
App Engine - where the app will be running (scaling, etc).
Cloud SQL working as MySQL db.
For that I've followed that Cloud SQL for MySQL tut and
Cloud SQL instance is created with root user.
Both App Engine app and Cloud SQL instance are in the same project.
Cloud Storage
The SQL second generation instance is successfully created and a root user is set.
How I run or deploy
I use Cloud Shell to test the app - dev_appserver.py $PWD and deploy the app from Cloud Shell - gcloud app deploy. It works at appspot.com till I try to use MySQL connection in it.
MySQL connection
The MySQL connection code is taken from here:
import MySQLdb
import webapp2
CLOUDSQL_CONNECTION_NAME = os.environ.get('CLOUDSQL_CONNECTION_NAME')
CLOUDSQL_USER = os.environ.get('CLOUDSQL_USER')
CLOUDSQL_PASSWORD = os.environ.get('CLOUDSQL_PASSWORD')
DB_NAME='test-db'
def connect_to_cloudsql():
# When deployed to App Engine, the `SERVER_SOFTWARE` environment variable
# will be set to 'Google App Engine/version'.
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
# Connect using the unix socket located at
# /cloudsql/cloudsql-connection-name.
cloudsql_unix_socket = os.path.join(
'/cloudsql', CLOUDSQL_CONNECTION_NAME)
db = MySQLdb.connect(
unix_socket=cloudsql_unix_socket,
user=CLOUDSQL_USER,
passwd=CLOUDSQL_PASSWORD)
# If the unix socket is unavailable, then try to connect using TCP. This
# will work if you're running a local MySQL server or using the Cloud SQL
# proxy, for example:
#
# $ cloud_sql_proxy -instances=your-connection-name=tcp:3306
#
else:
db = MySQLdb.connect(
host='127.0.0.1', user=CLOUDSQL_USER, passwd=CLOUDSQL_PASSWORD, db=DB_NAME)
return db
db = connect_to_cloudsql()
Variables are set in app.yaml:
runtime: python27
api_version: 1
threadsafe: true
env_variables:
CLOUDSQL_CONNECTION_NAME: coral-heuristic-215610:us-central1:db-basic-1
CLOUDSQL_USER: root
CLOUDSQL_PASSWORD: xxxxx
When app is run in test mode thru dev_appserver.py $PWD and I choose to use MySQL connection I got an error:
ERROR 2018-09-13 08:37:42,492 wsgi.py:263]
Traceback (most recent call last):
...
File "/home/.../mysqldb.py", line 35, in connect_to_cloudsql
host='127.0.0.1', user=CLOUDSQL_USER, passwd=CLOUDSQL_PASSWORD)
File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (2003, 'Can\'t connect to MySQL server on \'127.0.0.1\' (111 "Connection refused")')
Cloud SQL Proxy
I've downloaded and run the Cloud Proxy for Win-64 (https://dl.google.com/cloudsql/cloud_sql_proxy_x64.exe ) yet still the problem persists... Seems that proxy background app is only for connection to Cloud SQL from my local machine.
You do not need to use the proxy or configure SSL to connect to Cloud SQL from the App Engine standard or flexible environment. (source)
Why is the connection refused?
Should I use rather first generation Cloud sql instance to simplify connection from App Engine?
Update 1
I edit code at the Cloud Console and so far Cloud Console works good.
Update 2
I've succeded to conenect to the sql instance thru Cloud Shell:
(coral-heuristic-215610)$ gcloud sql connect db-basic-1 --user=root
Whitelisting your IP for incoming connection for 5 minutes...done.
Connecting to database with SQL user [root].Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 48841
Server version: 5.7.14-google-log (Google)
MySQL [(none)]>
Update 3
The comment on a similar issue concerns the regions where Cloud SQL instance and App Engine app should be, that is in the same region.
In my case I've checked:
Cloud SQL instance to connect to: us-central1-a
App Engine app: us-central
Are these of one region? - turned out these of one region.
Update 4
I could have figured out to open db connection:
DB connection: <_mysql.connection open to '127.0.0.1' at 7f628c02bc00>
But this seems happened only after I've opened another Cloud Shell instance with the same project (coral-heuristic-215610). At that instance I've started connection to SQL instance and it was successful:
(coral-heuristic-215610)$ gcloud sql connect db-basic-1 --user=root
Whitelisting your IP for incoming connection for 5 minutes...done.
Connecting to database with SQL user [root].Enter password:
I guess that the first cloud shell instance started to connect to db because the second instance has white-listed my IP, isn't it?
The GAE app and Google Cloud SQL instance have to be deployed in the same region if you’re using MySQL First Generation, otherwise, I verified that they can be in different regions as long as you’re using MySQL Second Generation.
I had trouble understanding where are you trying to connect from. I assume you want to connect from the Google Cloud Shell using the proxy and the Cloud SDK Credentials. According to the documentation regarding Cloud SQL Proxy:
The Cloud SQL Proxy provides secure access to your Cloud SQL Second
Generation instances without having to whitelist IP addresses or
configure SSL.
The Cloud SQL Proxy works by having a local client, called the proxy,
running in the local environment. Your application communicates with
the proxy with the standard database protocol used by your database.
The proxy uses a secure tunnel to communicate with its companion
process running on the server.
Remember, since you’re not deploying the application, it is not using the environment variables you have established in the app.yaml. Therefore you have to export and set them yourself in the local machine:
export CLOUDSQL_CONNECTION_NAME=your-connection-name
export CLOUDSQL_USER=root
export CLOUDSQL_PASSWORD=xxxxx
Verify that they are set by doing e.g echo $CLOUDSQL_CONNECTION_NAME. When you deploy your app with gcloud app deploy, this is not necessary since GAE sets the whatever environment variables are specified in the app.yaml.
The proxy has to launched before trying to establish a connection following these steps:
Download the proxy with:
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
Give it execution permission with:
chmod +x cloud_sql_proxy
Start the proxy replacing <INSTANCE_CONNECTION_NAME> with your Cloud SQL instance connection name:
./cloud_sql_proxy -instances=<INSTANCE_CONNECTION_NAME>=tcp:3306
You should see at the end something similar to this:
2018/11/09 13:24:32 Rlimits for file descriptors set to {&{8500 1048576}}
2018/11/09 13:24:35 Listening on 127.0.0.1:3306 for my-project:cloud-sql-region:cloud-sql-name
2018/11/09 13:24:35 Ready for new connections
At this point, is when you can connect to the proxy running locally from the Google Cloud Shell instance, which in turn will connect you to the Cloud SQL instance.
Open another Cloud Shell session (or tab) and launch your code with python myapp.py. You will be connected to the proxy running locally. You can also test the connection by running mysql -h 127.0.0.1 --user=root -p.
I solved the connection refused problem by adding to the app.yaml the following:
beta_settings:
cloud_sql_instances: "<CONNECTION_NAME>"
Related
I am relatively new to this topic. I try to build a webapp using flask. The webapp uses data from a postgresql database which is running local (Mac OS Monterey 12.2.1).
My application uses a python code which accesses data from the database by connecting to the database with psycopg2:
con = psycopg2.connect(
host = "192.168.178.43"
database = self.database,
port = "5432",
user = "user",
password = "password")
I already added the relevant entries to the "pg_hba.conf" file and to the "postgresql.conf" file to the needed configurations for an access in my home network. But i still got an error when starting the container. The app runs perfect outside the container. I think I miss some important steps to complete a successful connection.
This error is the following
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
while running the rest api which is deployed in heroku. I am getting below error which says not able to access mongodb
Currently i am accessing mongodb using pymongo.MongoClient("mongodb://localhost:27017/") ,may i know how to configure this ,so that i can access the db .
Thanks
Sumesh
Since you said that you haven't hosted the DB somewhere, I'll explain how to set things up with mLab and get the DB up and running.
Create an account in mLab and create a database in it.
Then create a user inside that database. We can use this to deploy your local database to mLab
Create a dump of your DB
mongodump -d <DB_NAME>
Then restore the db to your mLab instance
mongorestore -h <DB_URL> -u <DB_USERNAME> -p <DB_PASSWORD> --authenticationDatabase <MLAB_DB_NAME> -d <LOCAL_DB_NAME> <DB_DUMP_LOCATION>
After successfully deploying the db, you can set the connection string in Heroku
heroku config:set MONGOLAB_URI=mongodb://username:password#<DB_URL>/<DB_NAME>
Then, set the connection string in python application
import os
pymongo.MongoClient(os.environ['MONGOLAB_URI'])
For more info:
https://medium.com/miguel-garcia/heroku-and-mlab-with-mongodb-free-the-easy-way-ec2ae80073f7
https://forum.freecodecamp.org/t/how-to-deploy-your-mongodb-app-to-heroku/19347
I have to access the MongoDB and I can easily access it remotely using Pymongo in a jupyter notebook when I run the sudo openvpn --config client.ovpn in my terminal and then connectiong the Pymongo like:
client = MongoClient(host='host_ip',port=port_num)
db = client['db_name']
db.authenticate(name='username',password='password')
But how can I do the same when I want to access the DB over a cloud?
I have tried
client = MongoClient(host='host_ip',port=port_num,ssl=True,
ssl_certfile='UserCertificate.pem',
ssl_keyfile='UserPrivateKey.key',)
db = client['db_name']
db.authenticate(name='username',password='password')
It throws error after the last line of code while authenticating
ServerSelectionTimeoutError: host_ip:port_num: timed out
Running a small web app on Google App Engine (Flexible) using Python 3, Flask, SQLAlchemy, the psycopg2-binary package, and CloudSQL Postgres. The app connects to CloudSQL properly in my local dev environment (I use the proxy), but it won't seem to connect when deployed in the cloud.
Worked fine the first time I deployed it last week. After pushing an update and re-deploying, the app no longer connects to CloudSQL in the cloud shell or when deployed -- even after rolling back to that previous commit. But it connects just fine on my local dev machine using the proxy.
I've verified through debugging that the necessary credentials are being properly inserted in my SQLALCHEMY_DATABASE_URI, which is: postgres+psycopg2://[user]:[pw]#/[db_name]?host=/cloudsql/breadsheet:us-west1:breadsheet.
Only change I've made on the platform since the initial deploy was upgrading from Cloud Datastore to Firestore, which is where I'm pulling my environment variables from to build the URI. No data ever came from Datastore, so that shouldn't matter.
I'm following the GAE Postgres connection guide for Unix. Here's my app.yaml:
runtime: python
env: flex
instance_class: F1
entrypoint: gunicorn -w 1 breadsheet:breadapp
error_handlers:
- file: app/templates/errors/default_error.html
- error_code: over_quota
file: app/templates/errors/over_quota.html
beta_settings:
cloud_sql_instances: breadsheet:us-west1:breadsheet
Here's the error when I run gunicorn from the cloud shell:
(venv) [me]#cloudshell:~/breadsheet (breadsheet)$ gunicorn -w 1 breadsheet:breadapp
[2019-04-07 10:23:16 -0700] [471] [INFO] Starting gunicorn 19.9.0
[2019-04-07 10:23:16 -0700] [471] [INFO] Listening at: http://127.0.0.1:8000 (471)
[2019-04-07 10:23:16 -0700] [471] [INFO] Using worker: sync
[2019-04-07 10:23:16 -0700] [474] [INFO] Booting worker with pid: 474
2019-04-07 09:40:08,838 Exception on / [GET]
Traceback (most recent call last):
[...]
File "/home/[me]/breadsheet/venv/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 437, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/[me]/breadsheet/venv/lib/python3.5/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/cloudsql/breadsheet:us-west1:breadsheet/.s.PGSQL.5432"?
The above exception was the direct cause of the following exception:
[line of code with the first database call in my app]
[...]
File "/home/[me]/breadsheet/venv/lib/python3.5/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/cloudsql/breadsheet:us-west1:breadsheet/.s.PGSQL.5432"?
Tried the following (and more!) to no avail:
Whitelisting various IPs in CloudSQL management
Adjusting IAM permissions
Using nano to comment out the Firestore code and hard-code the URI
Using nano to hard-code the SQLALCHEMY_DATABASE_URI under env_variables in my app.yaml
What to try next?
For those reading this via search in the future, I had previously skipped the export SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://[USER]:[PW]#127.0.0.1:5432/[DB_NAME] because I didn't particularly care about the cloud shell environment and assumed it wouldn't make a difference for deployment. Apparently it does.
Later, when troubleshooting the db connection in cloud shell, I found myself skipping the first part (sudo mkdir /cloudsql;) of this two-step command sudo mkdir /cloudsql; sudo chmod 777 /cloudsql because I had already created the /cloudsql folder. I'm still not sure why both parts of the command are needed -- especially when the folder already exists -- but the db connection didn't work for me without them both.
One more note: setting SQLALCHEMY_DATABASE_URI under env_variables: in the app.yaml is not required, provided you set that value elsewhere in the app. Adding to the env_variables: simply makes the value available as an easily-accessible variable.
I'm trying to connect to Cloud SQL from a Python application (using PyMySQL 0.7.9) running on top of Google App Engine.
My connection string looks like this (credentials are fake of course):
pymysql.connect(unix_socket='/cloudsql/gae_project_name:cloudsql_instance_name',
user='user', password='', db='database_name')
The error message I receive is:
OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 97] Address family not supported by protocol)")
It's like PyMySQL doesn't recognize that I'm trying to connect through a Unix socket and tries the default value for the host argument instead (which I presume is localhost)
I am able to connect with MySQLdb with the same connection string.
Why don't use MySQLdb instead then ?
I just had the same problem deploying a Flask application with PyMySQL, I tried a lot of fixes without success. My workaround was to use MySQLDb instead aha..!
Apparently, PyMySQL is not currently supported on the Google App Engine Standard environment, which only runs Python 2.7 (as of June 2018). This is from the maintainers of the GCP python project:
I can confirm that pymysql is not supported in the python27 runtime. However, for most use cases, it's possible to use pymysql locally and mysqldb in production by using a try: / except ImportError: to import one or the other conditionally. As they share the same interface, you can use import as to make the two different libraries share the same name for ease of use in your code.
See this Github thread for details
I got it to connect using a Python3 Flex App Engine and PyMySQL.
Here's what my app.yaml looks like:
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT api:app --timeout 180
runtime_config:
python_version: 3
env_variables:
SQLALCHEMY_DATABASE_URI: >-
mysql+pymysql://user:password#/database?unix_socket=/cloudsql/project-name:us-central1:instance-name
beta_settings:
cloud_sql_instances: us-central1:instance-name
# This sample incurs costs to run on the App Engine flexible environment.
# The settings below are to reduce costs during testing and are not appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/python/configuring-your-app-with-app-yaml
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
Make sure to replace the user, password, database, and instance connection in both the env_variables and the beta_settings.
Here's my python:
import pymysql.cursors
import pymysql
connection = pymysql.connect(unix_socket='/cloudsql/project-name:us-central1:instance-name',
user='user',
password='password',
db='database',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)