Develop Reference

How do I use Heroku Postgres with my flask-sqlalchemy app?

Right now I have an app that write data for users to a database file called users.sql. However, I pushed this to heroku through github and the data doesn't stick. If a certain time has passed, heroku will sleep and all the files will have to be redeployed when another login is detected. I heard that I could add heroku postgres as an addon to heroku but I am not sure what to do to integrate this with my current code. What do I do to set it up?

I stumbled across your question while trying to connect my app to Postgres and getting very frustrated at the lack of clear instructions. So here's the step by step process with all the errors I struggled through (because that's where the tutorials leave you gasping for air while they thunder on obliviously...)
TL;DR
Install Postgres
Install the Heroku CLI
Install psycopg2 or psycopg2-binary and gunicorn
Create an app in Heroku with a Postgres database and save the database url as the SQLALCHEMY_DATABASE_URI
Create Procfile, requirements.txt and runtime.txt
Push to Github and connect repo to Heroku
Deploy branch
from app import db db.create_all()
I work with WSL so all command line syntax in my answer is Bash, and I'm working on Windows 10.
I'm working partly off a youtube tutorial by Traversy Media. Here's the Github repo he works with (I definitely recommend initially working with a basic app instead of struggling with the errors that crop up in a complex app with extra packages).
Before you start
Create an account in Heroku and install the Heroku CLI
Download and install PostgreSQL (this will also give you pgAdmin 4)
Fork and clone the Github repo, and check it runs ok (if you're not working with your own app)
Preliminaries
Create a virtual env and install all packages. Ensure you have psycopg2and gunicorn
ERROR: Failed building wheel for psycopg2-binary
For whatever reason, psycopg refuses to install on my system. You can get round this by using pip3 install psycopg2-binary instead - there's no difference as far as our requirements go.
Connect to Postgres
Open pgAdmin 4 from the start menu. This may prompt you for a password if it's the first time you're logging in or if you were logged out.
could not connect to server: Connection refused (0x0000274D/10061) Is the
server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?
Use sudo service postgresql status to check if the PostgreSQL server
is running. If it is, try stopping and starting it. If it isn't, start
it.
sudo service postgresql start
sudo service postgresql stop
In the left-hand panel, right-click on Databases and click Create > Database.... Enter your database name (the tutorial uses "lexus") and save.
In app.py (or wherever you do your database config) find the line
if ENV == 'dev':
...
app.config['SQLALCHEMY_DATABASE_URI'] = ''
and put the database connection string inside the quotes. The syntax is
postgresql://[username]:[password]#[host name]/[db name]
# [username] - Right-click on your server in pgAdmin (probably a variation of `PostgreSQL 13`). Go to the Connection tab and check the Username field.
# [password] - The root password you set the first time you logged in to pgAdmin.
# [host name] - In the same place as your username, in the Host name/address field.
# [db name] - The name you entered when creating your database.
In my case, the connection string is:
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456#localhost/lexus'
Important Disclaimer
This solution has you put the database password in code: app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456#localhost/lexus'
This saves the password in your version control history, where it can/will eventually allow anyone on the Internet to control your database. Yikes! Get DB user & password from env vars instead.
Thanks #Paul Cantrell!
Enter the python console and create the database:
>>> from app import db
>>> db.create_all()
If you didn't get an error on that, consider yourself the luckiest thing since sliced bread. In pgAdmin go to Databases > [your db name] > Schemas > Tables to check they've been created.
ImportError: cannot import name 'db' from 'app' (unknown location)
db is not located in app in your application. Find where it's
defined and import it from there.
(psycopg2.OperationalError) FATAL: password authentication failed for user "postgres"
You're using the wrong password. I found this confusing
because it's unclear whether you need to use the master password (the
one you set when logging in to pgAdmin for the first time) or the
password for the database you're connecting to. It's the master password that's needed here.
Enter psql -h 127.0.0.1 postgres (psql -h [ip address] [username]) in the command line - the password that authenticates is the one you need for the connection string.
(psycopg2.OperationalError) FATAL: database "lexus" does not exist
You might be connecting to the wrong database, using the wrong
username, or your database hasn't been created. Check your connection string and pgAdmin to confirm that details are correct.
db.create_all() seemed to work, but I have no tables in pgAdmin!
If your database models are stored in a separate file, import them
into app.py after db is initialised. It should look like this:
app = Flask(__name__)
...
db = SQLAlchemy(app)
...
from app.models import *
Send data to Postgres
Run the app and submit some data through the form (or create some data in your table). In pgAdmin, right-click the table you just created and refresh, then right-click and view/edit data.
Could not send data to server: Socket is not connected (0x00002749/10057) could not send SSL negotiation packet: Socket is not connected (0x00002749/10057)
Disconnect and reconnect server (right-click on server and click Disconnect Server, then Connect Server) and refresh table again.
If that didn't work, right-click on server and click Properties, go to the Connection tab and change the Host name/address from
localhost to 127.0.0.1. Then repeat step 1.
Create Heroku app
Since I prefer using Heroku directly with Github instead of the CLI, I'm not using git init here, but note that if your code is not already a Github repo you would need to use it.
In the command line:
heroku login # If you have the Heroku CLI installed correctly, this will open a tab in the browser and log you in to Heroku.
heroku create unique-app-name # (replace `unique-app-name` with your own name). This creates an app on the Heroku server, and it's the place your code is cloned to so that it can be run. Check that it has appeared in your [Heroku dashboard](https://dashboard.heroku.com/).
heroku addons:create heroku-postgresql:hobby-dev -a unique-app-name # This creates an empty postgres database for your app. Heroku uses its own database, not your pgAdmin database.
heroku config -a unique-app-name # The url you set earlier as the `SQLALCHEMY_DATABASE_URI` points to the database you created in pgAdmin, but since your app is now going to be using Heroku's database you need a new url. Copy the DATABASE_URL that is returned.
Go back to the code where you set app.config['SQLALCHEMY_DATABASE_URI'] earlier, and insert the new url inside the else block. It should look more or less like this:
if ENV == 'dev':
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456#localhost/lexus'
else:
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres://urlyoujustgotfromherokuozftkndvkayeyc:691196bfb1b1ca8318b733935b10c97a19fd41#ec2-52-71-161-140.compute-1.amazonaws.com:5432/d3pofh2b55a2ct'
If you're working with the tutorial's github repo, set ENV = 'prod', otherwise set the config to use the heroku url however you want.
Add the following files to your root directory if they're not yet included in your project files:
# Procfile
web: gunicorn app:app # The first `app` refers to the app.py and the second `app` refers to the app object, ie `app = Flask(__name__)`. Heroku needs this to know how to run your code.
# runtime.txt
python-3.8.5 # Find your python version by entering `python --version` or `python3 --version` in the command line.
# requirements.txt
# Create this by running `pip3 freeze > requirements.txt`. According to the [Heroku docs](https://devcenter.heroku.com/articles/python-runtimes)
# this must be done after changing runtime versions, so it's best to do it last.
Push everything to Github (again, this is if you're already in a repo otherwise you'd be using the Heroku CLI process which I'm not following).
Deploy on Heroku
Click on the app you created previously in the Heroku dashboard and go to the Deploy tab. In the Deployment method options, click Github. Connect your Github account and repository.
Scroll down to the Manual deploy section and select the branch you want to deploy (or just leave it as master if that's the only one). Click Deploy Branch.
Requested runtime (python-3.7.2) is not available for this stack (heroku-20).
! Aborting. More info: https://devcenter.heroku.com/articles/python-support
! Push rejected, failed to compile Python app.
! Push failed
It looks like the python version is wrong. Check that runtime.txt is showing the right python version. If it
isn't, update it, delete and recreate requirements.txt, push to
Github and click Deploy Branch again.
Create Heroku database
Back in the command line:
heroku run python3
>>> from app import db
>>> db.create_all()
To check out your database, enter
heroku pg:psql --app unique-app-name
If you run select * from tablename; nothing will be returned because there isn't any data yet.
Click the View button in Heroku to open your app. Submit the form (or enter data however your app works), then run select * from tablename; again and you'll see the row of data.

In your init.py file you can set up your app.config['SQLALCHEMY_DATABASE_URI'] variable such that it will look for the one Heroku will give your application if it's running on heroku, OR use the configuration for your local database if it's running locally. You can achieve this as such:
app.config['SQLALCHEMY_DATABASE_URI'] = environ.get('DATABASE_URL') or "postgresql://postgres:password#localhost:5432/postgres"
replace what is after the OR w/ the connection details for your local connection
I would suggest considering setting up a local postgres database instead of sql if you're going to use heroku but you can technically use both. You just may run into minor issues, I have ran into issues w/ dates being handled slightly different, for example.

Firstly make heroku login from terminal:
$ heroku login
Now create the database for your app using following command:
$ heroku addons:create heroku-postgresql:hobby-dev --app app_name
Now after your Database has been created, get the URL of your Database using the following command:
$ heroku config --app app_name
Now that you have got your Database URL, replace the value of app.config[‘SQLALCHEMY_DATABASE_URI’] line in the “app.py” file with this Database URL:
app.config['SQLALCHEMY_DATABASE_URI'] = 'heroku_database_url'
And finally push code to heroku.

Accessing a MongoDB remotely using Pymongo where Ip filtering is used [requires openvpn - cyberoam unique IP]

I have to access the MongoDB and I can easily access it remotely using Pymongo in a jupyter notebook when I run the sudo openvpn --config client.ovpn in my terminal and then connectiong the Pymongo like:
client = MongoClient(host='host_ip',port=port_num)
db = client['db_name']
db.authenticate(name='username',password='password')
But how can I do the same when I want to access the DB over a cloud?
I have tried
client = MongoClient(host='host_ip',port=port_num,ssl=True,
ssl_certfile='UserCertificate.pem',
ssl_keyfile='UserPrivateKey.key',)
db = client['db_name']
db.authenticate(name='username',password='password')
It throws error after the last line of code while authenticating
ServerSelectionTimeoutError: host_ip:port_num: timed out

GCP at the standard environment: MySQL db connection refused

My goal is to take a working Python 2.7 project (MySQL + MS Word files) to work at GCP.
I realize that I need
App Engine - where the app will be running (scaling, etc).
Cloud SQL working as MySQL db.
For that I've followed that Cloud SQL for MySQL tut and
Cloud SQL instance is created with root user.
Both App Engine app and Cloud SQL instance are in the same project.
Cloud Storage
The SQL second generation instance is successfully created and a root user is set.
How I run or deploy
I use Cloud Shell to test the app - dev_appserver.py $PWD and deploy the app from Cloud Shell - gcloud app deploy. It works at appspot.com till I try to use MySQL connection in it.
MySQL connection
The MySQL connection code is taken from here:
import MySQLdb
import webapp2
CLOUDSQL_CONNECTION_NAME = os.environ.get('CLOUDSQL_CONNECTION_NAME')
CLOUDSQL_USER = os.environ.get('CLOUDSQL_USER')
CLOUDSQL_PASSWORD = os.environ.get('CLOUDSQL_PASSWORD')
DB_NAME='test-db'
def connect_to_cloudsql():
# When deployed to App Engine, the `SERVER_SOFTWARE` environment variable
# will be set to 'Google App Engine/version'.
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
# Connect using the unix socket located at
# /cloudsql/cloudsql-connection-name.
cloudsql_unix_socket = os.path.join(
'/cloudsql', CLOUDSQL_CONNECTION_NAME)
db = MySQLdb.connect(
unix_socket=cloudsql_unix_socket,
user=CLOUDSQL_USER,
passwd=CLOUDSQL_PASSWORD)
# If the unix socket is unavailable, then try to connect using TCP. This
# will work if you're running a local MySQL server or using the Cloud SQL
# proxy, for example:
#
# $ cloud_sql_proxy -instances=your-connection-name=tcp:3306
#
else:
db = MySQLdb.connect(
host='127.0.0.1', user=CLOUDSQL_USER, passwd=CLOUDSQL_PASSWORD, db=DB_NAME)
return db
db = connect_to_cloudsql()
Variables are set in app.yaml:
runtime: python27
api_version: 1
threadsafe: true
env_variables:
CLOUDSQL_CONNECTION_NAME: coral-heuristic-215610:us-central1:db-basic-1
CLOUDSQL_USER: root
CLOUDSQL_PASSWORD: xxxxx
When app is run in test mode thru dev_appserver.py $PWD and I choose to use MySQL connection I got an error:
ERROR 2018-09-13 08:37:42,492 wsgi.py:263]
Traceback (most recent call last):
...
File "/home/.../mysqldb.py", line 35, in connect_to_cloudsql
host='127.0.0.1', user=CLOUDSQL_USER, passwd=CLOUDSQL_PASSWORD)
File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (2003, 'Can\'t connect to MySQL server on \'127.0.0.1\' (111 "Connection refused")')
Cloud SQL Proxy
I've downloaded and run the Cloud Proxy for Win-64 (https://dl.google.com/cloudsql/cloud_sql_proxy_x64.exe ) yet still the problem persists... Seems that proxy background app is only for connection to Cloud SQL from my local machine.
You do not need to use the proxy or configure SSL to connect to Cloud SQL from the App Engine standard or flexible environment. (source)
Why is the connection refused?
Should I use rather first generation Cloud sql instance to simplify connection from App Engine?
Update 1
I edit code at the Cloud Console and so far Cloud Console works good.
Update 2
I've succeded to conenect to the sql instance thru Cloud Shell:
(coral-heuristic-215610)$ gcloud sql connect db-basic-1 --user=root
Whitelisting your IP for incoming connection for 5 minutes...done.
Connecting to database with SQL user [root].Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 48841
Server version: 5.7.14-google-log (Google)
MySQL [(none)]>
Update 3
The comment on a similar issue concerns the regions where Cloud SQL instance and App Engine app should be, that is in the same region.
In my case I've checked:
Cloud SQL instance to connect to: us-central1-a
App Engine app: us-central
Are these of one region? - turned out these of one region.
Update 4
I could have figured out to open db connection:
DB connection: <_mysql.connection open to '127.0.0.1' at 7f628c02bc00>
But this seems happened only after I've opened another Cloud Shell instance with the same project (coral-heuristic-215610). At that instance I've started connection to SQL instance and it was successful:
(coral-heuristic-215610)$ gcloud sql connect db-basic-1 --user=root
Whitelisting your IP for incoming connection for 5 minutes...done.
Connecting to database with SQL user [root].Enter password:
I guess that the first cloud shell instance started to connect to db because the second instance has white-listed my IP, isn't it?

The GAE app and Google Cloud SQL instance have to be deployed in the same region if you’re using MySQL First Generation, otherwise, I verified that they can be in different regions as long as you’re using MySQL Second Generation.
I had trouble understanding where are you trying to connect from. I assume you want to connect from the Google Cloud Shell using the proxy and the Cloud SDK Credentials. According to the documentation regarding Cloud SQL Proxy:
The Cloud SQL Proxy provides secure access to your Cloud SQL Second
Generation instances without having to whitelist IP addresses or
configure SSL.
The Cloud SQL Proxy works by having a local client, called the proxy,
running in the local environment. Your application communicates with
the proxy with the standard database protocol used by your database.
The proxy uses a secure tunnel to communicate with its companion
process running on the server.
Remember, since you’re not deploying the application, it is not using the environment variables you have established in the app.yaml. Therefore you have to export and set them yourself in the local machine:
export CLOUDSQL_CONNECTION_NAME=your-connection-name
export CLOUDSQL_USER=root
export CLOUDSQL_PASSWORD=xxxxx
Verify that they are set by doing e.g echo $CLOUDSQL_CONNECTION_NAME. When you deploy your app with gcloud app deploy, this is not necessary since GAE sets the whatever environment variables are specified in the app.yaml.
The proxy has to launched before trying to establish a connection following these steps:
Download the proxy with:
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
Give it execution permission with:
chmod +x cloud_sql_proxy
Start the proxy replacing <INSTANCE_CONNECTION_NAME> with your Cloud SQL instance connection name:
./cloud_sql_proxy -instances=<INSTANCE_CONNECTION_NAME>=tcp:3306
You should see at the end something similar to this:
2018/11/09 13:24:32 Rlimits for file descriptors set to {&{8500 1048576}}
2018/11/09 13:24:35 Listening on 127.0.0.1:3306 for my-project:cloud-sql-region:cloud-sql-name
2018/11/09 13:24:35 Ready for new connections
At this point, is when you can connect to the proxy running locally from the Google Cloud Shell instance, which in turn will connect you to the Cloud SQL instance.
Open another Cloud Shell session (or tab) and launch your code with python myapp.py. You will be connected to the proxy running locally. You can also test the connection by running mysql -h 127.0.0.1 --user=root -p.

I solved the connection refused problem by adding to the app.yaml the following:
beta_settings:
cloud_sql_instances: "<CONNECTION_NAME>"

I want to deploy using the entries that i have in my database

So i used postgres in development for my django project and have important entries in there and i want to deploy to my app in heroku
is there a simple way to do this?

Sure. You just need to export your local database and import it on the Heroku Postgres database. Heroku has a guide to do just that.
Create a dump from your local database. PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump
Upload mydb.dump somewhere Heroku can access it.
Import to heroku. heroku pg:backups restore 'https://s3.amazonaws.com/me/items/3H0q/mydb.dump' DATABASE_URL
Source

Django 1.8 migrating data from local database to heroku database

I first tried using heroku addons:add pgbackups but heroku docs say that it has been deprecated. The instead recommended using this command as specified here
PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump
But this throws the following error-
'PGPASSWORD' is not recognized as an internal or external command,
operable program or batch file.
I have already existing data on my local database and want to transfer those data into the heroku database. Any way to make this work or is there some other way?

You could try fixtures
To create the JSON file from your local database:
python manage.py dumpdata > a_fixture_file.json
Take that file to the server. Be sure your server database has the same migrations. Then
python manage.py loaddata a_fixture_file.json