Develop Reference

Deploy Django project on google Cloud App Engine with external libraries

Good day!
I need to deploy an app on Python Django with following libs:
certifi==2017.4.17
chardet==3.0.4
Django==1.11.2
httplib2==0.10.3
idna==2.5
oauth2==1.9.0.post1
psycopg2==2.7.1
pytz==2017.2
requests==2.18.1
urllib3==1.21.1
And I have to connect it to PostgreSQL database.
So, what environment shall I use? Standard or Flexible?
And how shall I install those libraries to the environment?
P.S.
I've tried everything: app.yaml, appengine_config.py, I has installed libs directly to source ('libs' folder), and adding libs to app.yaml, and even google.appengine.ext.ndb.django_middleware.NdbDjangoMiddleware
In the end I have:
ImproperlyConfigured: Error loading psycopg2 module: dynamic module
does not define init function (init_psycopg)

So, what environment shall I use? Standard or Flexible?
Your starting point should be the Choosing an App Engine Environment guide, taking into consideration all your requirements.
And I have to connect it to PostgreSQL database.
This requirement makes the choice pretty easy. From Connecting from App Engine:
This page provides language-specific links to informaton about how to
set up a connection from an App Engine flexible environment
application to a Cloud SQL for PostgreSQL instance.
Note: Connection from an App Engine standard environment application
to a PostgreSQL instance is not supported.
So flexible environment it is.
And how shall I install those libraries to the environment?
This dependends on the environment you use. What you tried was the standard env way.
In the flexible environment dependencies are handled differently. From Dependencies:
The runtime looks for a requirements.txt file in your
application's source directory and uses pip to install any
dependencies before starting your application. For more information on
declaring and managing packages, see Using Python Libraries

PyCharm - Can't create App Engine Application using DJango

The last few days I tried to create an App Engine Application based on DJango using PyCharm Professional 2016.3.2..
But everytime I try to create a new App Engine Application using Django as an third-party library PyCharm throws an Error and tells 'Create App Engine Application - Incorrect or corrupted App Engine SKD: cannot finde DJango installation'.
I already tried to install everything again, including Python, PyCharm, Django and the SDK, but it still doesn't work.
I would really appreciate any kind of help, because this problem drives me crazy.

You need to install extras.
gcloud components install app-engine-python-extras
And then try to create django (app engine) project again.

The legacy Appengine SDK ships with some Django versions included in its lib folder.
$ find 1.9.40/ -type d -name [Dd]jango*
1.9.40/google_appengine/lib/jinja2-2.6/examples/rwbench/django
1.9.40/google_appengine/lib/jinja2-2.6/ext/django2jinja
1.9.40/google_appengine/lib/django-1.4
1.9.40/google_appengine/lib/django-1.4/django
1.9.40/google_appengine/lib/PyAMF-0.6.1/pyamf/tests/adapters/django_app
1.9.40/google_appengine/lib/django-1.2
1.9.40/google_appengine/lib/django-1.2/django
1.9.40/google_appengine/lib/PyAMF-0.7.2/pyamf/adapters/tests/django_app
1.9.40/google_appengine/lib/django-0.96
1.9.40/google_appengine/lib/django-0.96/django
1.9.40/google_appengine/lib/django-1.5
1.9.40/google_appengine/lib/django-1.5/django
1.9.40/google_appengine/lib/django-1.3
1.9.40/google_appengine/lib/django-1.3/django
1.9.40/google_appengine/lib/django-1.9
1.9.40/google_appengine/lib/django-1.9/django
1.9.40/google_appengine/google/appengine/_internal/django
1.9.40/google_appengine/google/appengine/ext/django
1.9.40/google_appengine/google/appengine/ext/builtins/django_wsgi
1.9.40/google_appengine/google/storage/speckle/python/django
However the Google Cloud SDK does not include any Django versions in lib, only with Django utilities for its own internal use.
$ gcloud --version
Google Cloud SDK 138.0.0
app-engine-python 1.9.49
...
$ find google-cloud-sdk/ -type d -name [Dd]jango*
google-cloud-sdk/lib/third_party/oauth2client/contrib/django_util
google-cloud-sdk/lib/googlecloudsdk/third_party/appengine/ext/builtins/django_wsgi
google-cloud-sdk/platform/google_appengine/lib/jinja2-2.6/examples/rwbench/django
google-cloud-sdk/platform/google_appengine/lib/jinja2-2.6/ext/django2jinja
google-cloud-sdk/platform/google_appengine/lib/PyAMF-0.6.1/pyamf/tests/adapters/django_app
google-cloud-sdk/platform/google_appengine/lib/PyAMF-0.7.2/pyamf/adapters/tests/django_app
google-cloud-sdk/platform/google_appengine/google/appengine/_internal/django
google-cloud-sdk/platform/google_appengine/google/appengine/ext/django
google-cloud-sdk/platform/google_appengine/google/appengine/ext/builtins/django_wsgi
google-cloud-sdk/platform/google_appengine/google/storage/speckle/python/django
google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/contrib/django_util
google-cloud-sdk/.install/.backup/lib/googlecloudsdk/third_party/appengine/ext/builtins/django_wsgi
google-cloud-sdk/.install/.backup/platform/google_appengine/lib/jinja2-2.6/examples/rwbench/django
google-cloud-sdk/.install/.backup/platform/google_appengine/lib/jinja2-2.6/ext/django2jinja
google-cloud-sdk/.install/.backup/platform/google_appengine/lib/PyAMF-0.6.1/pyamf/tests/adapters/django_app
google-cloud-sdk/.install/.backup/platform/google_appengine/lib/PyAMF-0.7.2/pyamf/adapters/tests/django_app
google-cloud-sdk/.install/.backup/platform/google_appengine/google/appengine/_internal/django
google-cloud-sdk/.install/.backup/platform/google_appengine/google/appengine/ext/django
google-cloud-sdk/.install/.backup/platform/google_appengine/google/appengine/ext/builtins/django_wsgi
google-cloud-sdk/.install/.backup/platform/google_appengine/google/storage/speckle/python/django
google-cloud-sdk/.install/.backup/platform/gsutil/third_party/oauth2client/oauth2client/contrib/django_util
The solution is to install Django according to Google's instructions for installing third party packages, that is:
Install django in your third party libs folder:pip install django -t lib
Add from google.appengine.ext import vendor;vendor.add('lib') in your appengine_config.py module

As #snakecharmerb observed the django packages are missing in the app-engine-python component (1.9.49) of the cloud SDK (138.0.0).
They are, however, present in the standalone GAE SDK (also 1.9.49):
/usr/local # diff google_appengine_1.9.49/lib google-cloud-sdk-138.0.0/platform/google_appengine/lib | grep -vi common
Only in google_appengine_1.9.49/lib: django-0.96
Only in google_appengine_1.9.49/lib: django-1.2
Only in google_appengine_1.9.49/lib: django-1.3
Only in google_appengine_1.9.49/lib: django-1.4
Only in google_appengine_1.9.49/lib: django-1.5
Only in google_appengine_1.9.49/lib: django-1.9
Only in google_appengine_1.9.49/lib: graphy
So an alternative to vendoring django into your app might be to switch to this SDK instead of the cloud SDK. See What is the relationship between Google's App Engine SDK and Cloud SDK?, including comments.
Note that it's not the recommended way these days, tho.
You could also install both and try to copy/symlink the missing packages into the cloud SDK. YMMV.
To download the GAE SDK in the Download the SDK for App Engine page click on the Optionally, you can also download the original App Engine SDK for Python. "link" - it's actually a control for the expandable download section below:

Getting Django 1.7 to work on Google App Engine

Can anyone help to point us to instructions on how to get Django >1.5 working on Google App Engine? I have seen a number of people claim they have Django 1.6 working. We'd like to get 1.6 or 1.7 running. I have searched here for instructions on how to set this up. No luck so far.
Update:
In our development machine we have Django 1.7 installed (both /user/local and on virtualenv). However, if we modify GAE yaml to use Django 1.7 we get the following error messages:
google.appengine.api.yaml_errors.EventError: django version "1.7" is not supported, use one of: "1.2", "1.3", "1.4", "1.5" or "latest" ("latest" recommended for development only) in "./app.yaml",
The version 1.9.12 GoogleAppEngine sdk install in our /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib directory shows the following Django versions listed:
django-0.96 django-1.2 django-1.3 django-1.4 django-1.5
My question is related to how to get our development environment setup correctly for Django 1.7 on Google App Engine and how to make sure we successfully deploy our app with Django 1.7 when we deploy to Google App Engine in production. How do we get the Django 1.7 to deploy on GAE when we deploy our app?

You can use any pure Python third party libraries in your Google App Engine application. In order to use a third party library, simply include the files in your application's directory, and they will be uploaded with your application when you deploy it to our system. You can import the files as you would any other Python files with your application.
I have application using Django 1.7 this way and everything is working fine. However, sometimes you may need to sort of hack something due to the App Engine limitations and its specifics. But it depends on your use cases.
I would also suggest to use virtual environment for your project. Install each library that is not supported by App Engine directly via pip and then create a symlink in your application directory pointing to the given library.
This way you can keep all required packages in a file (e.g. requirements.txt) that can be stored in SCM system (e.g. Git) along with your source files and other team members can quite easily replicate your working environment.
Provided that you use virtual environment and install all needed libraries (Django, ...) via pip, here is the directory layout that should work for you.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
app.yaml
django-project-root
django-app-root
symlink-to-django -> lib/python2.7/site-packages/django
symlink-to-another-lib -> lib/python2.7/site-packages/...
Such a layout can be easily deployed with the below command.
$ appcfg.py update app-engine-project-root
Or tested with App Engine development server.
$ dev_appserver.py app-engine-project-root
UPDATE
Since App Engine Python SDK version 1.9.15 you can use the vendoring mechanism to set up third party libraries. You do not have to create symlinks in your application directory pointing to the Python lib folder anymore.
Create lib directory directly in your application root and tell your app how to find libraries in this directory by means of appengine_config.py file.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
New directory layout follows.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
lib
app.yaml
appengine_config.py
django-project-root
django-app-root
Use pip with the -t lib flag to install libraries in this directory.
$ pip install -t lib [lib-name]
Or
$ pip install -t lib -r requirements.txt

You cannot - GAE only supports 1.5, and even that is marked as experimental. If you need django 1.7, perhaps you should use Google Compute Engine, which is Google's brand name for virtual machines that you can spool up.
If you are not married to Google App Engine, Heroku supports django 1.7 without issues.
Do you have specific a guide on how to move a Django 1.7 project to
Google Compute Engine? There is a bunch of Google stuff without any
guides on how to make them work.
Here are the steps, but they are the same had you deployed on any other server because GCE just gives you a linux instance:
First, make sure your developer account has a billing method attached to it.
Go to the developer console
Create a new project by clicking on Projects, then Create Project.
Wait as the project is being created (you'll see a progress window on the bottom right of your screen).
Once the project is finished creating, the console will automatically shift to that project's settings:
You can create a new instance, or deploy a ready-made template from the second column. You can see there are popular stacks and software applications for which templates are created.
As there is no django template yet, you will start by creating an instance.
Billing is controlled on a per-project basis, so you'll have enable billing at this point if you haven't done so already.
The next page is where you configure the instance. The fields are self-explanatory. You set the type of machine you like (how many virtual CPUs and memory), where (physically) you prefer the machine to be located, if you want both HTTP and HTTPS ports open, and then a disk image from which the instance will boot:
Once you have configured the machine, it will be brought online booted up and then you'll have access to the terminal via SSH.
From this point forward, you should treat this instance like any linux server. Install whatever you need to make your project work using the normal packaging tools; upload your files, etc.
For Amazon, the process is a bit simpler as there is a large library of AMIs that you can use for a one-click deployment process. AMI is Amazon Machine Image - a template from which you can deploy an instance.
For Heroku, as its a PaaS, you don't have to worry about the hardware components; however as with most PaaS platforms, you don't have write access to the filesystem. So to manage your static assets you have to do some extra work. The easiest option is to create a S3 bucket on Amazon and use that with django-storages. The official django tutorial at heroku suggests the use of dj-static to serve files directly from Heroku. This works fine for testing, but if you want to start uploading files, then you need to handle those correctly.
However, once you sort that out the steps are even simpler:
Pre-requisites:
git
heroku toolbelt
dj-database-url Python package
gunicorn Python package
The basic steps:
Create a git repository (if you have not done already) in your source code directory with git init.
Create a requirements.txt at the root of your project. pip freeze > requirements.txt should do it if you are using a virtual environment. Otherwise, you can create a text file and list the packages you need.
Adjust your settings.py, by adding this line at the very bottom: import dj_database_url
DATABASES['default'] = dj_database_url.config()
Create a Procfile (case is important). This is how you tell Heroku what kind of dyno (process) you need for your application. For django, you need a web dyno so in this file the following line should do: web: gunicorn yourproject.wsgi --log-file -
Create an app on Heroku and deploy. You should run these commands from your source code directory:
heroku create --buildpack https://github.com/heroku/heroku-buildpack-python
heroku addons:add heroku-postgresql:dev
git push heroku master
heroku run python yourproject/manage.py migrate --noinput
heroku run python web/manage.py collectstatic
You only do the first two steps once, then whenever you need to update your application simply git push heroku master to create a new revision on Heroku.

App Engine's Python environment currently knows how to provide Django up to version 1.5 via the libraries: configuration mechanism. This doesn't mean that later versions of Django won't work, only that they aren't yet built in. (I'm not sure why the latest built-in version is 1.5. It may have something to do with AE's historical policy of bundling each supported version of Django with the SDK, which probably needs to be revised to keep the SDK from getting too large.)
You can try to include Django 1.7 with your application files. I haven't tried this with 1.7 specifically yet, but it's worked with previous versions. Some adjustments to sys.path will be needed in your main.py.
Note that there is a limit of 10,000 application files. If you're concerned about this limit, one option is to use Python's zipimport and include Django as a zip archive. https://docs.python.org/2/library/zipimport.html

Reusing app engine preloaded modules in my app

I'm wondering if appcfg.py in GAE can be smart enough to not upload Python modules that actually should be pre-installed for GAE apps.
E.g., in my workflow I'm developing a pure Django app in standard Django environment which is not aware about httplib2 module. For the project to be self-contained I'm using virtualenv and pip install missing modules. And since I couldn't make GAE's appcfg.py happy in virtualenv sandbox I simply created symlinks to all custom virtualenv modules in the root folder. So, for example, I ended up with a symlink httplib2 --> ../src/lib/site-packages/httplib2.
Then to check if httpblib2 had actually been uploaded to GAE I used appcfg.py download_app command and, bingo, httpblib2 was sitting there, eating my quota for no reason.
I could think of tweaking syspath etc. but it is interesting to know if there are some cleaner solutions...

Can't import Flask while using Google App Engine

I'm following this guide and trying to develop a Flask app to run on the Google App Engine. I followed the guide to the letter but when I launch the dev app server from the Launcher and go to http://localhost:8080/, I get a HTTP 500 error.
I check the logs and it says No module named flask. Then I check the interactive console in the admin console by running import flask and I get the same error message. I can import flask in any other python file without error.
Is there a way to fix this?

Working a bit with GAE and Flask I have realized this:
Running directly with Python
To run the app with python directly (python app.py) you need have the dependents packages installed in your environment using the command: pip install flask
Running with dev_appserver.py
To run the app with the dev_appserver.py provided by GAE SDK you need have all dependent packages inside your project, as: Flask, jinja2... Look in my another answer a example how to configure this packages : https://stackoverflow.com/a/14248647/1050818
UPDATED
Running Python, Virtualenv, Flask and GAE on Windows
Install Python
Install Python http://www.python.org/ftp/python/2.7.2/python-2.7.2.msi
Click in Windows Start button and search by "Edit the system environment" and open
Go to the tab Advanced and click on button "Environment Variables…"
When the Environment Variables window opens, choose Path from the System variables list and click Edit…
Add this ;C:\Python27;C:\Python27\Scripts at the end of the value and save
Install setuptools MS Windows installer (Necessary to install PIP on Windows)
Choose the correct installer for you in this page http://pypi.python.org/pypi/setuptools#files( I used this one: http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11.win32-py2.7.exe#md5=57e1e64f6b7c7f1d2eddfc9746bbaf20)
Download the installar and Install that
Install PIP
Download PIP http://pypi.python.org/pypi/pip#downloads
Extract it to any folder
From that directory, type python setup.py install
Install Virtualenv
Execute pip install virtualenv
Execute this mkdir c:\virtualenvs to create a folder to the Virtual Envs
Execute this cd c:\virtualenvs to access that folder
Execute virtualenv flaskdemo to create a virtualenv for you project
Active the virtualenv c:\virtualenvs\flaskdemo\scripts\activate
Install Google App Engine SDK
Install the SDK https://developers.google.com/appengine/downloads
Create the project
Create a directory for your project
Create the main of your application https://github.com/maxcnunes/flaskgaedemo/blob/master/main.py
Create the configuration of your appliction for Google App Engine https://github.com/maxcnunes/flaskgaedemo/blob/master/app.yaml
Create a file to let GAE initialize your application https://github.com/maxcnunes/flaskgaedemo/blob/master/initialize_gae.py
(Look a example of the code here: https://github.com/maxcnunes/flaskgaedemo )
Install Flask to run Locally
Execute pip install flask
Install Flask to run on the GAE
Download Flask https://github.com/mitsuhiko/flask/archive/0.9.zip and extract the folder flask inside your project
Download Werkzeug https://github.com/mitsuhiko/werkzeug/archive/0.8.3.zip and extract the folder werkzeug inside your project
Download Jinja2 https://github.com/mitsuhiko/jinja2/archive/2.6.zip and extract the folder jinja2 inside your project
Download Simple Json https://github.com/simplejson/simplejson/archive/v3.0.5.zip and extract the folder simplejson inside your project
Running the application with GAE SDK
Open Google App Engine Launcher
Add a new application
Run the application
Click in Browse button to open your application on browser
Finally click on Deploy button to deploy your application

Usually, templates come with a requirements.txt. If not, add your dependencies there and then run pip install -t lib -r requirements.txt to force the libraries to be saved in the lib folder.
Make sure you've added lib to appengine_config.py with vendor.add('lib') if it's not already there.

I was also facing the same issue and after spending 1 day on it have found out my silly mistake actually while refactoring my flask app I have changed
appengine_config.py to some other name.
Ideally appengine_config.py should look like below if you are having all your dependencies in lib folder only
from google.appengine.ext import vendor
#Add any libraries installed in the "lib" folder.
vendor.add('lib')
And because it was not able to find and execute appengine_config.py so lib folder was not registered as a dependency folder. To check you can try printing something in appengine_config.py to check if it's being executed on server startup.

tldr: use appengine_config.py and copy your virtualenv to a folder called lib, then make SURE you are running the app via dev_appserver.py
(the below is via bash in ubuntu)
SO after a long battle, I find that virtual env and gcloud dont play nice -
I copied everything from my virtual env dir
.../.virtualenvs/nimble/local/lib/python2.7/site-packages
into
[projectdir]/lib
and my appengine_config.py finally worked locally like it does in the cloud, but I absolutely HAVE to run
dev_appserver.py [my proj dir here]
or the google.appengine module wont load. did not know I should be using dev server. I feel very dumb.
for reference, heres the appengine_config.py
"""`appengine_config` gets loaded when starting a new application instance."""
print 'running app config yaya!'
from google.appengine.ext import vendor
vendor.add('lib')
print 'I am the line after adding lib, it should have worked'
import os
print os.getcwd()

Do you have Extra Libraries component for Python installed? It can be installed with
gcloud components install app-engine-python-extras
After installing this extra library you should be able to use built-in flask library without a problem. For more information, refer to this page
Source