Develop Reference

Why i can`t start Local Development Server [duplicate]

I've been successfully running the local development server daily and have made no changes except that I called "gcloud components update" just before it stopped working. Now I get:
..snip... <<PATH TO MY SDK>>/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/application_configuration.py", line 518, in _parse_configuration
with open(configuration_path) as f:
IOError: [Errno 2] No such file or directory: 'app.yaml'
Of course app.yaml hasn't moved.
Any thoughts?

It looks like there's an active issue on Google's issue tracker (opened on Oct 2, 2018) pertaining to this:
After updating to the Python (2.7) extensions for GAE to version
1.9.76, I am no longer able to run my code with dev_appserver.py
As of Oct 3, a fix appears to be in the works, but for now they suggest downgrading Google Cloud SDK to version 218.0.0:
It seems like you are affected by a known issue regarding ‘dev_appserver.py’ breaks in Google Cloud SDK version [219.0.1]. App Engine specialists are currently working to resolve it. However
there is no ETA at this moment. As a workaround you can downgrade the
Google Cloud SDK version using this command:
gcloud components update --version 218.0.0
The assignee of the issue will post an update on that issue when it has been resolved.
UPDATE (OCT 9, 2018): Cloud SDK version 220.0.0, which fixes the dev_appserver.py issue, is now available. I updated (via gcloud components update) and verified that it works. (Note: there are already a couple of complaints on the Issue Tracker that dev_appserver.py takes too long to load now. I didn't notice a significant difference from version 218, but I didn't compare timings.)

You may create make file and have something like this:
export SDK=dev_appserver.py
export APP_PATH=${CURDIR}
run:
$(SDK) $(APP_PATH)/path-to/app.yaml
And just use it with: make run so you don't have to worry about paths.

cd to the directory with app.yaml in it and try again

On Windows, dev_appserver.py %CD% is enough if your .yaml file has the default app.yaml name. Otherwise dev_appserver.py %CD%/your-file-name.yaml

This worked for me: in app.yaml, change
runtime: go
to
runtime: go111

Get app engine info via api commands

Seems like google cloud's shell commands are super limited.
I can see docs about using appcfg.py to upload and download code from app engine, but what are the commands to just get basic info about it?
Is there a list services command? Or a get running versions command? I still don't understand where the app engine code I upload lives.
What are the commands to just "poke" app engine stuff?

To get info about what appcfg.py can do:
$ appcfg.py help
Usage: appcfg.py help <action>
appcfg.py: error: Expected a single action argument. Must be one of:
...
list_versions: List all uploaded versions for an app.
...
$ appcfg.py help list_versions
Usage: appcfg.py [options] list_versions [directory]
List all uploaded versions for an app.
The 'list_versions' command outputs the uploaded versions for each module of
an application in YAML. The YAML is in formatted as an associative array,
mapping module_ids to the list of versions uploaded for that module. The
default version will be first in the list.
...
A bit newer than appcfg.py and the officially recommended one these days (but still not covering all the stuff from the Console) is the gcloud CLI from the Cloud SDK (and in particular the gcloud app submenu for GAE apps).
Your uploaded code lives in the cloud :) Donno exactly where, but you can see it in the Console's Debug page. Not directly accessible from the cloud shell's VM (AFAIK). If you want to access it in the cloud shell you need to download it from your app to your "cloud homedir" (using one of the 2 utilities).
As for "poking", you have to be very specific as different kinds of poking are done in different ways (many in the GUI only). Probably better to do it in a separate question (or questions).

Azure deployment not installing Python packages listed in requirements.txt

This is my first experience deploying a Flask web app to Azure.
I followed this tutorial.
The default demo app they have works fine for me.
Afterwards, I pushed my Flask app via git. The log shows deployment was successful. However, when I browse the hosted app via link provided in "Application Properties", I get a 500 error as follows:
The page cannot be displayed because an internal server error has
occurred.
Most likely causes: IIS received the request; however, an internal
error occurred during the processing of the request. The root cause of
this error depends on which module handles the request and what was
happening in the worker process when this error occurred. IIS was not
able to access the web.config file for the Web site or application.
This can occur if the NTFS permissions are set incorrectly. IIS was
not able to process configuration for the Web site or application. The
authenticated user does not have permission to use this DLL. The
request is mapped to a managed handler but the .NET Extensibility
Feature is not installed.
The only off-base thing I can see by browsing the wwwroot via KUDU is that none of the packages I have installed in my local virtual environment are installed on Azure despite the existence of the "requirements.txt" file in wwwroot.
My understanding is that Azure would pip install any non existent package that it finds in the requirements.txt upon GIT successful push. But it doesn't seem to be happening for me.
Am I doing something wrong and the missing packages is just a symptom or could it be the cause the issue?
Notes:
My Flask app works fine locally (linux) and on a 3rd party VPS
I redeployed several times starting from scratch to no avail (I use local GIT method)
I cloned the Azure Flask demo app locally, changed just the app folder and pushed back to Azure, yet no success.
Azure is set to Python 2.7 same as my virtual env locally
As suggested in the tutorial linked above, I deleted the "env" folder and redeployed to trick Azure to reinstall the virtual env. It did but with its own default packages not the one in my requirements.txt.
My requirements.txt has the following:
bcrypt==3.1.0 cffi==1.7.0 click==6.6 Flask==0.11.1 Flask-Bcrypt==0.7.1
Flask-Login==0.3.2 Flask-SQLAlchemy==2.1 Flask-WTF==0.12
itsdangerous==0.24 Jinja2==2.8 MarkupSafe==0.23 pycparser==2.14
PyMySQL==0.7.7 python-http-client==1.2.3 six==1.10.0 smtpapi==0.3.1
SQLAlchemy==1.0.14 Werkzeug==0.11.10 WTForms==2.1

As Azure Web Apps will run a deploy.cmd script as the deployment task to control which commands or tasks will be run during the deployment.
You can use the command of Azure-CLI azure site deploymentscript --python to get the python applications' deployment task script.
And you can find the following script in this deploy.cmd sciprt:
IF NOT EXIST "%DEPLOYMENT_TARGET%\requirements.txt" goto postPython
IF EXIST "%DEPLOYMENT_TARGET%\.skipPythonDeployment" goto postPython
echo Detected requirements.txt. You can skip Python specific steps with a .skipPythonDeployment file.
So the .skipPythonDeployment will skip all the following steps in deployment task, including creating virtual environment.
You can try to remove .skipPythonDeployment from your application, and try again.
Additionally, please refer to https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script for more info.

Accessing BigQuery on Google App Engine dev server

I am successfully able to get BQ data from one project to another from the advice in this answer. However this only works when deployed on my development/staging instance and not my local development server on Google App Engine.
My findings are that it works in production because you include:
libraries:
- name: pycrypto
version: "latest"
in app.yaml. However these libraries are not accessible from the dev server. I have tried installing everything locally (Pycrypto, oauth2client, openSSL) after digging through some docs and tracing the error but still cannot get it to work. I have tried installing through pip and manually doing the build/install from the raw files to no avail. Any advice on getting these queries to work on the local django server? Working on Ubuntu if that matters, perhaps it's looking in the wrong spot for the libraries?

If its just the libs that are mising follow this answer https://stackoverflow.com/a/11405769/3877822 to insatll pycrypto to the root of your project
As #Udi suggests in the comment below, the following command also
installs pycrypto and can be used in virtualenv as well:
easy_install
http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe
Notice to choose the relevant link for your setup from this list

Getting Django 1.7 to work on Google App Engine

Can anyone help to point us to instructions on how to get Django >1.5 working on Google App Engine? I have seen a number of people claim they have Django 1.6 working. We'd like to get 1.6 or 1.7 running. I have searched here for instructions on how to set this up. No luck so far.
Update:
In our development machine we have Django 1.7 installed (both /user/local and on virtualenv). However, if we modify GAE yaml to use Django 1.7 we get the following error messages:
google.appengine.api.yaml_errors.EventError: django version "1.7" is not supported, use one of: "1.2", "1.3", "1.4", "1.5" or "latest" ("latest" recommended for development only) in "./app.yaml",
The version 1.9.12 GoogleAppEngine sdk install in our /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib directory shows the following Django versions listed:
django-0.96 django-1.2 django-1.3 django-1.4 django-1.5
My question is related to how to get our development environment setup correctly for Django 1.7 on Google App Engine and how to make sure we successfully deploy our app with Django 1.7 when we deploy to Google App Engine in production. How do we get the Django 1.7 to deploy on GAE when we deploy our app?

You can use any pure Python third party libraries in your Google App Engine application. In order to use a third party library, simply include the files in your application's directory, and they will be uploaded with your application when you deploy it to our system. You can import the files as you would any other Python files with your application.
I have application using Django 1.7 this way and everything is working fine. However, sometimes you may need to sort of hack something due to the App Engine limitations and its specifics. But it depends on your use cases.
I would also suggest to use virtual environment for your project. Install each library that is not supported by App Engine directly via pip and then create a symlink in your application directory pointing to the given library.
This way you can keep all required packages in a file (e.g. requirements.txt) that can be stored in SCM system (e.g. Git) along with your source files and other team members can quite easily replicate your working environment.
Provided that you use virtual environment and install all needed libraries (Django, ...) via pip, here is the directory layout that should work for you.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
app.yaml
django-project-root
django-app-root
symlink-to-django -> lib/python2.7/site-packages/django
symlink-to-another-lib -> lib/python2.7/site-packages/...
Such a layout can be easily deployed with the below command.
$ appcfg.py update app-engine-project-root
Or tested with App Engine development server.
$ dev_appserver.py app-engine-project-root
UPDATE
Since App Engine Python SDK version 1.9.15 you can use the vendoring mechanism to set up third party libraries. You do not have to create symlinks in your application directory pointing to the Python lib folder anymore.
Create lib directory directly in your application root and tell your app how to find libraries in this directory by means of appengine_config.py file.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
New directory layout follows.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
lib
app.yaml
appengine_config.py
django-project-root
django-app-root
Use pip with the -t lib flag to install libraries in this directory.
$ pip install -t lib [lib-name]
Or
$ pip install -t lib -r requirements.txt

You cannot - GAE only supports 1.5, and even that is marked as experimental. If you need django 1.7, perhaps you should use Google Compute Engine, which is Google's brand name for virtual machines that you can spool up.
If you are not married to Google App Engine, Heroku supports django 1.7 without issues.
Do you have specific a guide on how to move a Django 1.7 project to
Google Compute Engine? There is a bunch of Google stuff without any
guides on how to make them work.
Here are the steps, but they are the same had you deployed on any other server because GCE just gives you a linux instance:
First, make sure your developer account has a billing method attached to it.
Go to the developer console
Create a new project by clicking on Projects, then Create Project.
Wait as the project is being created (you'll see a progress window on the bottom right of your screen).
Once the project is finished creating, the console will automatically shift to that project's settings:
You can create a new instance, or deploy a ready-made template from the second column. You can see there are popular stacks and software applications for which templates are created.
As there is no django template yet, you will start by creating an instance.
Billing is controlled on a per-project basis, so you'll have enable billing at this point if you haven't done so already.
The next page is where you configure the instance. The fields are self-explanatory. You set the type of machine you like (how many virtual CPUs and memory), where (physically) you prefer the machine to be located, if you want both HTTP and HTTPS ports open, and then a disk image from which the instance will boot:
Once you have configured the machine, it will be brought online booted up and then you'll have access to the terminal via SSH.
From this point forward, you should treat this instance like any linux server. Install whatever you need to make your project work using the normal packaging tools; upload your files, etc.
For Amazon, the process is a bit simpler as there is a large library of AMIs that you can use for a one-click deployment process. AMI is Amazon Machine Image - a template from which you can deploy an instance.
For Heroku, as its a PaaS, you don't have to worry about the hardware components; however as with most PaaS platforms, you don't have write access to the filesystem. So to manage your static assets you have to do some extra work. The easiest option is to create a S3 bucket on Amazon and use that with django-storages. The official django tutorial at heroku suggests the use of dj-static to serve files directly from Heroku. This works fine for testing, but if you want to start uploading files, then you need to handle those correctly.
However, once you sort that out the steps are even simpler:
Pre-requisites:
git
heroku toolbelt
dj-database-url Python package
gunicorn Python package
The basic steps:
Create a git repository (if you have not done already) in your source code directory with git init.
Create a requirements.txt at the root of your project. pip freeze > requirements.txt should do it if you are using a virtual environment. Otherwise, you can create a text file and list the packages you need.
Adjust your settings.py, by adding this line at the very bottom: import dj_database_url
DATABASES['default'] = dj_database_url.config()
Create a Procfile (case is important). This is how you tell Heroku what kind of dyno (process) you need for your application. For django, you need a web dyno so in this file the following line should do: web: gunicorn yourproject.wsgi --log-file -
Create an app on Heroku and deploy. You should run these commands from your source code directory:
heroku create --buildpack https://github.com/heroku/heroku-buildpack-python
heroku addons:add heroku-postgresql:dev
git push heroku master
heroku run python yourproject/manage.py migrate --noinput
heroku run python web/manage.py collectstatic
You only do the first two steps once, then whenever you need to update your application simply git push heroku master to create a new revision on Heroku.

App Engine's Python environment currently knows how to provide Django up to version 1.5 via the libraries: configuration mechanism. This doesn't mean that later versions of Django won't work, only that they aren't yet built in. (I'm not sure why the latest built-in version is 1.5. It may have something to do with AE's historical policy of bundling each supported version of Django with the SDK, which probably needs to be revised to keep the SDK from getting too large.)
You can try to include Django 1.7 with your application files. I haven't tried this with 1.7 specifically yet, but it's worked with previous versions. Some adjustments to sys.path will be needed in your main.py.
Note that there is a limit of 10,000 application files. If you're concerned about this limit, one option is to use Python's zipimport and include Django as a zip archive. https://docs.python.org/2/library/zipimport.html