Develop Reference

Unable to download google app engine standard project source in Windows 10

I am trying to download the latest version of one of my existing google appengine projects & I am tired of seeing the errors! I am running the appcfg.py command inside the google sdk python 2.7 dir. Here's what I ran.
python "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\appcfg.py" download_app -A <project> -V <version> C:\DEV_Tools\Dump\
Output :
Traceback (most recent call last):
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\appcfg.py", line 133, in <module>
run_file(__file__, globals())
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\appcfg.py", line 129, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\appcfg.py", line 66, in <module>
from google.appengine.api import appinfo
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\appinfo.py", line 54, in <module>
from google.appengine.api import validation
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\validation.py", line 46, in <module>
from google.appengine._internal.ruamel import yaml
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\_internal\ruamel\yaml\__init__.py", line 85, in <module>
from google.appengine._internal.ruamel.yaml.main import * # NOQA
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\_internal\ruamel\yaml\main.py", line 19, in <module>
from google.appengine._internal.ruamel.yaml.loader import BaseLoader, SafeLoader, Loader, RoundTripLoader # NOQA
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\_internal\ruamel\yaml\loader.py", line 10, in <module>
from google.appengine._internal.ruamel.yaml.constructor import (
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\_internal\ruamel\yaml\constructor.py", line 21, in <module>
from google.appengine._internal.ruamel.yaml.comments import * # NOQA
File "C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\_internal\ruamel\yaml\comments.py", line 642, in <module>
class CommentedMap(MutableMapping, ordereddict, CommentedBase):
File "C:\DEV_Tools\Google\Python27\lib\abc.py", line 86, in __new__
cls = super(ABCMeta, mcls).__new__(mcls, name, bases, namespace)
TypeError: Error when calling the metaclass bases
Cannot create a consistent method resolution
order (MRO) for bases CommentedBase, MutableMapping, ordereddict

There could be an issue with the installed files on your system. I have tried to reproduced the issue however I got different kind of error:
Traceback (most recent call last): File "C:\Program Files
(x86)\Google\google_appengine\appcfg.py", line 133, in
run_file(file, globals()) File "C:\Program Files
(x86)\Google\google_appengine\appcfg.py", line 129, in run_file
execfile(_PATHS.script_file(script_name), globals_) NameError: name
'execfile' is not defined
This was due to corrupted installation files. I have completely reinstalled Google App Engine Python SDK and it worked for me. Try the steps below as it might help you as well:
Downloading Source code from GAE:
It only works for App Engine Standard apps
You can confirm that as in you Google Cloud Console where you choose the version of a specific service. The dropdown menu Tools will not give option Source for Flex apps but only for Standard apps
You should have Python version 2.7 installed
You should have latest version of App Engine SDK for Python
Steps:
Go to the Control Panel > Programs & Features and uninstall your Google App Engine SDK.
Go to the path where GAE was installed and delete the google_appengine folder. In your occasion it is C:\DEV_Tools\Google\Cloud SDK\google-cloud-sdk\platform
Install Python 2.7 from here
Install and initialize Google Cloud SDK from Quickstart for Windows
Run Google Cloud SDK as administrator and execute $ gcloud init
Then execute $ gcloud config set project [PROJECT_ID] based on gcloud config set doc.
Execute $ gcloud components install app-engine-python to install GAE Python based on Download the Cloud SDK
Execute $ gcloud components install app-engine-python-extras to install extras.
Execute $ gcloud components update to update the components.
Install GAE SDK for Python as it is mentioned in Download and install the original App Engine SDK for Python
Restart your computer
Run Google Cloud SDK as Administrator
Execute the command $ gcloud info to verify that the correct project and correct user account is setup.
If it is not setup properly the execute $ gcloud init and set up properly to authenticate the SDK.
Execute appcfg.py -A [PROJECT_ID] -V [VERSION_NAME] download_app C:\PATH\TO\SAVE\SOURCE_CODE Make sure that the version is referring to an GAE STANDARD app
It will redirect you to an browser page so you could login with the same account that you set up the SDK for authentication.
You should see:
Authentication successful.
04:37 PM Fetching files...
04:37 PM [1/16] WEB-INF/lib/_ah_compiled_jsps-0000.jar
04:37 PM [2/16] WEB-INF/lib/org.apache.taglibs.taglibs-standard-impl-1.2.5.jar
04:37 PM [3/16] WEB-INF/min-quickstart-web.xml
04:37 PM [4/16] WEB-INF/appengine-generated/app.yaml
04:37 PM [5/16] WEB-INF/lib/org.eclipse.jdt.ecj-3.14.0.jar
04:37 PM [6/16] WEB-INF/web.xml
04:37 PM [7/16] WEB-INF/lib/org.mortbay.jasper.apache-el-8.5.24.2.jar
04:37 PM [8/16] WEB-INF/lib/org.mortbay.jasper.apache-jsp-8.5.24.2.jar
04:37 PM [9/16] source-context.json
04:37 PM [10/16] WEB-INF/quickstart-web.xml
04:38 PM [11/16] WEB-INF/classes/source-context.json
04:38 PM [12/16] WEB-INF/lib/org.apache.taglibs.taglibs-standard-spec-1.2.5.jar
04:38 PM [13/16] WEB-INF/appengine-web.xml
04:38 PM [14/16] WEB-INF/lib/org.eclipse.jetty.apache-jsp-9.4.14.v20181114-nolog
Otherwise if it is GAE Flex you will see this error:
54319 download_app C:\PATH\TO\SAVE\SOURCE_CODE 04:38 PM Host:
appengine.google.com 04:38 PM Fetching file list... Error 400: ---
begin server output --- Major version "[VERSION_NAME]" does not exist.
--- end server output ---

How to install libraries python-docx / docx on Google Cloud Shell?

I work with python-docx and docx in my PC, but when I've cloned the project to Google Cloud, the problems arose.
Both docx and python-docx are installed there:
igorsavinkin555#cloudshell:~/corrections-msword (coral-heuristic-5610)$ pip install docx --user
Requirement already satisfied: docx in /home/igorsavinkin555/.local/lib/python2.7/site-packages (0.2.4)
Requirement already satisfied: lxml in /home/igorsavinkin555/.local/lib/python2.7/site-packages (from docx) (4.2.4)
Requirement already satisfied: Pillow>=2.0 in /home/igorsavinkin555/.local/lib/python2.7/site-packages (from docx) (5.2.0)
igorsavinkin555#cloudshell:~/corrections-msword (coral-heuristic-215610)$ pip install python-docx --user
Requirement already satisfied: python-docx in /home/igorsavinkin555/.local/lib/python2.7/site-packages (0.8.7)
Requirement already satisfied: lxml>=2.3.2 in /home/igorsavinkin555/.local/lib/python2.7/site-packages (from python-docx) (4.2.4)
igorsavinkin555#cloudshell:~/corrections-msword (coral-heuristic-215610)$
Problem with docx.Document:
igorsavinkin555#cloudshell:~/corrections-msword (coral-heuristic-215610)$ dev_appserver.py $PWD
...
INFO 2018-09-07 14:31:48,503 api_server.py:275] Starting API server at: http://0.0.0.0:41739
INFO 2018-09-07 14:31:48,518 dispatcher.py:270] Starting module "default" running at: http://0.0.0.0:8080
INFO 2018-09-07 14:31:48,519 admin_server.py:152] Starting admin server at: http://0.0.0.0:8000
INFO 2018-09-07 14:31:50,533 instance.py:294] Instance PID: 727
ERROR 2018-09-07 14:32:00,844 wsgi.py:263]
Traceback (most recent call last):
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/igorsavinkin555/corrections-msword/main.py", line 2, in <module>
from docx.document import Document
ImportError: No module named docx.document
Update
The installing of those 3-d party libraries into project lib folder has benefited, now these packages in project/lib folder. Yet, now the error in lxml library:
File "/home/igorsavinkin555/corrections-msword/main.py", line 2, in <module>
from docx.document import Document
File "/home/igorsavinkin555/corrections-msword/lib/docx/__init__.py", line 3, in <module>
from docx.api import Document # noqa
File "/home/igorsavinkin555/corrections-msword/lib/docx/api.py", line 14, in <module>
from docx.package import Package
File "/home/igorsavinkin555/corrections-msword/lib/docx/package.py", line 11, in <module>
from docx.opc.package import OpcPackage
File "/home/igorsavinkin555/corrections-msword/lib/docx/opc/package.py", line 12, in <module>
from .part import PartFactory
File "/home/igorsavinkin555/corrections-msword/lib/docx/opc/part.py", line 12, in <module>
from .oxml import serialize_part_xml
File "/home/igorsavinkin555/corrections-msword/lib/docx/opc/oxml.py", line 12, in <module>
from lxml import etree
File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/sandbox.py
", line 1095, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named lxml.etree
By the way, this was a right way since:
You can install additional software packages on the Coogle Cloud Shell virtual machine instance but the installation will not persist after the instance terminates unless you install the software in your $HOME directory
(source).

Using Google Cloud Shell is only possible to install persistent data in the $HOME directory. Reaching inactive time limit of 60 minutes, VM Instance will be terminated. Accessing it afterwards, it will be provisioned from the image on the new VM Instance.
However, Google Cloud Shell has 5GB persistent dis storage located in $HOME directory, that does not expire, but may be recycled. User receives email prior recycle process.

App Engine + Django: ImportError: No module named django.core.wsgi

I'm deploying a simple Django app to Google App Engine. How can I fix the following?
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 96, in LoadObject
__import__(cumulative_path)
File "/base/data/home/apps/.../20171104t152156.405293023907909181/mysite/wsgi.py", line 12, in <module>
from django.core.wsgi import get_wsgi_application
ImportError: No module named django.core.wsgi

You're getting the ImportError because Django is not in your sys.path.
If you want to use one of the built-in Django versions from the App Engine SDK, simply add this to your app.yaml (it's not necessary to install the Django library separately):
libraries:
- name: django
version: "1.4"
Update:
If you want to include your own Django version with your app (eg. to use a recent version > 1.5), don't add the above line and instead install the library directly into your project's root directory:
$ cd myapp/
$ pip install django -t .

Cannot get gcloud to work with Python and Pycharm

I am trying to connect to the Google App Engine Datastore from my local machine. I have spent all day digging in to this without any luck.
I have tried the approach here (as well as alot of other suggestions from SO such as Using gcloud-python in GAE and Unable to run dev_appserver.py with gcloud):
How to access a remote datastore when running dev_appserver.py?
I first installed gcloud based on this description from google:
https://cloud.google.com/appengine/docs/python/tools/using-libraries-python-27
According to the description I should add the following to my appengine_config.py:
from google.appengine.ext import vendor
vendor.add('lib')
If I do that I get an error saying ImportError: No module named gcloud
If I then move the code to my main.py it seems to pickup the lib-folder and the modules there. That seems a bit strange to me, since I thought appengine_config was being run first to make sure things were initialised.
But now I am getting the following stack trace:
ERROR 2016-09-23 17:22:30,623 cgi.py:122] Traceback (most recent call last):
File "/Users/thomasd/Documents/github/myapp/main.py", line 10, in <module>
from gcloud import datastore
File "/Users/thomasd/Documents/github/myapp/lib/gcloud/__init__.py", line 17, in <module>
from pkg_resources import get_distribution
File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2985, in <module>
#_call_aside
File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2971, in _call_aside
f(*args, **kwargs)
File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 3013, in _initialize_master_working_set
dist.activate(replace=False)
File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2544, in activate
declare_namespace(pkg)
File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2118, in declare_namespace
_handle_ns(packageName, path_item)
File "/Users/thomasd/Documents/github/myapp/lib/pkg_resources/__init__.py", line 2057, in _handle_ns
loader.load_module(packageName)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py", line 246, in load_module
mod = imp.load_module(fullname, self.file, self.filename, self.etc)
File "/Library/Python/2.7/site-packages/google/cloud/logging/__init__.py", line 18, in <module>
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 999, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named google.cloud.logging.client
What am I doing wrong here?

The google-cloud library is not working on App Engine and most likely you don't even have to since you can use the build in functionality.
From the official docs you can use it like this:
import cloudstorage as gcs

I solved it this way:-
1.) Create a lib folder in your project path.
2.) Install gcloud libraries by running following command into terminal from your project path:-
pip install -t lib gcloud
3.) Create an appengine_config.py module in your project and add following lines of code:-
import sys
import os.path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))
4.) After this, you can import like this:-
from gcloud import datastore
5.) To save data into live google datastore from local:-
client = datastore.Client("project-id")
key = client.key('Person')
entity = datastore.Entity(key=key)
entity['name'] = ashish
entity['age'] = 23
client.put(entity)
It will save an entity named Person having properties name and age. Do not forget to specify your correct project id.

Old question but this may be worth including:
I'm unsure the state of your requirements.txt file but I scrounged mine a bit and noticed setuptools was not included.
pip freeze doesn't export setuptools related question
Assuming you're following the tutorial, you likely installed those libraries EXCEPT for setuptools to lib.
I added setuptools=={verionnumber} to requirements.txt and that fixed this related issue for me.

Unable to start App Engine application after updating it via Google Cloud SDK

Recently, I have updated Google App Engine from 1.9.17 to 1.9.18 via Google Cloud SDK by using command 'gcloud components update' in Windows 7 64 bit. After that I wasn't able to start any project using the App Engine launcher. Getting this error:
Traceback (most recent call last):
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\dev_appserver.py", line 83, in <module>
_run_file(__file__, globals())
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\dev_appserver.py", line 79, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\devappserver2.py", line 36, in <module>
from google.appengine.tools.devappserver2 import dispatcher
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\dispatcher.py", line 29, in <module>
from google.appengine.tools.devappserver2 import module
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\module.py", line 71, in <module>
from google.appengine.tools.devappserver2 import vm_runtime_factory
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\vm_runtime_factory.py", line 25, in <module>
from google.appengine.tools.devappserver2 import vm_runtime_proxy
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\vm_runtime_proxy.py", line 29, in <module>
from google.appengine.tools.devappserver2 import log_manager
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\log_manager.py", line 34, in <module>
from google.appengine.tools.docker import containers
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\docker\containers.py", line 47, in <module>
import docker
ImportError: No module named docker
2015-03-05 19:11:27 (Process exited with code 1)
I even installed the latest Google Cloud SDK, but I'm getting the same error.
I'm able to install the appengine SDK 1.9.18(without using Google Cloud SDK) and able to run the project successfully.
This error is happening only for the App Engine launcher installed via Google Cloud SDK in Windows 7.
This issue is raised in App Engine Issue Tracker: Issue 125. I recommend you to star this issue.

This has happened to me today to reinstall the app engine sdk. I could not run my code in the launcher.
I remember reading that is not used pip app engine, but now I have solved the problem.
In short what I did was:
Install pip the footsteps of https://pip.pypa.io/en/latest/installing.html (this also correctly install the setuptools)
Install docker-py by pip: pip install docker-py and ready, I can now run my code in the launcher
P.S.
Previously I tried to install the docker-py package, downloaded from https://github.com/docker/docker-py, but lacked setuptools, downloaded and installed the package did not work. So use this with pip.

This is currently an issue with the dev_appserver bundled in the Cloud SDK. A fix will be out soon. In the meanwhile, your options are:
1) Use gcloud preview app run to run your app when using the Cloud SDK
2) Install the standalone AppEngine SDK (which you already mentioned in your question)

If installing docker-py doesn't work and the stacktrace shows that the error line is:
from docker import docker
Change this line to:
import docker
Source

Jumping on the answer from #Tzach and adding some info.
The file to modify is containers.py
for me it is located here :
C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\docker
If you can't modify it because the file is open in an application, it is in fact that the folder is protected. Just copy/paste the file on your desktop and modify it from there. then copy it back in the original folder.