Develop Reference

ImportError when running dev_appserver.py from pycharm

I'm using the latest version of pycharm 2018.2, on Windows.
I've also updated the Google Cloud Sdk to the latest versions too.
When I use the built in App Engine run configuration in Pycharm:
I get an import error ImportError: cannot import name _common appearing in the run window:
"C:\Program Files\JetBrains\PyCharm 2018.1.1\bin\runnerw.exe" C:\Code\my-
project\env\Scripts\python.exe "C:/Program Files (x86)/Google/Cloud
SDK/google-cloud-sdk/bin/dev_appserver.py" --port 8484 app.yaml tasks.yaml
longtasks.yaml --log_level=debug Traceback (most recent call last):
File "C:/Program Files (x86)/Google/Cloud SDK/google-cloud-
sdk/bin/dev_appserver.py", line 12, in <module>
from bootstrapping import bootstrapping
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-
sdk\bin\bootstrapping\bootstrapping.py", line 24, in <module>
from . import setup # pylint:disable=g-import-not-at-top
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-
sdk\bin\bootstrapping\setup.py", line 55, in <module>
from googlecloudsdk.core import properties
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-
sdk\lib\googlecloudsdk\core\properties.py", line 36, in <module>
from googlecloudsdk.core.util import times
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-
sdk\lib\googlecloudsdk\core\util\times.py", line 55, in <module>
from dateutil.tz import _common as tz_common
ImportError: cannot import name _common
Process finished with exit code 1
The dateutil.tz package is located in C:\Python27\Lib\site-packages and I have a dateutil package in the project I'm running. I don't know whether that has something to do with the issue. I've tried changing the default python interpreter which also did not work. I've tried running Pycharm as administrator which also had no effect. Both the debug and non-debug run commands result in this issue.
If I run the command to start up dev_appserver.py directly in the terminal (with or without virtual environment python), it works correctly e.g. "C:/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin/dev_appserver.py" --port 8484 app.yaml tasks.yaml longtasks.yaml
Also, if I run the command that pycharm is running (except for the runnerw part), then it works correctly (from command line) too:
C:\Code\project>C:\Code\project\env\Scripts\python.exe "C:/Program Files
(x86)/Google/Cloud SDK/google-cloud-sdk/bin/dev_appserver.py" --port 8484
app.yaml tasks.yaml longtasks.yaml --log_level=debug
INFO 2018-07-29 08:11:15,566 devappserver2.py:178] Skipping SDK update
check.
INFO 2018-07-29 08:11:17,726 api_server.py:274] Starting API server at:
http://localhost:59886
INFO 2018-07-29 08:11:18,259 dispatcher.py:270] Starting module
"default" running at: http://localhost:8484
INFO 2018-07-29 08:11:18,519 dispatcher.py:270] Starting module "tasks"
running at: http://localhost:8485
INFO 2018-07-29 08:11:18,815 dispatcher.py:270] Starting module
"longtasks" running at: http://localhost:8486
INFO 2018-07-29 08:11:18,822 admin_server.py:152] Starting admin server
at: http://localhost:8000
It's also worth noting that I can run other GAE projects in Pycharm and they do run (which leads me to think it's something to do with my configuration of this project).

This is a bug in 2018.2.4 of PyCharm Pro. I had the same issue, and rolled back to 2018.1.5 and it worked fine.
The problem is that 2018.2.4 is loading the cloud-sdk appengine flexible environment classes instead of the standard environment classes.

A hacky way to overcome this is to open <path-to...>/googlecloudsdk/core/util/times.py in a text editor and comment out
from dateutil.tz import _common as tz_common
and
tz_common.PY3 = True # MONKEYPATCH!!! Fixes a Python 2 standard module bug.
Assuming you don't need this "monkeypatch", this solves the issue.

I installed version 211.0.0 -
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-211.0.0-darwin-x86_64.tar.gz
Followed by:
gcloud components install app-engine-python

#user2061291's comments it's true. Pycharm is trying to run google-cloud-sdk/bin/dev_appserver.py. If you want to run app engine from pycharm anyway because you want to debug your code or something else then you should create a new python runner like this:
Script path:
/{YOUR GOOGLE-CLOUD-SDK ROOT
PATH}/google-cloud-sdk/platform/google_appengine/dev_appserver.py
For example:

No module named warnings when starting GAE inside virtualenv locally

I am trying to run my GAE app locally inside a virtual environment.
I've followed these two articles [1], [2] as reference to setup, but when I source evn/bin/activate and then dev_appserver.py ., it keeps raising the error of ImportError: No module named warnings (more trace below)
Surprisingly, if I start it without activating virtual env by just running dev_appserver.py . inside root of project it runs without any issue.
Is there any solution or workaround for this problem?
INFO 2017-08-31 14:09:36,293 devappserver2.py:116] Skipping SDK update check.
INFO 2017-08-31 14:09:36,354 api_server.py:313] Starting API server at: http://localhost:52608
INFO 2017-08-31 14:09:36,357 dispatcher.py:226] Starting module "default" running at: http://localhost:8080
INFO 2017-08-31 14:09:36,359 admin_server.py:116] Starting admin server at: http://localhost:8000
Traceback (most recent call last):
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 103, in <module>
_run_file(__file__, globals())
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 97, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime.py", line 192, in <module>
main()
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime.py", line 172, in main
sandbox.enable_sandbox(config)
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 326, in enable_sandbox
__import__('%s.threading' % dist27.__name__)
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/threading.py", line 11, in <module>
import warnings
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 1076, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named warnings
ERROR 2017-08-31 14:09:39,070 instance.py:280] Cannot connect to the instance on localhost:52366

I've solved it by removing from the skip_files of my .yaml a - venv line. venv is my virtualenv folder, and I still have a - ^venv$ line. In case that helps someone ;-)

All dependencies for standard environment GAE apps (which aren't provided by GAE) must be installed in the app itself, not on your local system. See Using third-party libraries.
Since GAE doesn't care about your local system libraries (besides the basic python 2.7 installation needed to run the development server), using a virtualenv for developing standard env GAE apps doesn't make a lot of sense.
The use of the virtualenv as suggested by the articles you mentioned can actually cause trouble:
local python libraries can interfere with the GAE runtime-provided equivalent libraries (from the SDK) when running locally (I suspect this is somehow what you're experiencing)
at deployment time the content of the lib directory (which includes all the site packages for your python installation) will be uploaded to GAE, potentially causing clashes with the GAE runtime-provided libraries or exceeding the app file quota.
So my suggestion is to drop virtualenv (which, in a way, you did when you skipped the virtualenv activation) and follow the official documentation instead.

I had this problem when running two AppEngine projects in separate directories at once using dev_appserver.py in a virtualenv.
I found it works correctly if I use absolute paths instead.

"ImportError: No module named webapp2" after Linux SDK upgrade (1.9.35 -> 1.9.38)

After upgrading my GAE Python Linux SDK from version 1.9.35 to 1.9.38 my application stopped working, all requests are failing in similar manner:
ERROR 2016-06-10 17:55:06,100 wsgi.py:263]
Traceback (most recent call last):
File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/dancorn/src/apartsw/main/main.py", line 9, in <module>
import webapp2
ImportError: No module named webapp2
INFO 2016-06-10 17:55:06,102 module.py:788] default: "GET / HTTP/1.1" 500 -
I first thought that I somehow screwed up my PyCharm config, but simply rolling back to 1.9.35 (by pointing my /usr/local/google_appengine symlink back to the respective SDK version dir) got things back working as expected.
The SDK release notes don't appear to indicate anything related.
I found a similar report, but the suggested fix (rolling back) is not really acceptable down the road.
I also found this post, but I don't have protobuf (explicitly) installed: How to solve a webapp2 import error after upgrading Google App Engine launcher?. Still chewing on the related posts which led me to this potentially related delta in the SDK, which I'll investigate next:
diff google_appengine_1.9.35/_python_runtime.py google_appengine_1.9.38/_python_runtime.py
71a72,75
> if 'google' in sys.modules:
> google_path = os.path.join(os.path.dirname(__file__), 'google')
> google_module = sys.modules['google']
> google_module.__path__.append(google_path)
74,77c78
<
<
< if 'google' in sys.modules:
< del sys.modules['google']
---
> google_module.__file__ = google_path
Any other investigation leads?

Update: The issue was fixed in SDK version 1.9.40.
The issue is already known to Google, introduced in 1.9.37:
the acknowledged report with more info
tracked as GAE Issue 12963.
From the report thread, as per Jon Parrot:
Thanks everyone for reporting this. We've reproduced the issue and and
fix is incoming. It won't be in the next SDK release (38), but likely
will be in 39 or 40.
Until the fix is released an earlier unaffected SDK version (like 1.9.36) can be found here.

Installing gdata Python package on Heroku

First, a disclaimer that I am new to Python (I come from Ruby). I'm trying to integrate Google Contacts API into my Heroku Python app. Seem to having a hard time figuring out how to push the gdata package (https://code.google.com/p/gdata-python-client/) to Heroku. This is the Heroku error msg:
Traceback (most recent call last):
2014-07-03T09:03:58.205264+00:00 app[web.1]: File "message.py", line 16, in <module>
2014-07-03T09:03:58.205327+00:00 app[web.1]: import atom.data
2014-07-03T09:03:58.205364+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/atom/data.py", line 24, in <module>
2014-07-03T09:03:58.205429+00:00 app[web.1]: import atom.core
2014-07-03T09:03:58.205516+00:00 app[web.1]: ImportError: No module named core
2014-07-03T09:03:59.454062+00:00 heroku[web.1]: State changed from starting to crashed
My first thought was to include gdata and atom in my requirements.txt (it looks like gdata depends on atom)
I tried adding this to my requirements.txt: gdata==2.0.18 atom==0.3.7. However I don't think I'm going about this the right away. The gdata-python-client I downloaded came with a setup.py file that I think I'm supposed to use when pushing to heroku, but I can't seem to get to work either (always get the same error above).
Help is much appreciated :)

I seemed to have gotten around it, but the only way I could fix this error was by removing gdata from requirements.txt and including the /build and /atom folder from https://code.google.com/p/gdata-python-client/ in my root directory.

Add sqlalchemy library to google app engine in pycharm

I have following question to sqlalchemy in gae. I'm using the integrated google app engine sdk from Pycharm. Everything works fine, except the integration of sqlalchemy. The database itself works fine if I debug it with the Python interpreter. But when I'm trying to run it together with the google app engine I'm getting troubles with the import of the sqlalchemy library.
I already tried to solve it with virtualenv, but it didn't work either.
Here is my output log:
Connected to pydev debugger (build 131.618)
WARNING 2014-02-17 10:58:22,887 api_server.py:331] Could not initialize images API; you are likely missing the Python "PIL" module.
WARNING 2014-02-17 10:58:22,895 simple_search_stub.py:1018] Could not read search indexes from c:\users\user\appdata\local\temp\appengine.test\search_indexes
INFO 2014-02-17 10:58:22,947 api_server.py:138] Starting API server at: localhost:60878
INFO 2014-02-17 10:58:22,994 dispatcher.py:171] Starting module "default" running at: localhost:8080
INFO 2014-02-17 10:58:23,032 admin_server.py:117] Starting admin server at: localhost:8000
pydev debugger: process 4452 is connecting
pydev debugger: process 4196 is connecting
ERROR 2014-02-17 09:58:35,742 wsgi.py:262]
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 239, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 298, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 84, in LoadObject
obj = import(path[0])
File "C:\Users\user\PycharmProjects\test\main.py", line 16, in
from bin.database.DBList import DBList
File "C:\Users\user\PycharmProjects\test\bin\database\DBList.py", line 17, in
from sqlalchemy import Column, Integer, String
ImportError: No module named sqlalchemy
INFO 2014-02-17 10:58:36,410 module.py:617] default: "GET / HTTP/1.1" 500 -
So how is it possible to add a third party library in gae?

This is to be expected because GAE has no idea what things your virtualenv is using. To mitigate this problem of using a third party library, all you need to do is download the source code of the library and all its dependents, and put it into a folder underneath your root project directory.
Add the folder to your PYTHONPATH, you can do this via:
import sys; sys.path.append('<your relative path to directory here>')
And then you are all set. By convention this directory that you put all your third party libraries under, is called lib.