Develop Reference

ImportError: Importing the devappserver sandbox module from user application code is not permitted

I'm working on a Google App Engine application written in Python (Standard Environment) and I need to whitelist some additional modules in the development server environment.
I have been doing it using this code in the appengine_config file for a long time, and it worked very well:
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
As answered in this question, some time ago the Google Cloud SDK was updated and the previous import caused an ImportError. The import just needed to be changed to:
from google.appengine.tools.devappserver2.python.runtime import sandbox
With this change, everything worked well again, until I updated the Cloud SDK to version: 186.0.0.
Now it seems a "SandboxAccessPreventionImportHook" class has been added to the sandbox module so it can't be imported from the App Engine application. This is the error the application is raising:
ImportError: Importing the devappserver sandbox module (google.appengine.tools.devappserver2.python.runtime.sandbox) from user application code is not permitted.
Does anyone have an idea on how to bypass this? Or is there another way to whitelist modules in the development server environment?
Thanks!!!

We had the exact same issue as OP but unfortunately Alex's solution didn't work for us.
While a really really hacky solution this is what worked for us. Please be aware that the modification of the sandbox must be redone after an update.
Modify the sandbox.py file located at
{appengine_sdk_dir}/google/appengine/tools/devappserver2/python/runtime/sandbox.py
and add _ssl and _socket like in Alex's first example to the _WHITE_LIST_C_MODULES list.
_WHITE_LIST_C_MODULES = [
// keep existing items
'_ssl',
'_socket'
]
Then we removed the imports and sandbox overrides from appengine_config.py.
vendor.add('lib')
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
# handle requests_toolbelt's monkeypatch as you see fit.
Let's hope there is a day when this is no longer necessary!

Context
So for me the root of this issue was the dev_appserver.py's inability to send outbound https requests / sockets.
The solution then was to put this in my appengine_config.py:
vendor.add('lib')
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
try:
from google.appengine.tools.devappserver2.python import sandbox
except ImportError:
from google.appengine.tools.devappserver2.python.runtime import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
else:
# Doing this on dev_appserver/localhost seems to cause outbound https requests to fail
import requests
from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()
Today I upgraded my cloud sdk and started getting
ImportError: Importing the devappserver sandbox module (google.appengine.tools.devappserver2.python.runtime.sandbox) from user application code is not permitted.
If I remove the _WHITE_LIST_C_MODULES stuff, I then get this error when using the python requests library to make outbound https requests:
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch.py", line 293, in fetch
return rpc.get_result()
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch.py", line 413, in _get_fetch_result
rpc.check_success()
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
self.__rpc.CheckSuccess()
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
self.request, self.response)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 222, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 241, in _MakeRealSyncCall
request_pb.set_request(request.Encode())
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/net/proto/ProtocolBuffer.py", line 103, in Encode
self.Output(e)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/net/proto/ProtocolBuffer.py", line 347, in Output
self.OutputUnchecked(e)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch_service_pb.py", line 481, in OutputUnchecked
out.putDouble(self.deadline_)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/net/proto/ProtocolBuffer.py", line 592, in putDouble
a.fromstring(struct.pack("<d", v))
error: required argument is not a float
I then found that stack trace in this issue
https://github.com/requests/requests/issues/4078
which seemed to indicate that this only started happening after python-requests version 2.16.0
Solution
All my third party libs were installed to a folder in my project root called lib using
pip install -t lib
Now, I have lib & localhost_libs and I did:
pip install -t localhost_libs requests==2.16
My appengine_config.py now has this instead:
vendor.add('lib')
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
vendor.add('localhost_libs')
import pkg_resources
pkg_resources.require("requests==2.16.0")
import requests
print "requests.__version__", requests.__version__
from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()
print "Appengine config done"
Edit: Modified solution to use pkg_resources and not require a prod_libs folder

The location of the sandbox module got moved into the runtime module.
from google.appengine.tools.devappserver2.python import runtime
runtime.sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']

The way #karloskar mentioned was fixed my case.
However, after I modified sandbox.py to allow _ssl and _socket to be WHITE_LIST, I got another library import issue - ImportError: No module named google.auth.
For who also got above google.auth import error, you could consider to update your appengine_config.py like below. That's the way fixed my import issue.
# appengine_config.py
import os
import google
from google.appengine.ext import vendor
lib_directory = os.path.dirname(__file__) + '/lib'
# Change where to find the google package (point to the lib/ directory)
google.__path__ = [os.path.join(lib_directory, 'google')] + google.__path__
# Add any libraries install in the "lib" folder.
vendor.add(lib_directory)
Reference1:
App Engine not finding google.auth file in dev_appserver.py
Reference2:
https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/169#issuecomment-315417916

Error importing psutil in PyCharm Google App Engine Project

I have recently run into an import error with psutils in the types.py called from google.cloud.pubsub_v1 on a GAE standard project inside Pycharm. I messed around for a day trying to find away around it, but no go. psutil imports in python run from the terminal and the console and from a vanilla python project but not from a GAE project. To double up on this I made a basic test of the two as seen below: Both using the same virtualenv...
Standard Vanilla Python Project:
main.py contains
import psutil
def x():
print "hello world"
if __name__ == "__main__":
x()
Google App Engine Project -
main.py contains
import webapp2
import psutil
class MainHandler(webapp2.RequestHandler):
def get(self):
self.response.write('Hello world!')
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)
The import in the vanilla python project works fine, but the import in the GAE app returns the traceback:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/Users/chez 1/google-app-engine-projects/untitled/main.py", line 18, in <module>
import psutil
ImportError: No module named psutil
Has anyone had similar problems and found a fix?

You're likely having the psutil package installed on your local system (thus working in terminal/console), but not vendored into your app (i.e installed in your app's lib dir) as required for standard env apps (the SDK emulates the real GAE, for which whatever is installed into your local python installation means nothing). From Installing a third-party library:
Create a directory to store your third-party libraries, such as lib/.
mkdir lib
Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous
step. For example:
pip install -t lib/ <library_name>
So just vendor into your app the psutil package.

In my experience this and similar 'No module named ...' errors have been caused by an incorrectly defined python interpreter. See the answers at: ImportError: No module named 'bottle' - PyCharm

Use ipdb with webapp2

I am django developer recently starting with webapp2 project. My question is how do I enter into debugging mode in webapp2 application using ipdb package as we do in django and flask.
my app structure:
helloapp
- libs/
- stylesheets/
- templates/
- .gitignore
- app.yaml
- index.yaml
- main.py
- webapp2.py
I have installed ipdb in libs folder using
sudo pip install -t github_projects/hellowebapp2/libs ipdb
main.py
from .libs import ipdb
class HelloWebapp2(webapp2.RedirectHandler):
def get(self):
import ipdb; ipdb.set_trace()
Error
/home/kishan/github_projects/hellowebapp2/main.py
ERROR 2016-11-07 06:48:01,566 wsgi.py:263]
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/kishan/github_projects/hellowebapp2/main.py", line 12, in <module>
from .libs import ipdb
ValueError: Attempted relative import in non-package
Update
I followed this link now I get different error ImportError: No module named termios :(

Its recommended that You should use virtualenvironments. Install virtualenvwrapper using pip.
pip install virtualenvwrapper
After that open your bashrc and insert relevant lines as mentioned in virtualenvwrapper docs. Then create a virtual environment.
mkvirtualenv myapp
install all python packages including webapp2 to your virtual environment.
pip install webapp2
pip install webob
pip install paste
pip install ipdb
This way, all your packages will be at one single location (/home/username/.virtualenvs/myapp)
And you can import any installed package you want without headaches from relative imports. And for missing termios, you may check SO Post

ImportError when importing elasticsearch.helpers in Python

I am using a docker container with Ubuntu 16.04 Xenial with a Python 3.5.2 virtual environment. Every time I am trying to initialize the server uWSGI I am getting the following Python error:
File "/env/lib/python3.5/site-packages/elasticsearch_dsl/__init__.py", line 5, in <module>
from .search import Search
File "/env/lib/python3.5/site-packages/elasticsearch_dsl/search.py", line 3, in <module>
from elasticsearch.helpers import scan
ImportError: No module named 'elasticsearch.helpers'
I have been running the bash manually inside the docker container to see what is wrong with the module. By running pip list in the virtual environment I can tell that the packages are correctly installed:
# pip list
elasticsearch (2.1.0)
elasticsearch-dsl (0.0.8)
Running sys.path shows that the site packages folder was correctly added to the python path:
import sys
sys.path
[
'',
'/env/lib/python35.zip',
'/env/lib/python3.5',
'/env/lib/python3.5/plat-x86_64-linux-gnu',
'/env/lib/python3.5/lib-dynload',
'/usr/lib/python3.5',
'/usr/lib/python3.5/plat-x86_64-linux-gnu',
'/env/lib/python3.5/site-packages'
]
There is a __init__.py file within the folder elasticsearch.helpers so that's not the problem. If I try the import from the Python console it will fail as well:
from elasticsearch.helpers import scan
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'elasticsearch.helpers'
But now if I go to the site-packages folder cd /env/lib/python3.5/site-packages and run python console from here, the import will work.
I don't really know why this is happening any help will be appreciated.

ouch... just a rookie mistake. One of the folders of my project was called elasticsearch and that was causing the issue.
Running the following command I found out that my app was simply loading the elasticsearch module from a different location.
import elasticsearch
import pprint, os
pprint.pprint(os.path.abspath(elasticsearch.__file__))
# /var/sites/my_app/elasticsearch/__init__.py

ImportError: cannot import name NullHandler

I'm trying to deploy a hello-world type app on Elastic Beanstalk. Just about everything seems to work, packages are installed, etc. up to the point where mod_wsgi attempts to retrieve the "application" object from wsgi.py. At that point, the following appears in the logs (once in the logs for each unsuccessfuly HTTP request):
mod_wsgi (pid=6114): Target WSGI script '/opt/python/current/app/myapp/wsgi.py' cannot be loaded as Python module.
mod_wsgi (pid=6114): Exception occurred processing WSGI script '/opt/python/current/app/myapp/wsgi.py'.
Traceback (most recent call last):
File "/opt/python/current/app/caserails/wsgi.py", line 20, in <module>
application = get_wsgi_application()
File "/opt/python/run/venv/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
django.setup()
File "/opt/python/run/venv/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
from django.utils.log import configure_logging
File "/opt/python/run/venv/lib/python2.7/site-packages/django/utils/log.py", line 16, in <module>
from logging import NullHandler # NOQA
ImportError: cannot import name NullHandler
Link to concurrent AWS Forum Post.

The NullHandler is only available on Python version 2.7+. You could create the NullHandler yourself on an ImportError:
import logging
try:
from logging import NullHandler
except ImportError:
class NullHandler(logging.Handler):
def emit(self, record):
pass
logging.getLogger(__name__).addHandler(NullHandler())
More information about logging.NullHandler: https://docs.python.org/3/library/logging.handlers.html#logging.NullHandler.

I had similar problem and in my case the issue was that for some unrelated project I created logging.py file in home folder and when I ran something in home, it was importing this file instead of the real module.
You can check which file is being imported like this:
import logging
print(logging.__file__)
I fixed it by deleting logging.py I created previously.

NullHandler was introduced in version 2.7. Are you sure you are running 2.7 on your server?

After much trial and error, the (immediate) problem was solved by removing python logging from requirements.txt and rebuilding the environment.
I do not yet fully understand why this is a problem. On my local machine, I'm able to install logging and run Django without error. I suspect as vikramls pointed out that something weird is happening in the intersection between mod_wsgi, the baselineenv in which mod_wsgi executes, and the virtualenv in which my app operates.
But, at least for now, I'm fixing this error by not including "logging" in requirements.txt when deploying Django 1.7 on Elastic Beanstalk.

I had this issue while creating a build for python 2.7 with PyInstaller.So, I uninstalled logger from my ENV.else you can also remove the package name from the requirement.txt.
This approach solves my issue [Happy Coding:)]