I have recently run into an import error with psutils in the types.py called from google.cloud.pubsub_v1 on a GAE standard project inside Pycharm. I messed around for a day trying to find away around it, but no go. psutil imports in python run from the terminal and the console and from a vanilla python project but not from a GAE project. To double up on this I made a basic test of the two as seen below: Both using the same virtualenv...
Standard Vanilla Python Project:
main.py contains
import psutil
def x():
print "hello world"
if __name__ == "__main__":
x()
Google App Engine Project -
main.py contains
import webapp2
import psutil
class MainHandler(webapp2.RequestHandler):
def get(self):
self.response.write('Hello world!')
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)
The import in the vanilla python project works fine, but the import in the GAE app returns the traceback:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/Users/chez 1/google-app-engine-projects/untitled/main.py", line 18, in <module>
import psutil
ImportError: No module named psutil
Has anyone had similar problems and found a fix?
You're likely having the psutil package installed on your local system (thus working in terminal/console), but not vendored into your app (i.e installed in your app's lib dir) as required for standard env apps (the SDK emulates the real GAE, for which whatever is installed into your local python installation means nothing). From Installing a third-party library:
Create a directory to store your third-party libraries, such as lib/.
mkdir lib
Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous
step. For example:
pip install -t lib/ <library_name>
So just vendor into your app the psutil package.
In my experience this and similar 'No module named ...' errors have been caused by an incorrectly defined python interpreter. See the answers at: ImportError: No module named 'bottle' - PyCharm
Related
I'm new to flask framework and I want to write a simple flask app that uses another python module (librosa package). I have successfully installed librosa in the same virtual environment that I have installed flask and I can easily import it in the python interpreter. Here is the python script.
# app.py
from flask import Flask
import librosa
app = Flask(__name__)
#app.route('/')
def hello():
return 'Hello'
if __name__ == '__main__':
app.run()
The problem is that when I want to run the flask app using the flask run command (after setting export FLASK_APP=app.py) I get the ModuleNotFoundError error as follows:
Error: While importing "app", an ImportError was raised:
Traceback (most recent call last):
File "/home/masoud/anaconda3/lib/python3.7/site-packages/flask/cli.py", line 240, in locate_app
__import__(module_name)
File "/home/masoud/projects/my_projects/seperator_rewriten/app.py", line 2, in <module>
import librosa
ModuleNotFoundError: No module named 'librosa'
The strange thing is that there is no import error when I run the flask server using python app.py command.
if you are using an IDE such as pycharm, then you may need to install it from the terminal in the IDE itself, not cmd
This may be the solution to your problem:
python -m flask run
The reason is that flask run may use the python executable somewhere else, not the virtual environment you created for the project.
You can add these lines of code in your script:
import sys
import flask
print(sys.executable)
print(flask.__version__)
And then you can check the python executable path using python app.py and flask run.
The problem
I start a flask app, which works fine when I use "run" or "debug." However, I got the error when use "profile," here is the log message
In folder C:/Users/Myname/PycharmProjects/ProjectName
"C:\Users\Myname\Anaconda3\envs\ProjectName\python.exe" "C:\Program Files\JetBrains\PyCharm
Professional Edition with Anaconda plugin 2019.2.4\helpers\profiler\run_profiler.py" 127.0.0.1 12055 -m flask run
Starting cProfile profiler
Traceback (most recent call last):
Snapshot saved to C:\Users\Myname/.PyCharm2019.2/system\snapshots\ProjectName1.pstat
File "C:\Program Files\JetBrains\PyCharm Professional Edition with Anaconda plugin 2019.2.4\helpers\profiler\run_profiler.py", line 173, in <module>
profiler.run(file)
File "C:\Program Files\JetBrains\PyCharm Professional Edition with Anaconda plugin 2019.2.4\helpers\profiler\run_profiler.py", line 89, in run
execfile(file, globals, globals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Professional Edition with Anaconda plugin 2019.2.4\helpers\profiler\prof_util.py", line 30, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:\Users\Myname\Anaconda3\envs\ProjectName\lib\site-packages\flask\__init__.py", line 19, in <module>
from . import json
ImportError: attempted relative import with no known parent package
What I have tried
I google it and found that it is an old problem.
Can't profile module using relative imports
Pycharm profile is not working
How to reproduce it
If you would like to reproduce it, you can create a flask app using pycharm, and then click "profile."
The default code for flask app is as follows. But I don't think it matters.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
app.run()
The error message can be different due to the change of the flask version or Pycharm version, but the reason is the same: File "xxx\lib\site-packages\flask\__init__.py", line 19, in <module> use relative import and cause the ImportError
My question
My question is, is there a way to make it work? Any hacks or tricks are also appreciated. I just want to use the profile now. Thank you
Apparently this is a long-standing issue with PyCharm's profiler, it does not run packages correctly resulting in sys.path being wonky and relative imports failing: https://youtrack.jetbrains.com/issue/PY-28509
I guess an option would be to make your own "launcher script" simply copying what flask.__main__ does.
Though things might also work out if you run -m flask.cli instead of just -m flask (I don't know, I didn't test it): it does the same thing but does not invoke a __main__ file so it could behave better.
[New solution]
Thanks to #Masklinn, use -m flask.cli instread of -m flask could solve this problem. However, the configuration in pycharm flask use -m flask by default and I cannot find out a way to modify it. Therefore, I suggest to use pycharm python. The script path should be set to your FLASK_APP, and add -m flask.cli in addtional options.
[Obsolete solution]
One solution is that directly modify the __init__.py to make it work. For example, change
from . import json
from .app import Flask
to
from flask import json
from flask.app import Flask
I'm working on a Google App Engine application written in Python (Standard Environment) and I need to whitelist some additional modules in the development server environment.
I have been doing it using this code in the appengine_config file for a long time, and it worked very well:
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
As answered in this question, some time ago the Google Cloud SDK was updated and the previous import caused an ImportError. The import just needed to be changed to:
from google.appengine.tools.devappserver2.python.runtime import sandbox
With this change, everything worked well again, until I updated the Cloud SDK to version: 186.0.0.
Now it seems a "SandboxAccessPreventionImportHook" class has been added to the sandbox module so it can't be imported from the App Engine application. This is the error the application is raising:
ImportError: Importing the devappserver sandbox module (google.appengine.tools.devappserver2.python.runtime.sandbox) from user application code is not permitted.
Does anyone have an idea on how to bypass this? Or is there another way to whitelist modules in the development server environment?
Thanks!!!
We had the exact same issue as OP but unfortunately Alex's solution didn't work for us.
While a really really hacky solution this is what worked for us. Please be aware that the modification of the sandbox must be redone after an update.
Modify the sandbox.py file located at
{appengine_sdk_dir}/google/appengine/tools/devappserver2/python/runtime/sandbox.py
and add _ssl and _socket like in Alex's first example to the _WHITE_LIST_C_MODULES list.
_WHITE_LIST_C_MODULES = [
// keep existing items
'_ssl',
'_socket'
]
Then we removed the imports and sandbox overrides from appengine_config.py.
vendor.add('lib')
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
# handle requests_toolbelt's monkeypatch as you see fit.
Let's hope there is a day when this is no longer necessary!
Context
So for me the root of this issue was the dev_appserver.py's inability to send outbound https requests / sockets.
The solution then was to put this in my appengine_config.py:
vendor.add('lib')
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
try:
from google.appengine.tools.devappserver2.python import sandbox
except ImportError:
from google.appengine.tools.devappserver2.python.runtime import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
else:
# Doing this on dev_appserver/localhost seems to cause outbound https requests to fail
import requests
from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()
Today I upgraded my cloud sdk and started getting
ImportError: Importing the devappserver sandbox module (google.appengine.tools.devappserver2.python.runtime.sandbox) from user application code is not permitted.
If I remove the _WHITE_LIST_C_MODULES stuff, I then get this error when using the python requests library to make outbound https requests:
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch.py", line 293, in fetch
return rpc.get_result()
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch.py", line 413, in _get_fetch_result
rpc.check_success()
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
self.__rpc.CheckSuccess()
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
self.request, self.response)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 222, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 241, in _MakeRealSyncCall
request_pb.set_request(request.Encode())
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/net/proto/ProtocolBuffer.py", line 103, in Encode
self.Output(e)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/net/proto/ProtocolBuffer.py", line 347, in Output
self.OutputUnchecked(e)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch_service_pb.py", line 481, in OutputUnchecked
out.putDouble(self.deadline_)
File "/Users/alexindaco/google-cloud-sdk/platform/google_appengine/google/net/proto/ProtocolBuffer.py", line 592, in putDouble
a.fromstring(struct.pack("<d", v))
error: required argument is not a float
I then found that stack trace in this issue
https://github.com/requests/requests/issues/4078
which seemed to indicate that this only started happening after python-requests version 2.16.0
Solution
All my third party libs were installed to a folder in my project root called lib using
pip install -t lib
Now, I have lib & localhost_libs and I did:
pip install -t localhost_libs requests==2.16
My appengine_config.py now has this instead:
vendor.add('lib')
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
vendor.add('localhost_libs')
import pkg_resources
pkg_resources.require("requests==2.16.0")
import requests
print "requests.__version__", requests.__version__
from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()
print "Appengine config done"
Edit: Modified solution to use pkg_resources and not require a prod_libs folder
The location of the sandbox module got moved into the runtime module.
from google.appengine.tools.devappserver2.python import runtime
runtime.sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
The way #karloskar mentioned was fixed my case.
However, after I modified sandbox.py to allow _ssl and _socket to be WHITE_LIST, I got another library import issue - ImportError: No module named google.auth.
For who also got above google.auth import error, you could consider to update your appengine_config.py like below. That's the way fixed my import issue.
# appengine_config.py
import os
import google
from google.appengine.ext import vendor
lib_directory = os.path.dirname(__file__) + '/lib'
# Change where to find the google package (point to the lib/ directory)
google.__path__ = [os.path.join(lib_directory, 'google')] + google.__path__
# Add any libraries install in the "lib" folder.
vendor.add(lib_directory)
Reference1:
App Engine not finding google.auth file in dev_appserver.py
Reference2:
https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/169#issuecomment-315417916
I'm figuring out how to deploy my script on google cloud platform.
ive already made a directory or folder that contains the script.pyand all the libraries in /lib folder.
what i dont get is setting up my app.yaml to run script.py (python 2.7) and access lib if it needs to.
I also dont know if i need to make requirments.txt since im using third party libraries.
here are all my imports inside script.py
import requests
import re
import mysql.connector
from urlparse import urlparse
from urlparse import urljoin
from bs4 import BeautifulSoup
Also, what i have in my lib are BeautifulSoup,requests and mysql.connector.
i dont know about the others i assume they're python2.7 built in since i cant install them using pip.
im using windows 10 by the way.
app.yaml
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /lib/requests
script: Scrape.app
handlers:
- url: /requests
script: Scrape.app
handlers:
- url: /mysql/connector
script: Scrape.app
handlers:
- url: /bs4/
script: Scrape.app
cron.yaml
cron:
- description: "Scrape"
url: /
schedule: every 10 mins
retry_parameters:
min_backoff_seconds: 2.5
max_doublings: 10
im getting errors like
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
/bin/sh: 1: Python: not found
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~tribal-bonito-157700/20170302t182530.399552845921654287/Scrape.py", line 3, in <module>
import requests
ImportError: No module named requests
From the script row in the Handlers element table:
A script: directive must be a python import path, for example,
package.module.app that points to a WSGI application. The last component of a script: directive using a Python module path is
the name of a global variable in the module: that variable must be a
WSGI app, and is usually called app by convention.
Note: just like for a Python import statement, each subdirectory
that is a package must contain a file named __init__.py
I'd recommend spending some time going through the code snippets from Quickstart for Python App Engine Standard Environment, where you'll see a basic structure of a simple app.
A requirements.txt file can be used to specify the list of packages to be installed in the lib directory, like this:
pip install -r requirements.txt -t lib
But it's not absolutely necessary, packages can be explicitly specified directly on the pip cmdline as well.
I'm trying to deploy a hello-world type app on Elastic Beanstalk. Just about everything seems to work, packages are installed, etc. up to the point where mod_wsgi attempts to retrieve the "application" object from wsgi.py. At that point, the following appears in the logs (once in the logs for each unsuccessfuly HTTP request):
mod_wsgi (pid=6114): Target WSGI script '/opt/python/current/app/myapp/wsgi.py' cannot be loaded as Python module.
mod_wsgi (pid=6114): Exception occurred processing WSGI script '/opt/python/current/app/myapp/wsgi.py'.
Traceback (most recent call last):
File "/opt/python/current/app/caserails/wsgi.py", line 20, in <module>
application = get_wsgi_application()
File "/opt/python/run/venv/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
django.setup()
File "/opt/python/run/venv/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
from django.utils.log import configure_logging
File "/opt/python/run/venv/lib/python2.7/site-packages/django/utils/log.py", line 16, in <module>
from logging import NullHandler # NOQA
ImportError: cannot import name NullHandler
Link to concurrent AWS Forum Post.
The NullHandler is only available on Python version 2.7+. You could create the NullHandler yourself on an ImportError:
import logging
try:
from logging import NullHandler
except ImportError:
class NullHandler(logging.Handler):
def emit(self, record):
pass
logging.getLogger(__name__).addHandler(NullHandler())
More information about logging.NullHandler: https://docs.python.org/3/library/logging.handlers.html#logging.NullHandler.
I had similar problem and in my case the issue was that for some unrelated project I created logging.py file in home folder and when I ran something in home, it was importing this file instead of the real module.
You can check which file is being imported like this:
import logging
print(logging.__file__)
I fixed it by deleting logging.py I created previously.
NullHandler was introduced in version 2.7. Are you sure you are running 2.7 on your server?
After much trial and error, the (immediate) problem was solved by removing python logging from requirements.txt and rebuilding the environment.
I do not yet fully understand why this is a problem. On my local machine, I'm able to install logging and run Django without error. I suspect as vikramls pointed out that something weird is happening in the intersection between mod_wsgi, the baselineenv in which mod_wsgi executes, and the virtualenv in which my app operates.
But, at least for now, I'm fixing this error by not including "logging" in requirements.txt when deploying Django 1.7 on Elastic Beanstalk.
I had this issue while creating a build for python 2.7 with PyInstaller.So, I uninstalled logger from my ENV.else you can also remove the package name from the requirement.txt.
This approach solves my issue [Happy Coding:)]