Develop Reference

No module named warnings when starting GAE inside virtualenv locally

I am trying to run my GAE app locally inside a virtual environment.
I've followed these two articles [1], [2] as reference to setup, but when I source evn/bin/activate and then dev_appserver.py ., it keeps raising the error of ImportError: No module named warnings (more trace below)
Surprisingly, if I start it without activating virtual env by just running dev_appserver.py . inside root of project it runs without any issue.
Is there any solution or workaround for this problem?
INFO 2017-08-31 14:09:36,293 devappserver2.py:116] Skipping SDK update check.
INFO 2017-08-31 14:09:36,354 api_server.py:313] Starting API server at: http://localhost:52608
INFO 2017-08-31 14:09:36,357 dispatcher.py:226] Starting module "default" running at: http://localhost:8080
INFO 2017-08-31 14:09:36,359 admin_server.py:116] Starting admin server at: http://localhost:8000
Traceback (most recent call last):
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 103, in <module>
_run_file(__file__, globals())
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 97, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime.py", line 192, in <module>
main()
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime.py", line 172, in main
sandbox.enable_sandbox(config)
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 326, in enable_sandbox
__import__('%s.threading' % dist27.__name__)
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/threading.py", line 11, in <module>
import warnings
File "/usr/local/share/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 1076, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named warnings
ERROR 2017-08-31 14:09:39,070 instance.py:280] Cannot connect to the instance on localhost:52366

I've solved it by removing from the skip_files of my .yaml a - venv line. venv is my virtualenv folder, and I still have a - ^venv$ line. In case that helps someone ;-)

All dependencies for standard environment GAE apps (which aren't provided by GAE) must be installed in the app itself, not on your local system. See Using third-party libraries.
Since GAE doesn't care about your local system libraries (besides the basic python 2.7 installation needed to run the development server), using a virtualenv for developing standard env GAE apps doesn't make a lot of sense.
The use of the virtualenv as suggested by the articles you mentioned can actually cause trouble:
local python libraries can interfere with the GAE runtime-provided equivalent libraries (from the SDK) when running locally (I suspect this is somehow what you're experiencing)
at deployment time the content of the lib directory (which includes all the site packages for your python installation) will be uploaded to GAE, potentially causing clashes with the GAE runtime-provided libraries or exceeding the app file quota.
So my suggestion is to drop virtualenv (which, in a way, you did when you skipped the virtualenv activation) and follow the official documentation instead.

I had this problem when running two AppEngine projects in separate directories at once using dev_appserver.py in a virtualenv.
I found it works correctly if I use absolute paths instead.

How to configure Apache 2 to run a Python script from a Ruby CGI app with %x{}

My Ruby CGI app is very simple, I would like to call a Python script, for example youtube-dl, or youtube-upload with %x{youtube-dl --help}
For the sake of simplicity, I would like to print only the help page of the youtube-dl Python script. So my Ruby script is also very simple:
#!/usr/bin/ruby
require "cgi"
cgi=CGI.new(:accept_charset => "UTF-8")
url=cgi['url']
puts "Content-Type: text/plain; charset=\"UTF-8\""
puts "Access-Control-Allow-Origin: *"
puts
puts url
puts "-----------------------------------"
puts %x{youtube-dl --help 2>&1}
Then I can invoke this CGI app called ytdl with http://example.com/cgi-bin/ytdl?url=a_youtube_url. Unfortunately I get only a lot of error messages from the Python interpreter which is unable to import some packages, especially site-packages:
Traceback (most recent call last):
File "/usr/bin/youtube-dl", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 36, in <module>
import plistlib
File "/usr/lib/python3.4/plistlib.py", line 65, in <module>
from xml.parsers.expat import ParserCreate
File "/usr/lib/python3.4/xml/parsers/expat.py", line 4, in <module>
from pyexpat import *
ImportError: /usr/lib/python3.4/lib-dynload/pyexpat.cpython-34m.so: undefined symbol: XML_SetHashSalt
I think I would configure my Apache 2 web server to provide some environment variables for the Python interpereter, where it can find the packages. Unfortunately I have no experience with Python, I only learned Ruby till now. My Apache server runs CGI apps as user daemon, and when I login as daemon I can run the youtube-dl script without error messages.
Apache provide these environment for my CGI apps (printed with puts %x{env}):
UNIQUE_ID=VZN9RyX3N7MAAAs95zsAAAAI
RUBYOPT=rubygems
GEM_HOME=/home/XXXXXX/.gem/ruby/2.2.0
HTTP_USER_AGENT=Opera/9.80 (X11; Linux i686) Presto/2.12.388 Version/12.16
HTTP_HOST=xx.xxx.xx.xxx
HTTP_ACCEPT=text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.9
HTTP_ACCEPT_ENCODING=gzip, deflate
HTTP_CONNECTION=Keep-Alive
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/XXXXXX/.gem/ruby/2.2.0/bin:/home/XXXXXX/util
LD_LIBRARY_PATH=/opt/lampp/lib:/opt/lampp/lib
SERVER_SIGNATURE=
SERVER_SOFTWARE=Apache/2.4.9 (Unix) OpenSSL/1.0.1g PHP/5.5.11 mod_perl/2.0.8-dev Perl/v5.16.3
SERVER_NAME=xx.xxx.xx.xxx
SERVER_ADDR=xx.xxx.xx.xxx
SERVER_PORT=80
REMOTE_ADDR=xx.x.xxx.xxx
DOCUMENT_ROOT=/opt/lampp/htdocs
REQUEST_SCHEME=http
CONTEXT_PREFIX=/cgi-bin/
CONTEXT_DOCUMENT_ROOT=/opt/lampp/cgi-bin/
SERVER_ADMIN=you#example.com
SCRIPT_FILENAME=/opt/lampp/cgi-bin/ytdl
REMOTE_PORT=37798
GATEWAY_INTERFACE=CGI/1.1
SERVER_PROTOCOL=HTTP/1.1
REQUEST_METHOD=GET
QUERY_STRING=
REQUEST_URI=/cgi-bin/ytdl
SCRIPT_NAME=/cgi-bin/ytdl
Meanwhile I realized that when I set the environment variable (either in Apache config httpd.conf with SetEnv directive, or with ENV['PYTHONHOME']=value in my Ruby script)
called PYTHONHOME to an empty string "" or "/usr/lib" , the error message disappers, and I get another one:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
When I set:
ENV['PYTHONHOME']="/usr/lib"
ENV['PYTHONPATH']="/usr/lib/python3.4"
I get another error message:
File "/usr/bin/youtube-dl", line 5, in <module>
from pkg_resources import load_entry_point
ImportError: No module named 'pkg_resources'
So my question is, how should I configure my Apache 2 webserver to make Python scripts run and make them callable from other apps?

When you run an application in a subshell you have to be aware of the environment being set up for that shell and application. The PATH and variables will be nonexistent or very limited which can affect its ability to find libraries.
Write a small CGI that outputs the Apache variables and environment passed to the script, either as a web page or to the log or a file and examine the results.
Also permissions and the user will be whatever the server sets up which is usually very restricted.
Also, seriously consider not using CGI. Instead use something more modern and flexible. Sinatra works nicely with Apache using something like Passenger to glue them together. Sinatra makes it really easy to do web-services, and actually works so well you can often do without a heavyweight server like Apache, especially during the development, test and early production stages. I have an API used in-house that handles a lot of enterprise-critical requests and it just cruises along.

After adding Django CMS to my project, management commands fail with a cache error. How can I fix it?

This is using Django 1.7 and Django CMS 3.1.0.
I've had a large Django project that has been in production for months without any issue. I'm currently adding Django CMS to it and found that if my cache is offline, I cannot issue any management commands without getting an exception. It used to work fine until I added Django CMS to my project.
This can be replicated by:
Manually installing Django CMS by following the instructions in the documentation.
Setting a default cache that connects to a cache service that can be offline. In my case I connect to a Redis server which is for my Django project's use only. This server is started manually before I run manage.py runserver in development and before my web application is brought online in production. Turning on the cache service makes the issue go away but sometimes I want to run some management commands with the cache service turned off.
If I run any management command when my Redis instance is offline, I get a failure. For instance, if I just run ./manage.py without a command, I get a trace like this one:
Traceback (most recent call last):
File "./manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute
django.setup()
File "env/local/lib/python2.7/site-packages/django/__init__.py", line 21, in setup
apps.populate(settings.INSTALLED_APPS)
File "env/local/lib/python2.7/site-packages/django/apps/registry.py", line 115, in populate
app_config.ready()
File "env/local/lib/python2.7/site-packages/django/contrib/admin/apps.py", line 22, in ready
self.module.autodiscover()
File "env/local/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 23, in autodiscover
autodiscover_modules('admin', register_to=site)
File "env/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 74, in autodiscover_modules
import_module('%s.%s' % (app_config.name, module_to_search))
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "env/local/lib/python2.7/site-packages/cms/admin/__init__.py", line 11, in <module>
plugin_pool.plugin_pool.discover_plugins()
File "env/local/lib/python2.7/site-packages/cms/plugin_pool.py", line 33, in discover_plugins
invalidate_cms_page_cache()
File "env/local/lib/python2.7/site-packages/cms/views.py", line 335, in invalidate_cms_page_cache
version = _get_cache_version()
File "env/local/lib/python2.7/site-packages/cms/views.py", line 280, in _get_cache_version
version = cache.get(CMS_PAGE_CACHE_VERSION_KEY)
File "env/local/lib/python2.7/site-packages/django_redis/cache.py", line 30, in _decorator
raise e.parent
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/tmp/foo/redis/foo_dev.redis.sock. No such file or directory.
If I just remove the cms app from INSTALLED_APPS, this is enough to prevent the problem but this is not an acceptable fix.
Is there a way I can fix this without removing Django CMS or having to turn on my cache service before I issue any management command?

The Problem
The issue is the same for Django 1.7 and 1.8 and for foreseeable future version. It is not impossible to happen with 1.6 and earlier versions but unlikely.
The issue is that if you use the configuration most likely to be used for a Django project, then Django CMS will cause an access to the cache to be made whenever the cms application is initialized.
Here is what happens when you run a management command, with some irrelevant steps omitted:
Django goes through the app initialization procedure. (Documented here.) When it gets to django.contrib.admin, it uses the default AppConfig for this app.
The default AppConfig for admin calls admin's autodiscover() function. (Documented here.)
Eventually autodiscover() gets to cms' own admin.py file, and loads it. This file initiates the discovery of Django CMS' plugins, which in turn causes Django CMS to invalidate its page cache. So you get a cache access there, which fails because your cache service is not turned on.
The Solution
What you can do is, as the documentation about admin explains, use django.contrib.admin.apps.SimpleAdminConfig in your INSTALLED_APPS rather than django.contrib.admin. This app configuration does not call autodiscover() automatically. When you use this configuration, then you must call autodiscover() yourself. Putting the call in your project-wide urls.py, like we were required to do prior to 1.7, is your best bet:
from django.contrib import admin
admin.autodiscover()
The above should make it clear that the OP's problem is unlikely to happen with Django versions prior to 1.6. People who followed Django's documentation to configure their project would have the call to autodiscover() in their urls.py file, which is read only when a network query is made. However, the problem can happen for for people who diverged from the recommended practices and ended up with a project that calls autodiscover() elsewhere.

Upgrade python to 2.7 and Django to 1.4, but it tells me No module named *

I have just upgraded my python to 2.7 on my Ubuntu server, and reinstalled the Django to 1.4. I use python flup to run the Django application as fastcgi, but when I try to restart the application, it keeps telling me No module named *
e.g.
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
...
File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
ImportError: No module named *
The issue was initiated by manage.py file in the old Django (1.3) project, but according to the document https://docs.djangoproject.com/en/dev/releases/1.4/#updated-default-project-layout-and-manage-py
that's how I run the project using python-flup
exec python ./manage.py runfcgi host=127.0.0.1 port=8083 maxchildren=40 pidfile=$PIDFILE --settings=expand.settings
The old-style manage.py will continue to work as before until Django 1.6. In 1.5 it will raise DeprecationWarning

I'm guesing by looking at django's source code that there's something wrong with either the layout of your project or your python path.
execute_from_command_line(sys.argv)
Runs the following snippet (from django/core/management/init.py), which is most likely the cause of your error:
# Import the project module. We add the parent directory to PYTHONPATH to
# avoid some of the path errors new users can have.
sys.path.append(os.path.join(project_directory, os.pardir))
import_module(project_name)
sys.path.pop()

Django running wrong version

Although running "python" from the shell runs Python v2.7, Django is loading files for python2.4, as shown in the error when I load a django site:
Mod_python error: "PythonHandler django.core.handlers.modpython"
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch
log=debug)
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 461, in import_module
f, p, d = imp.find_module(parts[i], path)
ImportError: No module named django
I think Django is installed for version 2.7 and that's why the bottom says "No module named django"
This is my first django install (it's on a mediatemple DV server) so I wouldn't be surprised if I'm doing something stupid. Thanks!

mod_python is built for 2.4, but Django is installed for 2.7. Either build mod_python for 2.7, install Django under 2.4, or put a local copy of Django with your project so that the version of Python doesn't matter.