I am trying to run my Django app on Google App Engine SDK (locally) inside a virtualenv with MySQL as a database. Everything in my requirements.txt file is installing perfectly. When I start the Google App Engine SDK environment, it throws me a nasty error that seems common, but Stack Overflow doesn't have any examples of solving this within Google App Engine SDK yet.
Here is my work flow from the root dir of my project...
virtualenv venv && mkdir lib
source venv/bin/activate
pip install -r requirements.txt -t lib/ && pip install -r requirements.txt
When I run the following command to start up the SDK....
dev_appserver.py app.yaml
I get the following error in my traceback...
File "/Users/username/Repositories/projectname/lib/django/db/utils.py", line 211, in __getitem__
backend = load_backend(db['ENGINE'])
File "/Users/username/Repositories/projectname/lib/django/db/utils.py", line 115, in load_backend
INFO 2018-06-26 20:09:30,812 module.py:846] default: "GET /_ah/start HTTP/1.1" 500 -
return import_module('%s.base' % backend_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/Users/username/Repositories/projectname/lib/django/db/backends/mysql/base.py", line 30, in <module>
'Did you install mysqlclient or MySQL-python?' % e
ImproperlyConfigured: Error loading MySQLdb module: No module named _mysql.
Did you install mysqlclient or MySQL-python?
My requirements.txt
Django==1.11.8
djangorestframework==3.8.2
facebook-sdk
oauth2client==2.0.1
google-api-python-client==1.6.2
facebookads==2.11.1
httplib2==0.10.3
enum==0.4.6
requests-toolbelt==0.8.0
google-cloud-storage==1.6.0
google-resumable-media==0.3.1
google-auth
requests==2.18.0
lxml==3.8.0
pycrypto==2.6.1
MySQL-python==1.2.5
Contents of lib/
I am calling MySQLdb in my app.yaml as well...
libraries:
- name: MySQLdb
version: "latest"
Contents of appengine_config.py
# [START vendor]
from google.appengine.ext import vendor
vendor.add('lib')
# [END vendor]
Some things I have checked off the list while debugging...
1) MySQL-python==1.2.5 is installed when I do a pip freeze in my virtual environment.
2) MySQL is installed and works perfectly on my local computer.
3) I've looked through the lionshare of Stack Overflow's questions thus far and none of them seem to help.
The installation was set up according to Google recommendations and it contained no visible errors. The problem could not be reproduced on external environment (a similar Django setup worked fine), so we just came to a workaround, see it below (set PYTHONPATH).
GAE requests installing third-party libraries into the lib/ directory, making it a poor man's virtual environment. Then lib/ is being added to a module search path by calling vendor.add('lib') in the appengine_config.py.
That worked correctly in the author's setup, so GAE managed to import Django from lib/django/. Then this working config failed misteriously for importing MySQLdb, which was surely installed into the same lib/.
During the investigation we have checked the idea if PYTHONPATH environment variable could affect anything. It did not, but setting it fixed the issue:
export PYTHONPATH=/Users/username/Repositories/projectname/lib/
Note: avoid using relative directories in PATH-like environment variables (./lib in Erik's comment), that creates a security flaw.
The MySQL-Python which is a C extension,will probably fail(No module named _mysql).
you can give a try with pymysql module
I'm trying to use the feed-parser module for this project im working on. When I upload the files to App Engine and I run the script it comes back with the error that the there is no module named feed-parser.
So I'm wondering if and how can I install this module on App Engine, so that I can fix this error and use RSS.
Error:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~vlis-mannipulus-bot/1.391465315184045822/main.py", line 7, in <module>
import feedparser
ImportError: No module named feed parser
Development 1:
So I've tried installing the module in the lib directory i created(in this fail example i forgot the /lib at the --prefix=..). And i get PYTHONERROR as is shown in the shell. Ive done some research on python paths and the solutions i tried didn't work for me.
kevins-MacBook-Pro-2:~ KevinH$ cd /Users/KevinH/Downloads/feedparser -5.2.1
kevins-MacBook-Pro-2:feedparser-5.2.1 KevinH$ sudo python setup.py install --prefix=/Users/KevinH/Documents/Thalia\ VMbot/Thalia-VMbot/
Password:
running install
Checking .pth file support in /Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: /Users/KevinH/Documents/Thalia VMbot/Thalia- VMbot//lib/python2.7/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site- packages/
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom- installation-locations
Please make the appropriate changes for your system and try again.
Then i tried with the "pip" command but then i get this:
can't open file 'pip': [Errno 2] No such file or directory
According to what I have read "pip" should be a default program installed with python 2.7 and up. So to be sure i did install python3.5 and ran it with that and still get the same error. I typed this with both pythons:
kevins-MacBook-Pro-2:feedparser KevinH$ python3 pip -m install feedparse
--
Not sure if this would work, but via terminal i went to the default directory where feed parser has been installed on my system and copied it to the lib directory i made. Then I've created the config file with the following:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
Deployed it and im still getting the same error as above no module named feeedparser.
Apologies if im doing something stupidly wrong, im still in the learning process.
The App Engine documentation that explains how to add third party modules is here
In summary, you will need to add a folder, usually named 'lib', to the top level off your app, and then install feedparser into that folder using the commands described in the documentation. You will also need to create an appengine_config.py file as descibed in the documentation.
Note that not all third party packages can be uploaded to App Engine - those with C extensions are forbidden. Feedparser looks OK to me though.
EDIT: further comments based on edit "development1" to the question.
Your appengine_config.py looks good.
You "lib" folder should be your application folder, that is the same folder as your app.yaml and appengine_config.py files.
You need to install the feedparser package into the lib folder. The Google docs recommend that you do this by running the command
pip install -t lib feedparser
This command will install the feedparser package into your lib folder.
You need to install and run a version of pip that works with Python2.x - the Python3 version will create a version of feedparser that only runs under Python3.
If you can't install a pip for Python2 this question might provide the right solution, otherwise I'd suggest you ask a separate question about how to install feedparser into a custom install directory on a Mac.I don't have a Mac so I can't help with this.
Once you have feedparser installed in your lib folder, verify that your app works locally; in particular verify that it's using your lib/feedparser installation: try logging feedparser.__file__ after importing feedparser to check the location of the file being imported.
Once everything is working locally you should be able to upload to GAE.
So in summary, your appengine_config.py looks good, but you need to make sure that the right version of feedparser is installed into the lib folder in your app folder before uploading to App Engine.
I'm using Django 1.4.1 with Active Python 2.7 on Win7. I have installed the MySQL module using pypm install mysql-python.
The database engine is django.db.backends.mysql.
import MySQLdb works in the interactive shell.
.\manage.py syncdb created the tables with no problem.
However, when I open the site in the browser, I get Error loading MySQLdb module: No module named MySQLdb:
Environment:
Request Method: GET
Request URL: http://whatever/
Django Version: 1.4.1
Python Version: 2.7.2
Installed Applications:
('django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')
Traceback:
File "C:\Python27\lib\site-packages\django\core\handlers\base.py" in get_response
89. response = middleware_method(request)
File "C:\Python27\lib\site-packages\django\contrib\sessions\middleware.py" in process_request
10. engine = import_module(settings.SESSION_ENGINE)
File "C:\Python27\lib\site-packages\django\utils\importlib.py" in import_module
35. __import__(name)
File "C:\Python27\lib\site-packages\django\contrib\sessions\backends\cached_db.py" in <module>
6. from django.contrib.sessions.backends.db import SessionStore as DBStore
File "C:\Python27\lib\site-packages\django\contrib\sessions\backends\db.py" in <module>
3. from django.db import IntegrityError, transaction, router
File "C:\Python27\lib\site-packages\django\db\__init__.py" in <module>
40. backend = load_backend(connection.settings_dict['ENGINE'])
File "C:\Python27\lib\site-packages\django\db\__init__.py" in __getattr__
34. return getattr(connections[DEFAULT_DB_ALIAS], item)
File "C:\Python27\lib\site-packages\django\db\utils.py" in __getitem__
92. backend = load_backend(db['ENGINE'])
File "C:\Python27\lib\site-packages\django\db\utils.py" in load_backend
24. return import_module('.base', backend_name)
File "C:\Python27\lib\site-packages\django\utils\importlib.py" in import_module
35. __import__(name)
File "C:\Python27\lib\site-packages\django\db\backends\mysql\base.py" in <module>
16. raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
Exception Type: ImproperlyConfigured at /
Exception Value: Error loading MySQLdb module: No module named MySQLdb
The settings for the sessions and messages apps are:
SESSION_ENGINE = "django.contrib.sessions.backends.cached_db"
MESSAGE_STORAGE = "django.contrib.messages.storage.cookie.CookieStorage"
How is this possible?
The problem was that MySQLdb was installed in my home directory C:\Users\alexei\AppData\Roaming\Python\Python27\site-packages\ which was not in Python's path. So I uninstalled it with pypm uninstall mysql-python and then reinstalled it globally using pypm -g install mysql-python (note the -g option).
The alternative is to add that path to the list sys.path.append("...path...") in wsgi.py
So, in case someone else is wondering, you can find out where MySQLdb (or any other module) is installed like so:
import MySQLdb
print MySQLdb.__file__
Make sure that that path is in Python's path list provided in Django's error message.
If you are using and virtual environment. you must run
cd virtualEnvPath
pip install MySQL-Python
Don't forget positioning inside of your virtual env. /bin PATH, to use local pip.
REGARDS!
i also faced this issue then i installed python mysql to my system
pip install mysql-python
then its workes
see this -->
working in my system >> $ python manage.py syncdb;
Creating tables ...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
.............working
Easiest way is to install MySQLdb from binary:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#mysql-python
I configured djando w/ mysql the following way:
1) set DJANGO_SETTINGS_MODULE environment variable in .bashrc and source it
2) change settings.py pointed by this env_var and local_settings.py (if any) like this (for mysql):
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'DB_NAME',
'USER': 'DB_USER',
'PASSWORD': 'DB_PASSWORD',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '3306',
}
}
3) install mysql dev libraries and header files, ensure that mysql is on PATH
sudo apt-get install libmysqlclient-dev
4) install python package for mysql
sudo pip install MySQL-python
5) test django's access to your mysql db:
python manage.py dbshell
or
django-admin dbshell
6) sync django to mysql from dir with your django app
python manage.py syncdb
I have Pythong2.6, psycopg2 and pgAdmin3 installed using Macports. My settings.py is:
DATABASE_ENGINE = 'postgresql_psycopg2' # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = 'mysite' # Or path to database file if using sqlite3.
DATABASE_USER = 'postgres' # Not used with sqlite3.
DATABASE_PASSWORD = '' # Not used with sqlite3.
DATABASE_HOST = '' # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = '' # Set to empty string for default. Not used with sqlite3.
The error I get when I run python manage.py syncdb is:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/__init__.py", line 362, in execute_manager
utility.execute()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/__init__.py", line 303, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 195, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 221, in execute
self.validate()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/validation.py", line 22, in get_validation_errors
from django.db import models, connection
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 41, in <module>
backend = load_backend(settings.DATABASE_ENGINE)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 17, in load_backend
return import_module('.base', 'django.db.backends.%s' % backend_name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 22, in <module>
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named psycopg2
Please note, I am a complete beginner in this stuff. I am originally a PHP-guy and trying out Python for a small personal project. Do I need to "turn on" Postgres?
Also, when I sudo python manage.py runserver 8080
I get this error:
Validating models...
Unhandled exception in thread started by <function inner_run at 0x1242670>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/commands/runserver.py", line 48, in inner_run
self.validate(display_num_errors=True)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/validation.py", line 22, in get_validation_errors
from django.db import models, connection
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 41, in <module>
backend = load_backend(settings.DATABASE_ENGINE)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 17, in load_backend
return import_module('.base', 'django.db.backends.%s' % backend_name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 22, in <module>
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named psycopg2
Please guide me. Any reply will be appreciated.
Thanks,
Wenbert!
There seems to be a problem with your psycopg2 installation – Python does not find it. This is a Python installation problem, not a Django issue.
You can try to load it manually using the Python interpreter and see if it works:
$ python
>>> import psycopg2
If you get an ImportError exception, your installation is erroneous. To get a list of all directories Python looks for modules, use sys.path:
$ python
>>> import sys
>>> print sys.path
You can also add custom directories to Python's module search path by modifying the sys.path variable. Do this somewhere before the respective import statement(s):
import sys
sys.path.append("my-path")
# ...
import psycopg2
If you have pip installed, simply install the missing extension by running:
$ pip install psycopg2
For the record I got the same error for a different reason:
I had put
'ENGINE': 'django.db.backends.postgresql'
instead of
'ENGINE': 'django.db.backends.postgresql_psycopg2'
in settings.py
Follow the steps:
Install python libraries on your OS:
python-dev
libpq-dev
Execute the command to install the psycopg2 library:
easy_install psycopg2
source: http://initd.org/psycopg/install/
I realized I didn't have psycopg2 installed
aptitude install python-psycopg2
Worked like a charm
I had this issue recently after updating homebrew on OSX.
psycopg2 was already listed in my virtualenv.
I just reinstalled psycopg2 and it worked again:
pip install --force-reinstall psycopg2
Although you installed it, Python can apparently not find the module psycopg2. This is usually due to the module not being in Python's path. See if you can find a folder named psycopg2 in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages. If it's not there, did MacPorts tell you where it put psycopg2? If you can locate it, just move it to the site-packages directory and you should be fine.
For me, psycopg2 was indeed installed, but not into the virtualenv in which Django was running. These two steps fixed it:
sudo apt-get build-dep python-psycopg2
sudo /opt/myenv/bin/pip install psycopg2
Yes Tim's answer works for me too.It works without prefix 'django.db.backends.' also. But remember to create database or schema you mentioned in settings.py:
DATABASE_NAME = 'your_db_name'
manually using your database client so when you run 'python manage.py syncdb' you don't get the same problem. I was stuck because i didn't create it manually. I got the same problem may be because I used buildout.
I got the same error, but it was because I was using python26 ./manage.py runserver when my virtualenv only had python and python2.6 executables (thus the system python26 was being used, which didn't have psycopg2 installed
Tim's answer worked for me too.
By default, settings.py shows options like 'postgresql_psycopg2', 'mysql', etc, without a package name. Prefixing with 'django.db.backends.' worked for me (for postgresql_psycopg2, at least).
I'm on Windows and had installed psycopg2, but the 64 bit version. So my fix was to download the 32 bit one from here then in PowerShell with my virtual environment activated, my fix was:
pip uninstall psycopg2
easy_install C:\WHEREVER_I_DOWNLOADED_IT\psycopg2-2.6.1.win32-py3.4-pg9.4.4-release.exe
(The Windows Python 3.4 installer automatically installs easy_install and pip, and pip is the easiest way to remove the package, even if originally installed using easy_install.)
THIS HAS HELPED ME:
I just added PostgreSQL bin path to ENV and it was able to fine the needed dll:
C:\Program Files (x86)\PostgreSQL\9.4\bin
https://groups.google.com/forum/#!topic/robotframework-users/qLq9KvHe1wE
same here, but maybe it was missed in current project, so I navigated to the main project directory and installed it.
pip install psycopg2
and it worked
You can get this error in heroku deployment with django postgresql as backend simply add psycopg2 to your requirements.txt file
This error also occurs when you forgot to activate your virtual environment.
Using Django 3.2.9 with Python 3.9.1, after pip3 install psycopg2, this is what I had to do in order to get the initial python manage.py migrate to work:
In my Django site's settings.py file, I added these lines to the top of the file (after the from pathlib import Path line):
import sys
sys.path.append('Users/MY_USER_NAME/Sites/SITE_NAME/env/lib/python3.9/site-packages')
Sorry... a ~/Sites/... path setting doesn't work (I tried), it needs to be an absolute, full path.
My Database settings are as follows:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'DATABASE_NAME',
'USER': os.getenv('PG_USER'),
'PASSWORD': os.getenv('PG_PASS'),
'PORT': '5432',
'USE_TZ': True,
}
}
Once I added the two lines to the top of the settings.py file, the migration ran successfully :)
For the module not found errors, usually a good
pip install module_name
Will solve the problem. If it's not the case look for the binary of the module you are trying to install. Install the binary then install the module again. Example for this case the module is
psycopg2
Binary is psychopg2-binary so
Install them like this
pip install psycopg2-binary
pip install psycopg2