Develop Reference

Python Module Import Error on Apache2 Server

When running a Python CGI script on Apache2, the server is unable to import certain python modules. For example, Pandas is installed locally:
And this is the script I want to run from the /var/www/html directory:
But the python script keeps giving an import error for pandas when running from the web. I have changed the shebang line a few times thinking that it could be an issue. I've tried /usr/bin/python3, /usr/bin/env python3, /usr/bin/python3.5 and /usr/bin/env python3.5 but it doesn't seem to make a difference.
I'm wondering if it's to do with permissions for the site-packages directory seeing as it's outside the web directory. Or if a completely different python environment is used when the CGI script is executed from the web. This is the site configuration:
Are there any additional authorizations that would allow for local modules to be called or is there another solution for this?

Custom modules are not found when calling python script in console VS pycharm

I have a python3 script that I am calling in terminal; I do not use Python prefix to run it, since I did add #!/usr/local/bin/python3 in my script (I have python3 from brew, on OSX).
The interesting thing is that if I run the script in terminal, I get an import error because one of my custom module hasn't been found. If I run the same script in pycharm, it works fine.
I assume Python launch and read all the various path that I use for modules in the same way, in both pycharm and terminal, but it seems not the case. How do I set up my scripts so the modules are found, independently from their path?
I may run the same script from other machines too, so I want to be prepared and do the right thing from the start.
EDIT
I am running pycharm on OSX; Python3 is installed via Brew, but the symlink is in /usr/local/bin.
My script is running from a folder inside my home directory, so
/Users/tester/git/python_test_app/main/base/app_main.py
The custom modules are in the same folder of the main py script, but one level above: /Users/tester/git/python_test_app/main/pyutils.py
The import statement from app_main.py is
import main.pyutils as utilities
This is the stack trace that I get when running the script:
Traceback (most recent call last):
File "main/base/app_main.py", line 13, in <module>
import main.pyutils as utilities
ModuleNotFoundError: No module named 'main'
EDIT 2 and solution
Thanks to The answers, I was able to figure out that the issue is related to how Pycharm handle projects. Basically it somehow save the path where the project is; so calling an import will result in the project folder being parsed, and that's why it works fine from Pycharm.
In Python, unless PYTHONPATH has the path to my project or other modules that I wrote, it won't be able to find them, hence, raise the error.
FIX:
in my main module that I use to run the application, I did retrieve the path of the file; which I know being one level below the modules I need; so I can explicitly add the folder to the current sys.path. This will end up making possible for me to call the import successfully.
import sys
current_dir = os.path.dirname(__file__)
sys.path.insert(0, , current_dir)
The only downside is that every file and resource that I use in my project, has to be directly referred by full path; so I have to pass the current_dir around the various files in the project.

PyCharm has project interpreter settings. Verify these are the same as your system Python. Go to:
File menu
Settings
Project: <project name>
Project Interpreter
View the path to the Python executable/binary being used by the project in PyCharm and verify it matches what your system is calling (e.g., which python3)
Alternatively, it may be that you declared your sources root within PyCharm and the system cannot properly run the module as it exists in the path you're running it from (especially if inside a package). You can get around this using the -m parameter and calling it from Python.
You can also try running it from the Terminal inside PyCharm and see what it adds to the path before initializing the shell session (you can sometimes see this in your Run configurations also). If you are referring to modules not installed via pip / into the Python path but rather loaded into your project path, then this may be the culprit.

On PyCharm, next to the green "RUN" arrow press the box and then press edit configurations (see image)
There you'll have Working Directory - that path is where PyCharm is running that script from (without errors).
Try running it from the terminal within that path - that should solve your import errors.

Cannot Import MySQLdb in PyCharm, runs fine in terminal.

I have problems trying to run a script which imports MySQLdb within PyCharm.
Running the script from terminal works just fine while running within PyCharm fails with
ImportError: No module named MySQLdb
I have tried this thread and it helped making things work in the terminal.
Trying to set the environment variables in the IDE though does not seem to work.
In PyCharm Run Config I set the environment variables
DYLD_LIBRARY_PATH - /usr/local/mysql/lib/libmysqlclient.18.dylib
PATH - /usr/local/mysql/lib/
but I still get the ImportError.

As #Dilettant pointed me to https://stackoverflow.com/a/34992894/1989141,
I realised there are two different places I was supposed to set the Python interpreter in PyCharm.
The first is in the main preferences (as in phil’s solution in the link) and allows me to specify a path to a local folder for mysqldb module.
The second is in the edit configuration settings for the script I want to run. I noticed the interpreters were different.
The script was set to run with the Python 2.7.6 /System/Library/Frameworks/...... version (which I believe is the OSx pre-installed version).
In the terminal and in PyCharm general settings I am using Python 2.7.9 in /usr/local/bin/python.
Matching the script interpreter to the one in the main settings (for which I manyally added mysqldb folder) solves the ImportError. Also I removed the environment variables I set up in the Edit Configuration as they are not needed.
Hope this helps. Thanks to both #Dilettant and #julivico for their suggestions.

Using custom module with Python CGI script

I'm running a python CGI script on my localhost that needs to import and use another python module that I wrote. I placed the CGI script in the Apache cgi-bin directory (I'm running this on windows). I've tried placing my custom module in the same directory, but it doesn't seem to be able to import that module. I would prefer to not have the custom module be another CGI script that is called via exec().

You need to put your Python module somewhere that Python's import can see it. The easy ways to do that are:
Make a directory for the module, and add that directory to your PYTHONPATH environment variable.
Copy the module into your Python site-packages directory, which is under your Python installation directory.
In either case, you will need to make sure your module's name is not the same as the name of some other module that might be imported by Python in your CGI script.

Managing Python Path When Moving Code from Development Computer to Target

I have a python project with this directory structure and these files:
/home/project_root
|---__init__.py
|---setup
|---__init__.py
|---configs.py
|---test_code
|---__init__.py
|---tester.py
The tester script imports from setup/configs.py with the reference "setup.configs". It runs fine on my development machine.
This works on the development (Linux) computer. When I move this to another (Linux) computer, I set the PYTHONPATH with
PYTHONPATH = "/home/project_root"
But when I run tester.py, it can't find the configs module. And when I run the interactive Python interpreter, sys.path doesn't include the /home/project_root directory. But /home/project_root does appear when I echo $PYTHPATH.
What am I doing wrong here?
(I don't want to rely on the .bashrc file to set the PYTHONPATH for the target machine -- the code is for a Django application, and will eventually be run by www-data. And, I know that the apache configuration for Django includes a specification of the PYTHONPATH, but I don't want to use that here as I'm first trying to make sure the code passes its unit tests in the target machine environment.)
CURIOUSER AND CURIOUSER
This seems to be a userid and permissions problem.
- When launched by a command from an ordinary user, the interpreter can import modules as expected.
- When launched by sudo (I'm running Ubuntu here), the interpreter cannot import modules as expected.
- I've been calling the test script with sudo, as the files are owned by www-data (b/c they'll be called by the user running apache as part of the Django application).
- After changing the files' ownership to that of an ordinary user, the test script does run without import errors (albeit, into all sorts of userid related walls).
Sorry to waste your time. This question should be closed.

Stick this in the tester script right before the import setup.configs
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), os.path.pardir))
sys.path is a list of all the directories the python interpreter looks for when importing a python module.
This will add the parent directory which contains setup module to the beginning of that list which means that the local directory will be checked first. That is important if you have your module installed system wide. More info on that here: sys doc.
EDIT: You could also put a .pth file in /usr/local/lib/python2.X/site-packages/ A .pth file is simply a text file with a directory path on each line that the python interpreter will search in. So just add a file with this line in it:
/home/project_root

Try explicitly setting your python path in your scripts. If you don't want to have to change it, you could always add something like "../" to the path in tester. That is to say:
sys.path.append("../")

(I don't want to rely on the .bashrc file to set the PYTHONPATH for the target machine -- the code is for a Django application, and will eventually be run by www-data. And, I know that the apache configuration for Django includes a specification of the PYTHONPATH, but I don't want to use that here as I'm first trying to make sure the code passes its unit tests in the target machine environment.)
If the code is for a Django application, is there a reason you're not testing it in the context of a Django project? Testing it in the context of a Django project gives a couple benefits:
Django's manage.py will set up your Python environment for you. It'll add the appropriate project paths to sys.path, and it'll set the environment variable DJANGO_SETTINGS_MODULE correctly.
Django's libraries include ample unit testing facilities, and you can easily extend that functionality to include your own testing facilities. Executing tests in a Django project is as easy as executing a single command via manage.py.