How to run a script within the module when the module is not in my Python path?
Consider some imaginary package:
package/
__init__.py
main.py
helper/
__init__.py
script.py
other/
__init__.py
anotherscript.py
Say we want to run script.py. When the package is in my Python path, this does the job:
python -m package.helper.script
But what if it's not the case? Is there a way to tell python the location of the module? Something like
python -m /path_to_my_package/package.helper.script
(clearly, the above doesn't work)
EDIT:
(1) I am looking for a solution that doesn't involve environmental variables.
(2) script.py contains relative import, so the full path to script.py does not solve the problem.
You could do this. I assume this is from a cmd prompt or batch file?
SET PYTHONPATH=..\..\path\to\my\package;c:\other\path\thats\needed
python -m package.helper.script
You could also just pass the full path to your python file. But that assumes your script is not expecting a particular environment path to be pre-set for it.
python -m path_to_my_package/package/helper/script.py
EDIT - If your script.py uses relative imports (and you don't want to change that), then there is no way to do it except getting that root path into the environment. You can do this in your script if you want, instead of setting it in the cmd shell or batch file. But it needs to get done somewhere. Here's how you can set the environment path in your script:
import sys
sys.path.append(r'..\..\path\to\my\package')
import package.other.anotherscript
Your script.py should look like:
#!/usr/bin/python
#here your python code
if __name__ == "__main__":
#run you helper
Then make your script executable: chmod +x script.py.
Run ./path_to_my_package/package/helper/script.py from console.
Related
main.py
import other
def main():
other.test()
main()
other.py
def test():
print("Hello")
By using python3 -m py_compile *.py, I can have 2 .pyc files.
However, main.pyc cannot be run if there is no module named other, which is the error I got from the terminal.
The idea is to compile the entire project from .py to .pyc so that people can run them without sharing the source code.
So, how to run this main.pyc which imports other libraries, while not sharing the source code?
Asked a machine learning group. Here is what I found.
As long as, main.py and other.py are compiled to main.pyc and other.pyc, I can run it by python3 main.pyc.
Before this, my python automatically converts other.py to other.cpython-35.pyc. In this case, main.pyc cannot import other since there is no other in the folder (it's called other.cpython-35 now).
Thus, make sure .pyc file have the same name as .py, and then you can run any of them and python will include .pyc file for you when you execute the command.
This can also be achieved by this command:
python -m compileall -b .
I have a Python script, say myscript.py, that uses relative module imports, ie from .. import module1, where my project layout is as follows:
project
+ outer_module
- __init__.py
- module1.py
+ inner_module
- __init__.py
- myscript.py
- myscript.sh
And I have a Bash script, say myscript.sh, which is a wrapper for my python script, shown below:
#!/bin/bash
python -m outer_module.inner_module.myscript $#
This works to execute myscript.py and forwards the arguments to my script as desired, but it only works when I call ./outer_module/inner_module/myscript.sh from within the project directory shown above.
How can I make this script work from anywhere? For example, how can I make this work for a call like bash /root/to/my/project/outer_module/inner_module/myscript.sh?
Here are my attempts:
When executing myscript.sh from anywhere else, I get the error: No module named outer_module.inner_module. Then I tried another approach to execute the bash script from anywhere, by replacing myscript.sh with:
#!/bin/bash
scriptdir=`dirname "$BASH_SOURCE"`
python $scriptdir/myscript.py $#
When I execute myscript.sh as shown above, I get the following:
Traceback (most recent call last):
File "./inner_module/myscript.py", line 10, in <module>
from .. import module1
ValueError: Attempted relative import in non-package
Which is due to the relative import on the first line in myscript.py, as mentioned earlier, which is from .. import module1.
As the error message says:
ValueError: Attempted relative import in non-package
The solution to this is to create a package, and have your script execute with that package in its path.
You already have a package, as you have the __init__.py files in those directories; but you only have that package in your path when you are calling it from the project directory, as you mentioned; that's because . is in your Python path by default.
To fix this, just add the project directory to your Python path, and then invoke it with python -m outer_module.inner_module.myscript:
#!/bin/bash
export PYTHONPATH=$PYTHONPATH:$(dirname "$BASH_SOURCE")/../..
python -m outer_module.inner_module.myscript $#
You need to include the path to the outer module's parent directory in the PYTHONPATH environmental variable, then you can use the same command you used in the first script from anywhere.
The PYTHONPATH is where python searches for any modules you try to import:
#!/bin/bash
export PYTHONPATH=$PYTHONPATH:PATH/TO/MODULE/PARENTDIR
python -m outer_module.inner_module.myscript $#
Is it possible to run a python script from one location but have the script execute as if it were located somewhere else (so imports etc all work as expected).
My scenario is this:
Application is located under /opt/foo/bin, application main entry point is foo.py
Under /opt/foo there are a bunch of packages/modules which foo.py then imports
We run application as cd /opt/foo/bin && python foo.py
I'd like to be able to copy just foo.py to /tmp and run it with cd /tmp && python foo.py. I'd like it to execute as if it were the foo.py living under /opt/foo/bin. I can modify the foo.py under /tmp if required.
Hopefully this makes sense to someone. Modifying the foo.py under /opt/foo/bin is not an option. One possibility might be creating a folder under /tmp i.e. /tmp/foo and symlinking everything except foo.py - but I'm hoping there's an easier way to accomplish what I want!
Here's a description of how Python searches for modules when you import them. One solution to your problem is to add the directory /opt/foo/bin to your PYTHONPATH. You could also modify your path within foo.py in order to find the module, e.g.
import sys
sys.path.append('/opt/foo/bin')
import foo
What is the correct way to fix this ImportError error?
I have the following directory structure:
/home/bodacydo
/home/bodacydo/work
/home/bodacydo/work/project
/home/bodacydo/work/project/programs
/home/bodacydo/work/project/foo
And I am in the directory
/home/bodacydo/work/project
Now if I type
python ./programs/my_python_program.py
I instantly get
ImportError: No module named foo.tasks
The ./programs/my_python_program.py contains the following line:
from foo.tasks import my_function
I can't understand why python won't find ./foo/tasks.py - it's there.
If I do it from the Python shell, then it works:
python
>>> from foo.tasks import my_function
It only doesn't work if I call it via python ./programs/my_python_program.py script.
Python does not add the current directory to sys.path, but rather the directory that the script is in. Add /home/bodacydo/work/project to either sys.path or $PYTHONPATH.
Do you have a file called __init__.py in the foo directory? If not then python won't recognise foo as a python package.
See the section on packages in the python tutorial for more information.
A better fix than setting PYTHONPATH is to use python -m module.path
This will correctly set sys.path[0] and is a more reliable way to execute modules.
I have a quick writeup about this problem, as other answerers have mentioned the reason for this is python path/to/file.py puts path/to on the beginning of the PYTHONPATH (sys.path).
Here is a step-by-step solution:
Add a script called run.py in /home/bodacydo/work/project and edit it like this:
import programs.my_python_program
programs.my_python_program.main()
(replace main() with your equivalent method in my_python_program.)
Go to /home/bodacydo/work/project
Run run.py
Explanation:
Since python appends to PYTHONPATH the path of the script from which it runs, running run.py will append /home/bodacydo/work/project. And voilà, import foo.tasks will be found.
Example solution for adding the library to your PYTHONPATH.
Add the following line into your ~/.bashrc or just run it directly:
export PYTHONPATH="$PYTHONPATH:$HOME/.python"
Then link your required library into your ~/.python folder, e.g.
ln -s /home/user/work/project/foo ~/.python/
In my mind I have to consider that the foo folder is a stand-alone library. I might want to consider moving it to the Lib\site-packages folder within a python installation. I might want to consider adding a foo.pth file there.
I know it's a library since the ./programs/my_python_program.py contains the following line:
from foo.tasks import my_function
So it doesn't matter that ./programs is a sibling folder to ./foo. It's the fact that my_python_program.py is run as a script like this:
python ./programs/my_python_program.py
If you have this problem when using an instaled version, when using setup.py, make sure your module is included inside packages
setup(name='Your program',
version='0.7.0',
description='Your desccription',
packages=['foo', 'foo.bar'], # add `foo.bar` here
I keep seeing sites mentioning that the directory that you execute 'python ' get added to the python path. For example on http://www.stereoplex.com/blog/understanding-imports-and-pythonpath, the author cd's to the /tmp folder then does 'print(sys.path)' and lo and behold, the /tmp folder appears in the path list. Here is me trying this out on my system (with 2.6.6 installed):
example structure:
app/
mymodule.py
inner_folder/
myscript.py
in myscript.py contains the line:
import mymodule.py
what I did:
cd app
python inner_folder/myscript.py # ImportError
Since I am executing the interpreter from the app/ directory, shouldn't 'app' be added to the python path? This is how a lot of the docs I have been reading have specified the behaviour should be.
Please enlighten!
(I have temporarily solved this by manually adding the folder I want into the environment but don't want to rely on that forever. Since many sites say this can be done, I'd like to reproduce it for myself)
It is the script's directory that is added, not the current directory. If you turn inner_folder/ into a package then you can use python -m inner_folder.myscript in order to run the script while having app/ added to sys.path.
Whether or not the current directory is in sys.path, import statements usually look like:
import mymodule
The code you wrote looks like:
import 'mymodule.py'