I have a python script that uses the __file__ variable to take some action. This was working fine before I used zipapp since __file__ matched the actual running file.
Now I am starting to use zipapp and the logic doesn't work anymore, because __file__ is "loinc.pyz" and not "loinc.py".
Is there a way that, within my Python code, I can tell if the file is actually loinc.pyz, say, rather than loinc.py?
The only way I can see to do it now is to just try to see if __file__ + "z" exists, and if it does, assume we're using zipapp. But I'd like something more elegant.
I looked at the specifications from zipapp https://docs.python.org/3/library/zipapp.html but couldn't find anything. Looked at the 8 pages of zipapp-referenced questions in Stack Overflow and nothing either.
Use sys.argv[0].
(Note: this example was created on a UNIX system. Command invocations on a ms-windows system will differ.)
Create a file __main__.py:
import sys
print(f"__file__ = {__file__}")
print(f"sys.argv[0] = {sys.argv[0]}")
(Zipped Python executables depend on the name __main__.py being present in the zipfile.)
Next, create a file named hdr:
#!/usr/bin/env python
Compress __main__.py:
zip -q foo __main__.py
This will create foo.zip.
Concatenate the header and the zipfile, and make the resulting file executable:
cat hdr foo.zip >foo.pyz
chmod u+x foo.pyz
Now, call __main__.py:
> python __main__.py
__file__ = /zstorage/home/rsmith/tmp/src/__main__.py
sys.argv[0] = __main__.py
Then call foo.pyz:
> ./foo.pyz
__file__ = /zstorage/home/rsmith/tmp/src/./foo.pyz/__main__.py
sys.argv[0] = ./foo.pyz
Note how __file__ ends with __main__.py in both cases!
Related
I want to know abspath of python script followed by steps below.
built it to byte code by py_compile.
execute it to check abspath.
But I got 2 results when I execute it.I found the results based on the path of script followed by py_compile.
Here is my script test.py :
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
Build it with py_compile, then got 2 results when I enter different path of test.py:
1.enter the folder and compile with only script name.Then chdir to execute
[~]cd /usr/local/bin/
[/usr/local/bin/]python -m py_compile test.py
[/usr/local/bin/]cd ~
[~]python /usr/local/bin/test.pyc
/home/UserXX
2.In other folder and compile with absolute script name.
[~]python -m py_compile /usr/local/bin/test.py
[~]python /usr/local/bin/test.pyc
/usr/local/bin
how come got 2 different results?
When we want to get the path of one python file, typically we can use any of following methods:
1.
a)
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))).replace('\\', '/')
b)
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.stack()[0][1])).replace('\\', '/')
2.
a)
import os
import sys
print os.path.dirname(os.path.abspath(__file__)).replace('\\', '/')
b)
import os
import sys
print os.path.dirname(os.path.abspath(sys.argv[0])).replace('\\', '/')
For most of scenarios, we use 1 is enough, we seldom to use inspect like 2, as inspect maybe slower.
When will we use 2? Say use inspect to get file path?
One scenario I can remember is execfile, when fileA.py execfile fileB.py in its program, and fileA & fileB not in the same folder. (Maybe more scenario)
Then if we use __file__ in fileB.py, you will find its directory is just same as fileA.py, because here the directory will be the caller's directory.
Then we had to use inspect to get fileB.py's directory.
Anyway, for your situation, if your test.py just at the top of callgraph, just suggest you to use __file__, it's quicker, no need to use inspect.
With this if you use -m py_compile, it works for you.
Finally, why inspect not work with -m py_compile?
Unfortunately, I did not find official document to explain this.
But suppose your test.py is in the folder tt, then let's do cd ..; python -m py_compile tt/test.py, you will get a test.pyc in tt folder.
Let open this pyc file, although you will see something not suitable for man to read, you still can find some clue:
One line is something like:
currentframe(^#^#^#^#(^#^#^#^#(^#^#^#^#s^G^#^#^#tt/a.pyt
Do you see the folder name tt already in pyc file?
If you use inspect.stack() to test, it will more clear, print inspect.stack()[0][1] will always take your current compile folder in pyc file if you use -m py_compile.
This directly means during the process of py_compile, something was fixed to pyc file. This something I call fix makes you can just run your program in the same folder you do -m py_compile.
Hope this can give you some clue & helps you.
How to run a script within the module when the module is not in my Python path?
Consider some imaginary package:
package/
__init__.py
main.py
helper/
__init__.py
script.py
other/
__init__.py
anotherscript.py
Say we want to run script.py. When the package is in my Python path, this does the job:
python -m package.helper.script
But what if it's not the case? Is there a way to tell python the location of the module? Something like
python -m /path_to_my_package/package.helper.script
(clearly, the above doesn't work)
EDIT:
(1) I am looking for a solution that doesn't involve environmental variables.
(2) script.py contains relative import, so the full path to script.py does not solve the problem.
You could do this. I assume this is from a cmd prompt or batch file?
SET PYTHONPATH=..\..\path\to\my\package;c:\other\path\thats\needed
python -m package.helper.script
You could also just pass the full path to your python file. But that assumes your script is not expecting a particular environment path to be pre-set for it.
python -m path_to_my_package/package/helper/script.py
EDIT - If your script.py uses relative imports (and you don't want to change that), then there is no way to do it except getting that root path into the environment. You can do this in your script if you want, instead of setting it in the cmd shell or batch file. But it needs to get done somewhere. Here's how you can set the environment path in your script:
import sys
sys.path.append(r'..\..\path\to\my\package')
import package.other.anotherscript
Your script.py should look like:
#!/usr/bin/python
#here your python code
if __name__ == "__main__":
#run you helper
Then make your script executable: chmod +x script.py.
Run ./path_to_my_package/package/helper/script.py from console.
What is the correct way to fix this ImportError error?
I have the following directory structure:
/home/bodacydo
/home/bodacydo/work
/home/bodacydo/work/project
/home/bodacydo/work/project/programs
/home/bodacydo/work/project/foo
And I am in the directory
/home/bodacydo/work/project
Now if I type
python ./programs/my_python_program.py
I instantly get
ImportError: No module named foo.tasks
The ./programs/my_python_program.py contains the following line:
from foo.tasks import my_function
I can't understand why python won't find ./foo/tasks.py - it's there.
If I do it from the Python shell, then it works:
python
>>> from foo.tasks import my_function
It only doesn't work if I call it via python ./programs/my_python_program.py script.
Python does not add the current directory to sys.path, but rather the directory that the script is in. Add /home/bodacydo/work/project to either sys.path or $PYTHONPATH.
Do you have a file called __init__.py in the foo directory? If not then python won't recognise foo as a python package.
See the section on packages in the python tutorial for more information.
A better fix than setting PYTHONPATH is to use python -m module.path
This will correctly set sys.path[0] and is a more reliable way to execute modules.
I have a quick writeup about this problem, as other answerers have mentioned the reason for this is python path/to/file.py puts path/to on the beginning of the PYTHONPATH (sys.path).
Here is a step-by-step solution:
Add a script called run.py in /home/bodacydo/work/project and edit it like this:
import programs.my_python_program
programs.my_python_program.main()
(replace main() with your equivalent method in my_python_program.)
Go to /home/bodacydo/work/project
Run run.py
Explanation:
Since python appends to PYTHONPATH the path of the script from which it runs, running run.py will append /home/bodacydo/work/project. And voilà, import foo.tasks will be found.
Example solution for adding the library to your PYTHONPATH.
Add the following line into your ~/.bashrc or just run it directly:
export PYTHONPATH="$PYTHONPATH:$HOME/.python"
Then link your required library into your ~/.python folder, e.g.
ln -s /home/user/work/project/foo ~/.python/
In my mind I have to consider that the foo folder is a stand-alone library. I might want to consider moving it to the Lib\site-packages folder within a python installation. I might want to consider adding a foo.pth file there.
I know it's a library since the ./programs/my_python_program.py contains the following line:
from foo.tasks import my_function
So it doesn't matter that ./programs is a sibling folder to ./foo. It's the fact that my_python_program.py is run as a script like this:
python ./programs/my_python_program.py
If you have this problem when using an instaled version, when using setup.py, make sure your module is included inside packages
setup(name='Your program',
version='0.7.0',
description='Your desccription',
packages=['foo', 'foo.bar'], # add `foo.bar` here
For some reason, I can not depend on Python's "import" statement to generate .pyc file automatically
Is there a way to implement a function as following?
def py_to_pyc(py_filepath, pyc_filepath):
...
You can use compileall in the terminal. The following command will go recursively into sub directories and make pyc files for all the python files it finds. The compileall module is part of the python standard library, so you don't need to install anything extra to use it. This works exactly the same way for python2 and python3.
python -m compileall .
You can compile individual files(s) from the command line with:
python -m compileall <file_1>.py <file_n>.py
It's been a while since I last used Python, but I believe you can use py_compile:
import py_compile
py_compile.compile("file.py")
I found several ways to compile python scripts into bytecode
Using py_compile in terminal:
python -m py_compile File1.py File2.py File3.py ...
-m specifies the module(s) name to be compiled.
Or, for interactive compilation of files
python -m py_compile -
File1.py
File2.py
File3.py
.
.
.
Using py_compile.compile:
import py_compile
py_compile.compile('YourFileName.py')
Using py_compile.main():
It compiles several files at a time.
import py_compile
py_compile.main(['File1.py','File2.py','File3.py'])
The list can grow as long as you wish. Alternatively, you can obviously pass a list of files in main or even file names in command line args.
Or, if you pass ['-'] in main then it can compile files interactively.
Using compileall.compile_dir():
import compileall
compileall.compile_dir(direname)
It compiles every single Python file present in the supplied directory.
Using compileall.compile_file():
import compileall
compileall.compile_file('YourFileName.py')
Take a look at the links below:
https://docs.python.org/3/library/py_compile.html
https://docs.python.org/3/library/compileall.html
I would use compileall. It works nicely both from scripts and from the command line. It's a bit higher level module/tool than the already mentioned py_compile that it also uses internally.
Normally the following command compilies a python project:
python -m compileall <project-name>
In Python2 it compiles all .py files to .pyc files in a project which contains packages as well as modules.
Whereas in Python3 it compiles all .py files to __pycache__ folders in a project which contains packages as well as modules.
With browning from this post:
You can enforce the same layout of .pyc files in the folders as in
Python2 by using:
python3 -m compileall -b <pythonic-project-name>
The option -b triggers the output of .pyc files to their
legacy-locations (i.e. the same as in Python2).
To match the original question requirements (source path and destination path) the code should be like that:
import py_compile
py_compile.compile(py_filepath, pyc_filepath)
If the input code has errors then the py_compile.PyCompileError exception is raised.
create a new python file in the directory of the file.
type import (the name of the file without the extension)
run the file
open the directory, then find the pycache folder
inside should be your .pyc file
There are two ways to do this
Command line
Using python program
If you are using command line, use python -m compileall <argument> to compile python code to python binary code.
Ex: python -m compileall -x ./*
Or,
You can use this code to compile your library into byte-code:
import compileall
import os
lib_path = "your_lib_path"
build_path = "your-dest_path"
compileall.compile_dir(lib_path, force=True, legacy=True)
def moveToNewLocation(cu_path):
for file in os.listdir(cu_path):
if os.path.isdir(os.path.join(cu_path, file)):
compile(os.path.join(cu_path, file))
elif file.endswith(".pyc"):
dest = os.path.join(build_path, cu_path ,file)
os.makedirs(os.path.dirname(dest), exist_ok=True)
os.rename(os.path.join(cu_path, file), dest)
moveToNewLocation(lib_path)
look at ☞ docs.python.org for detailed documentation
I have a Python script that uses built-in modules but also imports a number of custom modules that exist in the same directory as the main script itself.
For example, I would call
python agent.py
and agent.py has a number of imports, including:
import checks
where checks is in a file in the same directory as agent.py
agent/agent.py
agent/checks.py
When the current working directory is agent/ then everything is fine. However, if I call agent.py from any other directory, it is obviously unable to import checks.py and so errors.
How can I ensure that the custom modules can be imported regardless of where the agent.py is called from e.g.
python /home/bob/scripts/agent/agent.py
Actually your example works because checks.py is in the same directory as agent.py, but say checks.py was in the preceeding directory, eg;
agent/agent.py
checks.py
Then you could do the following:
path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not path in sys.path:
sys.path.insert(1, path)
del path
Note the use of __file__.
You should NOT need to fiddle with sys.path. To quote from the Python 2.6 docs for sys.path:
As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to
invoke the Python interpreter. If the script directory is not
available (e.g. if the interpreter is invoked interactively or if the
script is read from standard input), path[0] is the empty string,
which directs Python to search modules in the current directory first.
Notice that the script directory is inserted before the entries
inserted as a result of PYTHONPATH.
=== amod.py ===
def whoami():
return __file__
=== ascript.py ===
import sys
print "sys.argv", sys.argv
print "sys.path", sys.path
import amod
print "amod __file__", amod.whoami()
=== result of running ascript.py from afar ===
C:\somewhere_else>\python26\python \junk\timport\ascript.py
sys.argv ['\\junk\\timport\\ascript.py']
sys.path ['C:\\junk\\timport', 'C:\\WINDOWS\\system32\\python26.zip', SNIP]
amod __file__ C:\junk\timport\amod.py
and if it's re-run, the last line changes of course to ...amod.pyc. This appears not to be a novelty, it works with Python 2.1 and 1.5.2.
Debug hints for you: Try two simple files like I have. Try running Python with -v and -vv. Show us the results of your failing tests, including full traceback and error message, and your two files. Tell us what platform you are running on, and what version of Python. Check the permissions on the checks.py file. Is there a checks.something_else that's causing interference?
You need to add the path to the currently executing module to the sys.path variable. Since you called it on the command line, the path to the script will always be in sys.argv[0].
import sys
import os
sys.path.append(os.path.split(sys.argv[0])[0])
Now when import searches for the module, it will also look in the folder that hosts the agent.py file.
There are several ways to add things to the PYTHONPATH.
Read http://docs.python.org/library/site.html
Set the PYTHONPATH environment variable prior to running your script.
You can do this python -m agent to run agent.py from your PYTHONPATH.
Create .pth files in your lib/site-packages directory.
Install your modules in lib/site-packages.
I think you should consider making the agent directory into a proper Python package. Then you place this package anywhere on the python path, and you can import checks as
from agent import checks
See http://docs.python.org/tutorial/modules.html
If you know full path to check.py use this recipe (http://code.activestate.com/recipes/159571/)
If you want to add directory to system path -- this recipe (http://code.activestate.com/recipes/52662/). In this case I have to determine application directory (sys.argv[0]) an pass this value to AddSysPath function. If you want to look at production sample please leave a comment on this thread so I post it later.
Regards.
To generalize my understanding of your goal, you want to be able to import custom packages using import custom_package_name no matter where you are calling python from and no matter where your python script is located.
A number of answers mention what I'm about to describe, but I feel like most of the answers assume a lot of previous knowledge. I'll try to be as explicit as I can.
To achieve the goal of allowing custom packages to be imported via the import statement, they have to be discoverable somewhere through the path that python uses to search for packages. Python actually uses multiple paths, but we'll only focus on the one that can be found by combining the output of sys.prefix (in your python interpreter) with /lib/pythonX.Y/site-packages (or lib/site-packages if you're using windows) where X.Y is your python version.
Concretely, find the path that your python uses by running:
import sys
your_path = sys.prefix + '/lib/pythonX.Y/site-packages'
print(your_path)
This path should look something like /usr/local/lib/python3.5/site-packages if you're using python 3.5, but it could be much different depending on your setup.
Python uses this path (and a few others) to find the packages that you want to import. So what you can do is put your custom packages in the /usr/local/lib/python3.5/site-packages folder. Don't forget to add an init.py file to the folder.
To be concrete again, in terminal type:
cd your_path
cp path_to_custom_package/custom_package ./
Now you should be able to import everything your custom package like you would if the package was located in the same directory (i.e. import package.subpackage for each subpackage file in your package should work).