For some reason, I can not depend on Python's "import" statement to generate .pyc file automatically
Is there a way to implement a function as following?
def py_to_pyc(py_filepath, pyc_filepath):
...
You can use compileall in the terminal. The following command will go recursively into sub directories and make pyc files for all the python files it finds. The compileall module is part of the python standard library, so you don't need to install anything extra to use it. This works exactly the same way for python2 and python3.
python -m compileall .
You can compile individual files(s) from the command line with:
python -m compileall <file_1>.py <file_n>.py
It's been a while since I last used Python, but I believe you can use py_compile:
import py_compile
py_compile.compile("file.py")
I found several ways to compile python scripts into bytecode
Using py_compile in terminal:
python -m py_compile File1.py File2.py File3.py ...
-m specifies the module(s) name to be compiled.
Or, for interactive compilation of files
python -m py_compile -
File1.py
File2.py
File3.py
.
.
.
Using py_compile.compile:
import py_compile
py_compile.compile('YourFileName.py')
Using py_compile.main():
It compiles several files at a time.
import py_compile
py_compile.main(['File1.py','File2.py','File3.py'])
The list can grow as long as you wish. Alternatively, you can obviously pass a list of files in main or even file names in command line args.
Or, if you pass ['-'] in main then it can compile files interactively.
Using compileall.compile_dir():
import compileall
compileall.compile_dir(direname)
It compiles every single Python file present in the supplied directory.
Using compileall.compile_file():
import compileall
compileall.compile_file('YourFileName.py')
Take a look at the links below:
https://docs.python.org/3/library/py_compile.html
https://docs.python.org/3/library/compileall.html
I would use compileall. It works nicely both from scripts and from the command line. It's a bit higher level module/tool than the already mentioned py_compile that it also uses internally.
Normally the following command compilies a python project:
python -m compileall <project-name>
In Python2 it compiles all .py files to .pyc files in a project which contains packages as well as modules.
Whereas in Python3 it compiles all .py files to __pycache__ folders in a project which contains packages as well as modules.
With browning from this post:
You can enforce the same layout of .pyc files in the folders as in
Python2 by using:
python3 -m compileall -b <pythonic-project-name>
The option -b triggers the output of .pyc files to their
legacy-locations (i.e. the same as in Python2).
To match the original question requirements (source path and destination path) the code should be like that:
import py_compile
py_compile.compile(py_filepath, pyc_filepath)
If the input code has errors then the py_compile.PyCompileError exception is raised.
create a new python file in the directory of the file.
type import (the name of the file without the extension)
run the file
open the directory, then find the pycache folder
inside should be your .pyc file
There are two ways to do this
Command line
Using python program
If you are using command line, use python -m compileall <argument> to compile python code to python binary code.
Ex: python -m compileall -x ./*
Or,
You can use this code to compile your library into byte-code:
import compileall
import os
lib_path = "your_lib_path"
build_path = "your-dest_path"
compileall.compile_dir(lib_path, force=True, legacy=True)
def moveToNewLocation(cu_path):
for file in os.listdir(cu_path):
if os.path.isdir(os.path.join(cu_path, file)):
compile(os.path.join(cu_path, file))
elif file.endswith(".pyc"):
dest = os.path.join(build_path, cu_path ,file)
os.makedirs(os.path.dirname(dest), exist_ok=True)
os.rename(os.path.join(cu_path, file), dest)
moveToNewLocation(lib_path)
look at ☞ docs.python.org for detailed documentation
Related
I have a python script that uses the __file__ variable to take some action. This was working fine before I used zipapp since __file__ matched the actual running file.
Now I am starting to use zipapp and the logic doesn't work anymore, because __file__ is "loinc.pyz" and not "loinc.py".
Is there a way that, within my Python code, I can tell if the file is actually loinc.pyz, say, rather than loinc.py?
The only way I can see to do it now is to just try to see if __file__ + "z" exists, and if it does, assume we're using zipapp. But I'd like something more elegant.
I looked at the specifications from zipapp https://docs.python.org/3/library/zipapp.html but couldn't find anything. Looked at the 8 pages of zipapp-referenced questions in Stack Overflow and nothing either.
Use sys.argv[0].
(Note: this example was created on a UNIX system. Command invocations on a ms-windows system will differ.)
Create a file __main__.py:
import sys
print(f"__file__ = {__file__}")
print(f"sys.argv[0] = {sys.argv[0]}")
(Zipped Python executables depend on the name __main__.py being present in the zipfile.)
Next, create a file named hdr:
#!/usr/bin/env python
Compress __main__.py:
zip -q foo __main__.py
This will create foo.zip.
Concatenate the header and the zipfile, and make the resulting file executable:
cat hdr foo.zip >foo.pyz
chmod u+x foo.pyz
Now, call __main__.py:
> python __main__.py
__file__ = /zstorage/home/rsmith/tmp/src/__main__.py
sys.argv[0] = __main__.py
Then call foo.pyz:
> ./foo.pyz
__file__ = /zstorage/home/rsmith/tmp/src/./foo.pyz/__main__.py
sys.argv[0] = ./foo.pyz
Note how __file__ ends with __main__.py in both cases!
I want to know abspath of python script followed by steps below.
built it to byte code by py_compile.
execute it to check abspath.
But I got 2 results when I execute it.I found the results based on the path of script followed by py_compile.
Here is my script test.py :
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
Build it with py_compile, then got 2 results when I enter different path of test.py:
1.enter the folder and compile with only script name.Then chdir to execute
[~]cd /usr/local/bin/
[/usr/local/bin/]python -m py_compile test.py
[/usr/local/bin/]cd ~
[~]python /usr/local/bin/test.pyc
/home/UserXX
2.In other folder and compile with absolute script name.
[~]python -m py_compile /usr/local/bin/test.py
[~]python /usr/local/bin/test.pyc
/usr/local/bin
how come got 2 different results?
When we want to get the path of one python file, typically we can use any of following methods:
1.
a)
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))).replace('\\', '/')
b)
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.stack()[0][1])).replace('\\', '/')
2.
a)
import os
import sys
print os.path.dirname(os.path.abspath(__file__)).replace('\\', '/')
b)
import os
import sys
print os.path.dirname(os.path.abspath(sys.argv[0])).replace('\\', '/')
For most of scenarios, we use 1 is enough, we seldom to use inspect like 2, as inspect maybe slower.
When will we use 2? Say use inspect to get file path?
One scenario I can remember is execfile, when fileA.py execfile fileB.py in its program, and fileA & fileB not in the same folder. (Maybe more scenario)
Then if we use __file__ in fileB.py, you will find its directory is just same as fileA.py, because here the directory will be the caller's directory.
Then we had to use inspect to get fileB.py's directory.
Anyway, for your situation, if your test.py just at the top of callgraph, just suggest you to use __file__, it's quicker, no need to use inspect.
With this if you use -m py_compile, it works for you.
Finally, why inspect not work with -m py_compile?
Unfortunately, I did not find official document to explain this.
But suppose your test.py is in the folder tt, then let's do cd ..; python -m py_compile tt/test.py, you will get a test.pyc in tt folder.
Let open this pyc file, although you will see something not suitable for man to read, you still can find some clue:
One line is something like:
currentframe(^#^#^#^#(^#^#^#^#(^#^#^#^#s^G^#^#^#tt/a.pyt
Do you see the folder name tt already in pyc file?
If you use inspect.stack() to test, it will more clear, print inspect.stack()[0][1] will always take your current compile folder in pyc file if you use -m py_compile.
This directly means during the process of py_compile, something was fixed to pyc file. This something I call fix makes you can just run your program in the same folder you do -m py_compile.
Hope this can give you some clue & helps you.
main.py
import other
def main():
other.test()
main()
other.py
def test():
print("Hello")
By using python3 -m py_compile *.py, I can have 2 .pyc files.
However, main.pyc cannot be run if there is no module named other, which is the error I got from the terminal.
The idea is to compile the entire project from .py to .pyc so that people can run them without sharing the source code.
So, how to run this main.pyc which imports other libraries, while not sharing the source code?
Asked a machine learning group. Here is what I found.
As long as, main.py and other.py are compiled to main.pyc and other.pyc, I can run it by python3 main.pyc.
Before this, my python automatically converts other.py to other.cpython-35.pyc. In this case, main.pyc cannot import other since there is no other in the folder (it's called other.cpython-35 now).
Thus, make sure .pyc file have the same name as .py, and then you can run any of them and python will include .pyc file for you when you execute the command.
This can also be achieved by this command:
python -m compileall -b .
EDIT: Based on discussions below, I think my question really could apply to any language. Python naturally has a packaging system and installation procedure via pip. But let's say this was C code or a perl script. Users would still download the program files, and have a way to execute the code in the command line via capsall input.txt.
I have a python script that takes an input file and gives an output file.
For a concrete example, this script file1.py takes in an input text file and outputs a text file with all letters capitalized:
import sys
inFile = sys.argv[1]
outFile = sys.argv[2]
with open(inFile,'r') as input_file:
lines = input_file.readlines()
# process the input file somehow
# here we simply capitalize the text,
# but naturally something more complex is possible
capitalized_lines = []
for line in lines:
capitalized_lines.append(line.upper())
with open(outFile,'w') as output_file:
for line in capitalized_lines:
output_file.write(line)
The way users execute this code now with
python file1.py input.txt output.txt
Let's say I wanted to distribute this code such that users would download a tarball and be able to execute the above in the command line with (for example)
capsall input.txt
which would run python file1.py and output the file output.txt. Does one write a bash script? If so, how do you distribute the code such that users will have this in their PATH?
Add a "hash bang" at the top of the script file to tell bash to invoke the Python interpreter. Also make your script executable:
#!/usr/bin/env python
import sys
inFile = sys.argv[1]
outFile = sys.argv[2]
...
Make the script file executable:
$ cp file1.py capsall
$ chmod +x capsall
Now you can run the script with:
$ ./capsall input.txt output.txt
Or if capsall is on your path:
$ capsall input.txt output.txt
I would recommend using python packaging (e.g., [pip]) for this. You could use the OS specific packaging method as well, something like apt, yum, msiexec, whatever, but I wouldn't unless you have to. Your users already have python installed since they are used to explicitly passing your script to the interpreter already. If you are interested in using python packaging, then read on.
First, I would put your script into a package along with whatever other processing is necessary. Let's call it mylibrary. Your project should look something like:
myproject
|-- README.rst
`-- myproject
|-- __init__.py
`-- capsall.py
Keep __init__.py simple and make sure that you can import it without dependencies. I use something like:
version_info = (0, 0, 0)
version = '.'.join(str(v) for v in version_info)
Next, add a file named setup.py at the root. This is what will define your package, it's metadata, and the scripts to install into your user's $PATH. The following will work nicely:
#!/usr/bin/env python
import setuptools
import myproject
setuptools.setup(
name='myproject',
version=myproject.version,
description='Package of useful utilities.',
long_description=open('README.rst').read(),
url='https://github.com/me/myproject',
author='Me',
author_email='me#example.com',
packages=['myproject'],
entry_points={
'console_scripts': ['capsall=myproject.capsall:main'],
},
license='BSD',
)
If you plan on uploading to pypi.python.org, then you will need at least that much metadata. I would recommend adding classifiers as well. The Python Packaging Authority docs on writing a setup.py are invaluable.
The part that you are most interested in is the entry_points keyword parameter. It defines various ways that your package can be invoked. You are looking for console_scripts since it creates shell scripts that are installed into the local path. See the setuptools documentation for more details.
The console_scripts definition that I gave above creates a script named capsall that invokes the my project.capsall.main function. I would repackage your code into a project. Bundle your script into the main function of the capsall.py module. Then generate a source repository and upload it to pypi.python.org (./setup.py sdist register upload). Once you have it uploaded, your users can install it using pip install myproject.
Chances are that you have code that you don't want to give to the outside world. If that is the case, then you can generate a source distribution and make the tarball available on an internal server -- ./setup.py sdist will generate a tarball in the dist directory. Then pip install whatever/myproject-0.0.0.tar.gz will install your script.
How to run a script within the module when the module is not in my Python path?
Consider some imaginary package:
package/
__init__.py
main.py
helper/
__init__.py
script.py
other/
__init__.py
anotherscript.py
Say we want to run script.py. When the package is in my Python path, this does the job:
python -m package.helper.script
But what if it's not the case? Is there a way to tell python the location of the module? Something like
python -m /path_to_my_package/package.helper.script
(clearly, the above doesn't work)
EDIT:
(1) I am looking for a solution that doesn't involve environmental variables.
(2) script.py contains relative import, so the full path to script.py does not solve the problem.
You could do this. I assume this is from a cmd prompt or batch file?
SET PYTHONPATH=..\..\path\to\my\package;c:\other\path\thats\needed
python -m package.helper.script
You could also just pass the full path to your python file. But that assumes your script is not expecting a particular environment path to be pre-set for it.
python -m path_to_my_package/package/helper/script.py
EDIT - If your script.py uses relative imports (and you don't want to change that), then there is no way to do it except getting that root path into the environment. You can do this in your script if you want, instead of setting it in the cmd shell or batch file. But it needs to get done somewhere. Here's how you can set the environment path in your script:
import sys
sys.path.append(r'..\..\path\to\my\package')
import package.other.anotherscript
Your script.py should look like:
#!/usr/bin/python
#here your python code
if __name__ == "__main__":
#run you helper
Then make your script executable: chmod +x script.py.
Run ./path_to_my_package/package/helper/script.py from console.