Is there a way to put together Python files, akin to JAR in Java? I need a way of packaging set of Python classes and functions, but unlike a standard module, I'd like it to be in one file.
After looking for a solution to the same problem, I ended up writing a simple tool which combines multiple .py files into one: PyBreeder
It will only work with pure-Python modules and may require some trial-and-error to get the order of modules right, but it is quite handy whenever you want to deploy a script with some dependencies as a single .py. Comments/patches/critique are very welcome!
Take a look at Python Eggs: http://peak.telecommunity.com/DevCenter/PythonEggs
Or, you can use regular zips: http://docs.python.org/library/zipimport.html
The simplest approach is to just use zip. A jar file in Java is a zipfile containing some metadata such as a manifest; but you don't necessarily need the metatada -- Python can import from inside a zipfile as long as you place that zipfile on sys.path, just as you would do for any directory. In the zipfile you can have the sources (.py files), but then Python will have to compile them on the fly each time a process first imports them; or you can have the bytecode files (.pyc or .pyo) but then you're limited to a specific release of Python and to either absence (for .pyc) or presence (for .pyo) of flag -O (or -OO).
As other answers indicated, there are formats such as .egg that enrich the zipfile with metatada in Python as well, like Java .jar, but whether in a particular use case that gives you extra value wrt a plain zipfile is a decision for you to make
You can create zip files containing Python code and import from zip files using zipimport. A system such as PyInstaller (cross-platform) or py2exe (Windows) will do all this for you.
Read this PEP for informations.
Also, Import modules from Zip.
stickytape (mentioned before) worked for me.
stickytape importingSomeModules.py > resultWithEmbeddedImports.py
Sometimes you just want to redistribute just a single .py file, so neither PyInstaller, zipimport nor Python Eggs cut the deal then.
PyBreeder is (currently) throwing exceptions.
Related
I've tried looking online and I'm honestly lost at this point.
I'm trying to find if there's a way to import python scripts and run them AFTER my Python program has been compiled.
For an example, let's say I have a main.py such that:
import modules.NewModule
a = NewModuleClass()
a.showYouWork()
then I compile main.py with pyinstaller so that my directory looks like:
main.exe
modules/NewModule.py
My end goal is to make a program that can dynamically read new Python files in a folder (this will be coded in) and use them (the part I'm struggling with). I know it's possible, since that's how add-ons work in Blender 3D but I've struggled for many hours to figure this out. I think I'm just bad at choosing the correct terms in Google.
Maybe I just need to convert all of the Python files in the modules directory to .pyc files? Then, how would I use them?
Also, if this is a duplicate on here (it probably is), please let me know. I couldn't find this issue on this site either.
You may find no detailed answer simply because there is no problem. PyInstaller does not really compile Python scripts into machine code executables. It just assembles then into a folder along with an embedded Python interpretor, or alternatively creates a compressed single file executable that will automatically uncompress itself at run time into a temporary folder containing that.
From then on, you have an almost standard Python environment, with normal .pyc file which can contain normal Python instructions like calls to importlib to dynamically load other Python modules. You have just to append the directory containing the modules to sys.path before importing them. An other possible caveat, is that pyinstaller only gets required modules and not a full Python installation, so you must either make sure that the dynamic modules do not rely on missing standard modules, or be prepared to face an ImportError.
Why is the import of *.so files from ZIP files disallowed in Python?
The documentation (https://docs.python.org/2/library/zipimport.html) is very clear:
Any files may be present in the ZIP archive, but only files .py and .py[co] are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed.
But the documentation doesn't name any reason for this strange limitation. Is is because importing from ZIP files is generally discouraged in Python? Or is it because of security reasons? If so, which ones? Is the any official statement about this?
From PEP 273, Subdirectory Equivalence:
You can't satisfy dynamic modules from a zip file. Dynamic modules
have extensions like .dll, .pyd, and .so. They are operating system
dependent, and probably can't be loaded except from a file. It might
be possible to extract the dynamic module from the zip file, write it
to a plain file and load it. But that would mean creating temporary
files, and dealing with all the dynload_*.c, and that's probably not a
good idea.
My interpretation is that this decision was made to avoid having the interpreter extract the dynamic module and save it to disk, as the OS's dynamic library facilities don't allow for loading from within a zip (See Windows' LoadLibraryA and Linux's dlopen).
While it looks like it's not technically impossible, a reason why doing the work to implement this functionality may not be worthwhile is the platform-dependence of these libraries. A .zip containing a dynamic module that is distributed with the code that relies on it may not work in a 32-bit environment if it was created from a 64-bit environment, and wouldn't work in Linux if it was created in Windows. Disallowing dynamic modules in .zips may be a conscious decision to ensure that .zips containing Python modules will work cross-platform.
Is there a way to change the directory where .pyc file are created by the Python interpreter? I saw two PEPs about that subject (0304 and 3147), but none seems to be implemented in the default interpreter (I'm working with Python 3).
Did I miss something ?
Yes, starting from Python 3.8 you can control this behavior. The original discussion starts from pep 304 in 2003.
While this original PEP was withdrawn, a variant of this feature was
eventually implemented for Python 3.8 in https://bugs.python.org/issue33499
In the result, you can control using PYTHONPYCACHEPREFIX=path, -X pycache_prefix=path and sys.pycache_prefix.
This might be useful for some:
Miscellaneous options, especially -B option:
If given, Python won’t try to write .pyc files on the import of source
modules. See also PYTHONDONTWRITEBYTECODE.
There's no way to change where the .pyc files go. Python 3.2 implements the __pycache__ scheme whereby all the .pyc files go into a directory named __pycache__. Python 3.2 alpha 1 is available now if you really need to keep your directories clean.
Until 3.2 is released, configure as many tools as you can to ignore the .pyc files.
Python files are NOT always stored in the same directory and will get directed to pycache if you use pycache. Any answers on being able to have custom locations and names, because importing the module is rather temperamental and doesn't always work. You will be able see what I mean at github.com/LolexInc/Lolex-Tools/tree/Beta and view JTToolsInstaller.py. There is a lot of mess in it.
Currently, I'm deploying a full python distribution (the original python 2.7 msi) with my app. Which is an embedded web server made with delphi.
Reading this, I wonder if is possible to embed the necessary python files with my app, to decrease load files and avoid conflict with several python versions.
I have previous experience with python for delphi so I only need to know if only shipping the python dll + zip with the distro + own scripts will work (and if exist any caveats I must know or a sample where I can look)
zipimport should work just fine for you -- I'm not familiar with Python for Delphi, but I doubt it disables that functionality (an embedding application can do that, but it's an unusual choice). Just remember that what you can zip up and import directly are the Python-coded modules (or just their corresponding .pyc or .pyo byte codes) -- DLLs (even if renamed as .pyds;-) need to be on disk to be loaded (so if you have a zipfile with them it will need to be unzipped at the start of the app, e.g. into a temporary directory).
Moreover, you don't even need to zip up all modules, just those you actually need (by transitive closure) -- and you can easily find out exactly which modules those are, with the modulefinder module of the standard Python library. The example on the documentation page I just pointed to should clarify things. Happy zipping!
Yes it is possible.
I'm actually writing automatisation script in Python with the Zipimport library. I actually included every .py files in my zip as well as configuration or xml files needed by those script.
Then, I call a .command file targeting a __main__.py class that redirect towards the desired script according to my sys.argv parameters which is really useful!
The hindrance we have to ship python is the large size of the standard library.
Is there a minimal python distribution or an easy way to pick and choose what we want from
the standard library?
The platform is linux.
If all you want is to get the minimum subset you need (rather than build an exe which would constrain you to Windows systems), use the standard library module modulefinder to list all modules your program requires (you'll get all dependencies, direct and indirect). Then you can zip all the relevant .pyo or .pyc files (depending on whether you run Python with or without the -O flag) and just use that zipfile as your sys.path (plus a directory for all the .pyd or .so native-code dynamic libraries you may need -- those need to live directly in the filesystem to let the OS load them in as needed, can't be loaded directly from a zipfile the way Python bytecode modules can, unfortunately).
Have you looked at py2exe? It provides a way to ship Python programs without requiring a Python installation.
Like Hank Gay and Alex Martelli suggest, you can use py2exe. In addition I would suggest looking into using something like IronPython. Depending on your application, you can use libraries that are built into the .NET framework (or MONO if for Linux). This reduces your shipping size, but adds minimum requirements to your program.
Further, if you are using functions from a library, you can use from module import x instead of doing a wildcard import. This reduces your ship size as well, but maybe not by too much
Hope this helps