I've tried looking online and I'm honestly lost at this point.
I'm trying to find if there's a way to import python scripts and run them AFTER my Python program has been compiled.
For an example, let's say I have a main.py such that:
import modules.NewModule
a = NewModuleClass()
a.showYouWork()
then I compile main.py with pyinstaller so that my directory looks like:
main.exe
modules/NewModule.py
My end goal is to make a program that can dynamically read new Python files in a folder (this will be coded in) and use them (the part I'm struggling with). I know it's possible, since that's how add-ons work in Blender 3D but I've struggled for many hours to figure this out. I think I'm just bad at choosing the correct terms in Google.
Maybe I just need to convert all of the Python files in the modules directory to .pyc files? Then, how would I use them?
Also, if this is a duplicate on here (it probably is), please let me know. I couldn't find this issue on this site either.
You may find no detailed answer simply because there is no problem. PyInstaller does not really compile Python scripts into machine code executables. It just assembles then into a folder along with an embedded Python interpretor, or alternatively creates a compressed single file executable that will automatically uncompress itself at run time into a temporary folder containing that.
From then on, you have an almost standard Python environment, with normal .pyc file which can contain normal Python instructions like calls to importlib to dynamically load other Python modules. You have just to append the directory containing the modules to sys.path before importing them. An other possible caveat, is that pyinstaller only gets required modules and not a full Python installation, so you must either make sure that the dynamic modules do not rely on missing standard modules, or be prepared to face an ImportError.
Related
I am working on a bunch of Python 3 scripts that are organized like so:
/workspace
/project_one
script_a.py
script_b.py
/project_two
script_c.py
There are many project folders and every once in a while, one is added, with varying numbers of .py files inside. I want to be able to import each script from each folder in each other script, but since I switched to VS Code, I have no idea how to do that. I tried adding __init__.py files in every directory and various syntaxes for the import statements.
From what I've read, this way of importing is not actually supported in Python 3 (which I find weird, isn't it one of the most common use cases of the import system?)--yet it was really easy in Eclipse with PyDev where I could just go into the project options and select "referenced" projects. I was then able to write e.g. import project_one in script_c.py and it worked perfectly. I now assume that maybe the absolute paths were stored in some Eclipse project file that way? Does VS Code have such a feature? If not, how would I continue programming without having to use some of the "dirty hacks" I read that maybe enable this?
I'm looking at statically link Python with my application. The reason for this is because in some test cases I've seen a 10% speed increase. My application uses the Python C-API heavily, and it seems that Whole Program Optimization is able to do some good optimizations. I expect Profile Guided Optimizations will gain a little more too. This is all being done in MSVC2015
So far I've recompiled the pythoncore project (python35.dll) into a static library and linked that with my application (let's call it myapp.exe). FYI other than changing the project type to static, the only other thing that needs doing is setting the define Py_NO_ENABLE_SHARED during the static lib compile, and when compiling myapp.exe. This works fine and it's how I was able to obtain the 10% speed improvement test result.
So the next step is continuing to support external python modules that have .pyd files (which are .dll files renamed to .pyd). These modules will have been compiled expecting to dynamically link with python35.dll, so I need to provide a workaround for that requirement, since all of the python functions are now embedded into myapp.exe.
First I use a .def file to export all of the public Python functions from myapp.exe. This works fine.
The missing piece is how do I create a python35.dll which redirects all the calls to the functions exported from myapp.exe.
My first attempt is using DLL forwarding. I made a custom python35.dll which has a .def file with lines such as:
PyArg_Parse=myapp.PyArg_Parse
In theory, this works. If I use Dependency Walker on socket.pyd, it correctly opens my python35.dll and shows that all the calls are being forwarded to myapp.exe.
However when actually running myapp.exe and trying to import socket, it fails to load the required entry points from myapp.exe. 'import socket' in Python will cause a LoadLibrary("socket.pyd") to occur. This will load my custom python35.dll implicitly. The failure occurs while trying to load python35.dll, it's unable to find the entry points for it's forwards. It seems like the reason for this is because myapp.exe won't be part of the library search path. I seem to be able to verify this by copying myapp.exe to myapp.dll. If I do that then the python35.dll load works, however this isn't a solution since that will result in 2 copies of the Python enviroment (one in myapp.exe, one in myapp.dll)
Possible other avenues I've also looked into but haven't found the right solution for:
Somehow getting .exe files to be part of the library search path
Using Windows manifest/configuration to redirect the library somehow
Manually using declspec(naked) and jmp statements to more explicitly wrap the .dll. I'm working in x64, so I don't think that's possible anymore?
I could manually do the whole Python API and wrap each function manually. This is doable if I can find a way to create the function definitions of all the exports so it's not an insane amount of manual work.
In summary, is there a way to redirect/forward calls to a .dll to functions/data exported from an .exe. Thanks!
I ended up going with the solution that #martineau suggested in the comments, which was to put all of my application, including Python, into a single .dll instead of an .exe. Then the .exe is just a simple file that calls into the .dll and does nothing else.
This question is not about how to bundle a python application into a exe, or a binary. I can (almost) figure that by myself, with all the doc on the internet.
My question is more about what the final user will be able to see about my program. And for my personal culture.
For example, with cx_freeze, if I try to compile my app, I end up with a build directory. Inside, I have the binary of my app, let's call it "gui". But around, I have a bunch of *.so files.
Basically, the user can see every lib I used to build my app. I thought (maybe naively) that if I could create a unique binary file, all the libs would be included in the binary, and so, not "visible" by the user.
Do I think right or is it completely wrong ? Is it possible to bundle all the libs into one single binary, and mask them ? (I know cx_freeze can't handle a single bundle file).
You may want to consider pyinstaller. With the --onefile switch you can bundle everything into one file. However, it only fully supports python X2. This is a good tutorial to get started with it .
If you cannot use pyinstaller, you may want to try to use the answer posted here.
EDIT:
After having read the cx_freeze documentation, it turns out that you cannot bundle everything into a single exe. You will have to create a self extracting file with 7zip or IEXPRESS (if you are on Windows) that just unpacks all of your .so files.
Currently, I'm deploying a full python distribution (the original python 2.7 msi) with my app. Which is an embedded web server made with delphi.
Reading this, I wonder if is possible to embed the necessary python files with my app, to decrease load files and avoid conflict with several python versions.
I have previous experience with python for delphi so I only need to know if only shipping the python dll + zip with the distro + own scripts will work (and if exist any caveats I must know or a sample where I can look)
zipimport should work just fine for you -- I'm not familiar with Python for Delphi, but I doubt it disables that functionality (an embedding application can do that, but it's an unusual choice). Just remember that what you can zip up and import directly are the Python-coded modules (or just their corresponding .pyc or .pyo byte codes) -- DLLs (even if renamed as .pyds;-) need to be on disk to be loaded (so if you have a zipfile with them it will need to be unzipped at the start of the app, e.g. into a temporary directory).
Moreover, you don't even need to zip up all modules, just those you actually need (by transitive closure) -- and you can easily find out exactly which modules those are, with the modulefinder module of the standard Python library. The example on the documentation page I just pointed to should clarify things. Happy zipping!
Yes it is possible.
I'm actually writing automatisation script in Python with the Zipimport library. I actually included every .py files in my zip as well as configuration or xml files needed by those script.
Then, I call a .command file targeting a __main__.py class that redirect towards the desired script according to my sys.argv parameters which is really useful!
This question already has answers here:
Closed 13 years ago.
Possible Duplicate:
Gather all Python modules used into one folder?
I don't think this has been asked before-I have a folder that has lots of different .py files. The script I've made only uses some-but some call others & I don't know all the ones being used. Is there a program that will get everything needed to make that script run into one folder?
Cheers!
Since Python is not statically linked language, this task would be rather a challenging one. Especially if some of your code uses eval(...) or exec(...).
If your script is not very big, I would just move it out, make sure that your python.exe does not load modules from that directory and would run the script and add missing modules until it works.
I you have multiple scripts like this, then this manual work is not really the way to go. But in this case also having lots of different .py files in a directory is not a good deployment technique and you should think about packaging them into installable modules and install into your python site-packages.
Still you may use snakefood package to find our the dependencies (has already been discussed here). Again, it just cannot be 100% accurate, but should give you an easy start.
you should be able to extract the needed information from a so called call graph
See for example
http://pycallgraph.slowchop.com/ or
http://blog.prashanthellina.com/2007/11/14/generating-call-graphs-for-understanding-and-refactoring-python-code/
Also, py2exe converts a python call into an executable and in this process it gathers all used modules. I think py2exe is cross platform
I'd try 2½ solutions, one elaborate and 1½ quick-and-dirty:
elaborate: a custom import hook, logging all imports
quick and dirty, part a: os.utime the *.py[co]? (re notation, not glob) files to having access times of yesterday, then run the program and collect all recent access times. Prerequisite: a filesystem that marks access times (by itself and by its mount options).
quick and dirty, part b: remove all *.py[co] files (same in glob and re notation), run the program, see which have been created. Prerequisite: user should have write access to the folder.