I am using PyInstaller to create a single-file executable. Is it possible for my script to perform an import such that i) the imported module is imported from the same directory as the exe (i.e. it's not packaged into the exe) and ii) that imported module can import other modules that were packaged into the exe?
The background here is that the imported module contains configuration that the user should be able to modify. This may include creation of custom derived classes and use of enums from the packaged modules.
I haven't found any advice on this, though it's a difficult search because there are so many similar topics that use basically the same keywords.
The following steps allow a Python module (named external_module here) outside of an executable created by PyInstaller to be imported and for that module to import modules that were bundled into the executable.
Add excludes=['external_module'] to the Analysis object used in the PyInstaller spec. This prevents external_module.py being bundled into the executable.
Add sys.path.append(os.path.dirname(sys.executable)) where external_module is imported in your application. This allows it to be imported from the directory the executable is in, which is different to the directory that the application will run in (due to being decompressed to a temporary folder). See below for my recommended method of achieving this.
Make sure any imports performed by external_module.py are also performed by one of the bundled modules before external_module.py is imported. The interpreter will not resolve the external module's imports against bundled modules, but will use ones that already exist in sys.modules.
In order to set up the paths correctly you can use the following:
if getattr(sys, 'frozen', False):
app_path = os.path.dirname(sys.executable)
sys.path.append(app_path)
else:
app_path = os.path.dirname(os.path.abspath(__file__))
frozen is only available in generated executables, not when running as a script directly. This snippet will add the executable's location to sys.path if required as well as giving you easy access to the executable or script's location for use in code.
As an example of the final bullet point, consider the following.
# bundled_module1.py
import external_module
# bundled_module2.py
# module content
# external_module.py
import bundled_module2
This will fail in external_module.py because bundled_module2 can't be found. However, the following modification to bundled_module1.py will work:
# bundled_module1.py
import bundled_module2
import external_module
This will be fine if there are a limited set of bundled modules that the external one should be able to import. It may get unwieldy for larger sets.
Given that the documentation states that the interpreter will resolve imports against modules bundled into the executable, this feels like a possible bug. Interoperating with modules outside of the executable isn't explicitly called out though.
Type in Pyinstaller -h. It will give you info about pyinstaller and tell you about --runtime-hook. I presume adding this to the executable should work. There's actually a whole page of documentation for this. Surprised you could not find that.
Anyway,
The docs say put in:
pyinstaller --additional-hooks-dir=. myscript.py.
I presume then something like pyinstaller --additional-hooks-dir=C:\pathtofolder myscript.py should work in theory. Yet to test it. Tell us how it goes and what made kinks made it work for you.
Lastly, if you want to be hipster try integrating cython for speed and obfuscation. Fair warning, cython is not as user friendly as pyinstaller appears to be. I have yet to use it successfully.
Related
I have two packages (say, dataread and datainspector) that were somehow not detected by PyInstaller. Because of this, the application terminates when the running application reaches the point where it needs to import modules from those packages.
The easiest solution would be to copy dataread and datainspector into packaged app. But this will break the intention of packaging a binary version of the application.
I've read about hidded imports and hook, and I think that both can solve the problem, but I am not sure of which one to use.
Any suggestions?
PS: both these packages may contain nested directories.
Hooks are files that specify additional actions when PyInstaller finds import statements.
If you add a hook-data.py file which contains a line hiddenimports = ['_proxy', 'utils', 'defs'], PyInstaller will check it to find additional imports when it sees import data.
You have to specify the path to the hook directory via --additional-hooks-dir (this is useful if you don't want to mix your source code with compile-only files, and instead keep them in a separate directory).
The simpler solution is to use --hidden-import=modulename along with the PyInstaller script. It will add modulename as import statement silently.
Hooks are better if you want to specify which import needs what additional modules. --hidden-import is simpler as a one-shot or for debugging.
More info - https://pyinstaller.readthedocs.io/en/stable/hooks.html
Use Auto-py-to-exe to add hindden libraries.
In additional files add Rasterio folder (C:/users/Admin/anaconda3/envs/name/Lib/site-packages/rasterio) and then convert to exe.
It worked for me.
if it says some other libraries are missing then add folder of that library too.
I am creating a Python package with multiple modules. I want to make sure that when I import modules within the package that they are importing only from the package and not something outside the package that has the same name.
Is the correct way of doing this is to use relative imports? Will this interfere when I move my package to a different location on my machine (or gets installed wherever on a customer's machine)?
Modern relative imports (here's a reference) are package-relative and package-specific, so as long as the internal structure of your package does not change you can move the package as a whole around wherever you want.
While Joran Beasley's answer should work as well (though does not seem necessary in those older versions of Python where absolute imports aren't the default, as the old style of importing checked within the package's directory first), I personally don't really like modifying the import path like that when you don't have to, especially if you need to load some of those other packages or modules that your modules or packages now shadow.
A warning, however: these do require that the module in question is loaded as part of a package, or at least have their __name__ set to indicate a location in a package. Relative imports won't work for a module when __name__ == '__main__', so if you're writing a simple/casual script that utilizes another module in the same directory as it (and want to make sure the script will refer to the proper directory, things won't work right if the current working directory is not set to the script's), you could do something like import os, sys; sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) (with thanks to https://stackoverflow.com/a/1432949/138772 for the idea). As noted in S.Lott's answer to the same question, this probably isn't something you'd want to do professionally or as part of a team project, but for something personal where you're just doing some menial task automation or the like it should be fine.
the sys.path tells python where to look for imports
add
import sys
sys.path.insert(0,".")
to the top of your main python script this will ensure local packages are imported BEFORE builtin packages (although tbh I think this happens automagically)
if you really want to import only packages in your folder do
import sys
sys.path = ["."]
however I do not recommend this at all as it will probably break lots of your stuff ...
most IDE's (eclipse/pycharm/etc) provide mechanisms to set up the environment a project uses including its paths
really the best option is not to name packages the same as builtin packages or 3rd party modules that are installed on your system
also the best option is to distribute it via a correctly bundled package, this should more than suffice
I recently tried pyInstaller and there are some things i don't quite get. i have been trying to create some executables (NOTE: all of them use numpy, scipy, OpenCV, BLAS etc) but i have been failing. There is always something missing. So my question is, can someone explain better to me what are hiddenimports and hooks, and how can i tell pyInstaller the directories of all the dependencies in my code so it can pack the in the final executable.
Thank you.
From the pyinstaller documentation
hiddenimports
A list of modules names (relative or absolute) the module imports in some untrackable way.
Some Python imports are untrackable during static analysis of your program. eg Your code may create the name of a module to import using Python code, and then import that module. In this case, pyinstaller will be unable to work out during its code analysis, what is the name of the module to import. If you know ahead of time, then you can tell pyinstaller to unconditionally include these modules.
Hooks are a way for you to bundle a set of hidden imports and other parameters to do with finding modules. Hooks are named hook-<module>.py where module is a fully qualified module name. eg hook-xml.dom.py. If your code does import xml.dom, then the contents of the hook script are read to include any hidden imports specific to xml.dom.
If you create your own module and it requires hidden imports, you can create a hook script with the appropriate hidden imports settings and store in the PyInstaller hooks directory. The next time you use PyInstaller to freeze a program which imports your module, it will automatically find your hook file and pull in the required hidden imports without you having to remember each time what the hidden imports are for your module.
The documentation has more information about how all of this works but hopefully this provides some more background information.
My python program uses plugins (python files) which I import dynamically using __import__.
I bundle my python program into a Windows exe using py2exe.
I've just spend searching 2 hours why my plugin python file couldn't be loaded properly from the .exe version. I got an ImportError: "no module named urllib2"
It appeared my plugin was using urllib2 (through an import urllib2 statement), and that standard library module was apparently not bundled into the exe. Other modules used in the plugin (re, urllib, ...) gave no problem, but perhaps they were already references in python files I statically include in my program.
How can I know which standard Python library modules py2exe bundles by default in the exe? (so I know whether I or somebody else can use them in a plugins). The py2exe documentation doesn't give an hints, except for saying that it includes a lot of modules from the standard library.
To see which modules are included look inside the library.zip (if there is no library.zip file - then try opening the EXE in any ZIP application - or rename it to .ZIP and try and open it).
You will be able to see a list of *.pyc. You can look at the list of files and directories to get an impression of which modules are included or not.
If you require a specific package to be added - add it to the 'packages' list.
As to why it doesn't include everything or how it chooses to include some and not others? My understanding is that py2exe looks in your code to figure out what you are using and includes those (and some that it probably needs itself) but maybe it also has some heuristics to add other modules too (I haven't checked :)
In a Python system for which I develop, we usually have this module structure.
mymodule/
mymodule/mymodule/feature.py
mymodule/test/feature.py
This allows our little testing framework to easily import test/feature.py and run unit tests. However, we now have the need for some shell scripts (which are written in Python):
mymodule/
mymodule/scripts/yetanotherfeature.py
mymodule/test/yetanotherfeature.py
yetanotherfeature.py is installed by the module Debian package into /usr/bin. But we obviously don't want the .py extension there. So, in order for the test framework to still be able to import the module I have to do this symbolic link thingie:
mymodule/
mymodule/scripts/yetanotherfeature
mymodule/scripts/yetanotherfeature.py # -> mymodule/scripts/yetanotherfeature
mymodule/test/yetanotherfeature.py
Is it possible to import a module by filename in Python, or can you think of a more elegant solution for this?
The imp module is used for this:
daniel#purplehaze:/tmp/test$ cat mymodule
print "woho!"
daniel#purplehaze:/tmp/test$ cat test.py
import imp
imp.load_source("apanapansson", "mymodule")
daniel#purplehaze:/tmp/test$ python test.py
woho!
daniel#purplehaze:/tmp/test$
You could most likely use some tricker by using import hooks, I wouldn't recommend it though. On the other hand I would also probably do it the other way around , have your .py scripts somewhere, and make '.py'less symbolic links to the .py files. So your library could be anywhere and you can run the test from within by importing it normall (since it has the py extension), and then /usr/bin/yetanotherfeature points to it, so you can run it without the py.
Edit: Nevermind this (at least the hooks part), the import imp solution looks very good to me :)
Check out imp module:
http://docs.python.org/library/imp.html
This will allow you to load a module by filename. But I think your symbolic link is a more elegant solution.
Another option would be to use setuptools:
"...there’s no easy way to have a script’s filename match local conventions on both Windows and POSIX platforms. For another, you often have to create a separate file just for the “main” script, when your actual “main” is a function in a module somewhere... setuptools fixes all of these problems by automatically generating scripts for you with the correct extension, and on Windows it will even create an .exe file..."
https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation