I’m trying to write a script that dynamically imports and uses any modules a user places in a folder. The dynamic importing works fine when I’m running it via python, but when I try to compile it into a Pyinstaller executable, it breaks down and throws me a ModuleNotFoundError, saying it can't find a module with the same name as the folder the modules are placed in. The executable sits alongside this folder, which contains all the modules to be dynamically imported, so my import statements look like __import__("FOLDERNAME.MODULENAME"). The script must be able to run the modules dropped in this folder without being recompiled. What's strange is that the ModuleNotFoundError says No module named 'FOLDERNAME', despite that just being the name of the folder containing the modules, I'd expect it to complain about No module named 'FOLDERNAME.MODULENAME' instead.
In my googling, I found this question (pyinstaller: adding dynamically loaded modules), which is pretty similar, but the answer they provided from the docs doesn’t really help. How do I give additional files on the command line if I don’t know what files are going to be in the folder in the first place? That kind of beats the purpose of dynamic importing. I've attempted to use the hidden-import command line flag, but the compiler output said Hidden import '[X]' not found. Maybe I'm just using it wrong? And I have no idea how to modify the spec file or write a hook file to do what I need. Any help would be greatly appreciated.
I was working on a similar functionality to implement a Plugin Architecture and ran into the same issue. Quoting #Gao Yuan from a similar question :-
Pyinstaller (currently v 3.4) can't detect imports like importlib.import_module(). The issue and solutions are detailed in Pyinstaller's documentation, which I pasted below as an entry point.
But of-course there is always a way. Instead you can use importlib.util.spec_from_file_location to load and then compile the module
Minimum wokring example
iterface.py
# from dependency import VARIABLE
# from PySide6.QtCore import Qt
def hello_world():
print(f"this is a plugin calling QT {Qt.AlignmentFlag.AlignAbsolute}")
print(f"this is a plugin calling DEPENDENCY {VARIABLE}")
cli.py
import sys
import types
from pprint import pprint
import importlib.util
import sys
if __name__ == "__main__":
module_name = "dependency"
module_file = "plugins/abcplugin/dependency.py"
if spec:=importlib.util.spec_from_file_location(module_name, module_file):
dependency = importlib.util.module_from_spec(spec)
sys.modules[module_name] = dependency
spec.loader.exec_module(dependency)
module_name = "interface"
module_file = "plugins/abcplugin/interface.py"
if spec:=importlib.util.spec_from_file_location(module_name, module_file):
interface = importlib.util.module_from_spec(spec)
sys.modules[module_name] = interface
spec.loader.exec_module(interface)
sys.modules[module_name].hello_world()
project structure
cli.exe
plugins
abcplugin
__init__.py
interface.py
dependency.py
complus
__init__.py
...
Thumb Rules
Plugin must always be relative to .exe
As you can notice I commented out # from dependency import VARIABLE in line one of interface.py. If you scripts depend on scripts in the same plugin, then you must load dependency.py before loading interface.py. You can then un-comment the line.
In pyinstaller.spec file you need to add hiddenimports in this case PySide6 and then un-comment # from PySide6.QtCore import Qt
Always use absolute imports when designing a plugin in reference to your project root folder. You can then set the module name to plugins.abcplugin.interface and plugins.abcplugin.dependency and also update from dependency import VARIABLE to from plugins.abcplugin.dependency import VARIABLE
Hope people find this usefull, cheers!!
Related
I have three scripts:
C:\code\voiceTerm\master.py:
from voice_terminal_module.voice_terminal import VoiceTerminal
vterm = VoiceTerminal()
C:\code\voiceTerm\voice_terminal_module\voice_terminal.py:
from chatbot_module.chatbot_module import Chatbot
class VoiceTerminal:
print("INITIALIZING VOICE TERMINAL")
cb = Chatbot()
C:\code\voiceTerm\voice_terminal_module\chatbot_module\chatbot_module.py:
class Chatbot:
print("CHATBOT INITIALIZED")
Here is the wierd thing: When I run chatbot_module.py it works, and if I run voice_terminal.py it works. For some reason however, master.py errors out with the following message:
Traceback (most recent call last):
File "c:\code\voiceTerm\master.py", line 1, in <module>
from voice_terminal_module.voice_terminal import VoiceTerminal
File "c:\code\voiceTerm\voice_terminal_module\voice_terminal.py", line 1, in <module>
from chatbot_module.chatbot_module import Chatbot
ModuleNotFoundError: No module named 'chatbot_module'
Why does it work sometimes, but sometimes not?
You'll need to restructure your project in a way that you have either: (1) two separate and installed packages containing each module or; (2) one package containing all modules. The simplest way forward is (2) and it looks like:
master.py
src/ # Name it anything you wish
__init__.py
voice_terminal.py
chatbot_module.py
Then use relative imports in voice_terminal.py
from .chatbot_module import Chatbot
Rule: Packages does not know others exists unless the other package is installed, added to sys.path, a subpackage, or part of a parent package.
Python general importing guide (for your own code)
I'm sure there should be a duplicate for this by now, but I can't find it, so I'm writing a full essay.
To solve the problem properly, you must take care of two things:
Make sure that sys.path - the path that Python searches for modules - includes the path to the project root.
Make sure that the import statements in the code are written to work with that path.
The search path
For the first part, you must understand how sys.path works. It is a list of folders where Python will look, that is set up automatically at startup (even if you don't import sys - just like sys.argv is).
Normally, it will be a list which contains, in order:
A path to the entry point for execution (the details of this will depend on how Python is started; typically this is '' when running a .py file directly from the same directory, a path the another directory if you do e.g. python somewhere/else/file.py, and an explicit path when using the -m switch)
Paths to system libraries
Paths to virtual environment libraries, if a virtual environment is active
Paths to things that were explicitly installed by the user
You can modify this list, with the expected impact on future import statements; but you ordinarily should not.
To ensure your project root is on the path, normally you should just install the project - ideally into a virtual environment. Please see the Python packaging guide for details. Failing that, make sure to start the project from just outside the folder containing your "top-level" code. In OP's case, that means C:\code. This will ensure that C:\code (or something equivalent) is on sys.path, which we can then rely upon for the second step.
The imports
We can fundamentally do this in two ways: By absolute imports specifying the path from the root, or by relative imports specifying the path from the current source file. Read more on Stack Overflow about relative imports here and here.
Absolute imports
In either case, we want to treat the root folder for our project (here, C:\code\voiceTerm) as a package. For absolute imports, this means we will always mention the root folder name in our import path.
Thus:
In C:\code\voiceTerm\master.py: from voiceTerm.voice_terminal_module.voice_terminal import VoiceTerminal
In C:\code\voiceTerm\voice_terminal_module\voice_terminal.py: from voiceTerm.voice_terminal_module.chatbot_module.chatbot_module import Chatbot
(You don't really want to have _module in your folder or file names. It doesn't really add information, makes this considerably harder to type, and is actually a bit misleading.)
We can also import entire modules: import voiceTerm.master, import voiceTerm.voice_terminal_module.voice_terminal; import voiceTerm.voice_terminal_module.chatbot_module.chatbot_module. Additionally, by naming a file __init__.py, we become able to import the folder as a package: import voiceTerm; import voiceTerm.voice_terminal_module; import voiceTerm.voice_terminal_module.chatbot_module. We can also import a module from one of those packages: from voiceTerm import master; from voiceTerm.voice_terminal_module import voice_terminal; from voiceTerm.voice_terminal_module.chatbot_module import chatbot_module.
Relative imports
With relative imports, we must use the from syntax, and we can only import things from within our own package hierarchy. However, we have the advantage that we don't have to specify full paths, and we can rename packages without having to edit the code. My personal recommendation is to use relative imports where possible; this also makes a strong visual distinction with imports from system libraries or other third-party packages.
For a relative import, first we specify the package or sub-package that we're importing from, using a relative import path. By starting with a single ., we start from the current directory, looking for another module or package within the same folder.
Thus:
In C:\code\voiceTerm\master.py: from .voice_terminal_module.voice_terminal import VoiceTerminal to import the class; from .voice_terminal_module import voice_terminal to import the module.
In C:\code\voiceTerm\voice_terminal_module\voice_terminal.py: from .chatbot_module.chatbot_module import Chatbot to import the class; from .chatbot_module import chatbot_module to import the module.
Additional .s navigate up the package hierarchy (but we cannot go beyond the root this way). For example, from ... import master to import the top-level master.py from the lower-down chatbot_module.py. (Of course, this wouldn't actually work in this case, because it would be a circular import.) To import another source file (as a module) from the same directory, simply from . import other_file. You get the idea.
I solved it. You can appearantly use importlib for this. I replaced the import code in my voice_terminal.py script with this:
chatbot_module = SourceFileLoader("chatbot_module", "C:/code/voiceTerm/voice_terminal_module/chatbot_module/chatbot_module.py").load_module()
I have been creating an application that will run just like Microsoft Notepad. My program comprises of the following file and folder:
libs (Folder)
scripts (Folder)
test.py
The test.py is the file in which I am trying the following code:
from .libs import tkinter
from .libs import pyglet
from .libs import threading
test_window = Tk()
test_window.mainloop()
In the folder libs I have added the modules that are required in the application.
Example: Tkinter, Pyglet and Threading
When I am trying to import them into test.py using the code above and then running the program. I see the following Traceback:
Traceback (most recent call last):
File "c:\Users\Bhavyadeep\Desktop\NuclearPad\test.py", line 1, in <module>
from .libs import tkinter
ImportError: attempted relative import with no known parent package
I don't really know how I solve this problem. Is it because I added the module in the folder where Import in not available? Or is it because I did something wrong in this module folder copying-pasting to a folder not at Python's PATH?
from .libs import tkinter
Means "import tkinter from the package called lib which is located in the same directory as file (this script)"
so your first line should say something like
test_window = tkinter.Tk()
unless you use either of the following:
from .libs.tkinter import *
from .libs.tkinter import Tk
The error is likely due to you using relative import syntax from within the script you are trying to execute. Remove the leading dots in order to execute test.py, otherwise append its directory to sys.path or use site.addsitedir(path_to_directory_containing_test_py) and import it as follows import test.
With that said, you should not call the module's file test.py because there is already something with that name in python's default Lib directory or builtins. If you insist, there should be a hacky way to get it done without changing the name through importlib.
As for why this error occurs, I think it's because the interpreter is designed to execute scripts but not modules/packages whose contents are placed in .pyc files in __pycache__s when imported.
In order to import from scripts or libs, they must be packages. In order to turn them into packages you must add __init__.py files to them.
Lastly, It's a pretty terrible idea to place third-party libraries in your source like this. At the very least you should have installed them properly on your machine and then copied their sources from site-packages/the equivalent for venvs, which will fail if they use any executables and/or have any external dependencies
Documentation for distributing python packages
As per screen print, import shows error in Python 3.7 version, earlier it was working fine in version Python 2.7 and I am using IntelliJ Idea.
If you see, EOC related .py files are in the same folder and have classes which are being called in Main_EOC.py by passing objects which are inter-related. It's amazing to see the red line while importing files from same folder.
Please help me why it's showing such error
"This inspection detects names that should resolve but don't. Due to dynamic dispatch and duck typing, this is possible in a limited but useful number of cases. Top-level and class-level items are supported better than instance items.`"
Also, if you see the line which have full path, is not showing error
from EOC_Module.eoc.script.config import Config
Please help me if there is a way to add this full path on top of the code or other option.
The behavior of import path search changed between python2 and python3. The import path always includes the directory from which the main module was loaded, but it no longer includes directories from which modules were imported.
You need to change your import statement syntax as follows, if you want to import a module that lives in the same directory as the module in which you do the import:
# old way, import works if the named module is in this module's directory
import x
# new (Python3) way:
from . import x
For the second part: adding a path so all code can import from a certain directory: if that directory is (and will always be) relative to your main: you can add a few lines in the main module to make it available. Something like this:
import sys # if you haven't imported it already
import os.path
home = os.path.dirname(sys.argv[0])
sys.path.append( os.path.join(home, "EOC_Module/eoc/script") )
# now, you can import straight from the script directory
import EOC_Intraction
When using pycharm the root directory for your python executable is the same as the root directory of your project, this means that python will start looking for files in the root directory with this files:
.idea/
EOC_module/
logs/
reports/
sql/
This is the reason of why: from EOC_Module.eoc.script.config import Config works.
If you execute your code from the terminal with: python3 Main_EOC.py (not pycharm) the root directory for your python will be the same as the one containing the file, all the other imports will work but from EOC_Module.eoc.script.config import Config not.
So you need to make your imports from project directory if you are using pycharm.
Desiring to improve my Python style, I ran a PEP-8 style checker on one of my script and it complained about something I don't know how to fix. The prologue of the script is something like:
#! /bin/env python3
import sys
import os
exe_name = os.path.basename(os.path.realpath(__file__))
bin_dir = os.path.dirname(os.path.realpath(__file__))
inst_dir = os.path.dirname(bin_dir)
sys.path.insert(0, inst_dir+'/path/to/packages')
import mypackage.mymodule
and the style checker complain on the import mymodule line, stating that it should be a top of file. But I obviously can't move it before setting the path where it will be found. Is there a good way to achieve this (mandating an environment variable or a shell wrapper are not what I find better than my current code) while respecting PEP-8 recommendations at the same time?
If you want to avoid path manipulation, you may be able to do so by using the under-known .pth feature.
sys.path should begin with the directory containing the main program either by name or by reference as ''. I assume that the file importing mymodule is not part of mypackage, so that the '' entry is not useful for importing mymodule.
sys.path should end with the site-packages directory for the executing binary. That is the normal place for added packages. If you do not want to move mypackage into site-packages, you can extend the latter 'vitually' by putting a mystuff.pth file in it. It should contain one line: the path to the directory containing mypackage. Call it myprojects. Then mypackage and any other package in myprojects can be imported as if they were in site-packages.
One advantage of .pth files is that you can put identical copies in multiple site-packages directories. For instance, I have multiple projects in F:/python. I have multiple versions of Python installed. So I have put python.pth containing that one line in the site-packages for each.
The best strategy would be to put the sys.path related code in separate file and import it in working code file.
So, I will split above code in two files. One named my_module.py and other named working_module.py
Now my_module will contain below lines
import sys
import os
exe_name = os.path.basename(os.path.realpath(__file__))
bin_dir = os.path.dirname(os.path.realpath(__file__))
inst_dir = os.path.dirname(bin_dir)
sys.path.insert(0, inst_dir+'/path/to/packages')
and working_module will contain
import my_module
import mypackage.mymodule
because we are importing my_module before mypackage, it will execute the path related code before and your package will be available in path.
You can use importlib module (python 3.5) or imp for python 2.7 to load mypackage.mymodule programatically. Both have the same purpose:
mechanisms used to implement the import statement
This question might help you more:
How to import a module given the full path?
https://docs.python.org/3/library/importlib.html#examples
https://docs.python.org/2/library/imp.html
I'm trying to build an app with py2app. I can build the app but when I run it I get ImportError in the console, the import error is that there is No module named PythonApp which is the folder all my source is in, the location of which is in my PYTHONPATH.
I have been importing local files like from PythonApp import file instead of just import file to help avoid namespace issues.
I have tried to build it with the following flag python setup.py py2app --use-pythonpath but this hasn't appeared to make any difference.
Should I just change the import statements throughout to import file?
How can I make py2app realise my PYTHONPATH?
If I understood your problem well, you have the following layout:
PythonApp/
__init__.py
file.py
foo.py
And you try to import the module file from foo.py by from PythonApp import file. If this the only reason you want to set PYTHONPATH, there is a simpler solution: use relative imports:
from . import file
You can use from .. import file in a sub-package of PythonApp, and so on. This way you can avoid name collisions with standard modules.
If you need to hack the import path for some other reasons, you can also set the sys.path variable in the startup script (probably py2app has some options for that, too). Keep in mind though that if you add external directories into the import path, it will be harder to distribute the app bundle.
Also, a more trivial explanation for the ImportError is that py2app did not copy your package into the app bundle. Make sure you have all your packages listed in the packages parameter of setup().