Develop Reference

Dynamic Importing with Pyinstaller Executable

I’m trying to write a script that dynamically imports and uses any modules a user places in a folder. The dynamic importing works fine when I’m running it via python, but when I try to compile it into a Pyinstaller executable, it breaks down and throws me a ModuleNotFoundError, saying it can't find a module with the same name as the folder the modules are placed in. The executable sits alongside this folder, which contains all the modules to be dynamically imported, so my import statements look like __import__("FOLDERNAME.MODULENAME"). The script must be able to run the modules dropped in this folder without being recompiled. What's strange is that the ModuleNotFoundError says No module named 'FOLDERNAME', despite that just being the name of the folder containing the modules, I'd expect it to complain about No module named 'FOLDERNAME.MODULENAME' instead.
In my googling, I found this question (pyinstaller: adding dynamically loaded modules), which is pretty similar, but the answer they provided from the docs doesn’t really help. How do I give additional files on the command line if I don’t know what files are going to be in the folder in the first place? That kind of beats the purpose of dynamic importing. I've attempted to use the hidden-import command line flag, but the compiler output said Hidden import '[X]' not found. Maybe I'm just using it wrong? And I have no idea how to modify the spec file or write a hook file to do what I need. Any help would be greatly appreciated.

I was working on a similar functionality to implement a Plugin Architecture and ran into the same issue. Quoting #Gao Yuan from a similar question :-
Pyinstaller (currently v 3.4) can't detect imports like importlib.import_module(). The issue and solutions are detailed in Pyinstaller's documentation, which I pasted below as an entry point.
But of-course there is always a way. Instead you can use importlib.util.spec_from_file_location to load and then compile the module
Minimum wokring example
iterface.py
# from dependency import VARIABLE
# from PySide6.QtCore import Qt
def hello_world():
print(f"this is a plugin calling QT {Qt.AlignmentFlag.AlignAbsolute}")
print(f"this is a plugin calling DEPENDENCY {VARIABLE}")
cli.py
import sys
import types
from pprint import pprint
import importlib.util
import sys
if __name__ == "__main__":
module_name = "dependency"
module_file = "plugins/abcplugin/dependency.py"
if spec:=importlib.util.spec_from_file_location(module_name, module_file):
dependency = importlib.util.module_from_spec(spec)
sys.modules[module_name] = dependency
spec.loader.exec_module(dependency)
module_name = "interface"
module_file = "plugins/abcplugin/interface.py"
if spec:=importlib.util.spec_from_file_location(module_name, module_file):
interface = importlib.util.module_from_spec(spec)
sys.modules[module_name] = interface
spec.loader.exec_module(interface)
sys.modules[module_name].hello_world()
project structure
cli.exe
plugins
abcplugin
__init__.py
interface.py
dependency.py
complus
__init__.py
...
Thumb Rules
Plugin must always be relative to .exe
As you can notice I commented out # from dependency import VARIABLE in line one of interface.py. If you scripts depend on scripts in the same plugin, then you must load dependency.py before loading interface.py. You can then un-comment the line.
In pyinstaller.spec file you need to add hiddenimports in this case PySide6 and then un-comment # from PySide6.QtCore import Qt
Always use absolute imports when designing a plugin in reference to your project root folder. You can then set the module name to plugins.abcplugin.interface and plugins.abcplugin.dependency and also update from dependency import VARIABLE to from plugins.abcplugin.dependency import VARIABLE
Hope people find this usefull, cheers!!

Import package from parent folder [duplicate]

This question already has answers here:
Relative imports for the billionth time
(12 answers)
Closed last year.
So basicly I want to acces a created module from a folder on the parent directory from the folder I am
Currently I'm at Twitter.py and I want to access /utils/magic_eden.py
On the __init__.py file I've got:
from .magic_eden import MagicEden
from .open_sea import OpenSea
from .tools import Tools
Now inside the Twitter.py file im trying to import these classes of the module by doing:
from utils import MagicEden
But im getting ModuleNotFoundError: No module named 'utils'.
I've tried so many sort of things but none worked. What should I do?
(btw if I execute the __init__.py file I get ImportError: attempted relative import with no known parent package)

From what I see, it seems that utils is a utility package logically separate from the code in Twitter.py. Given this, what you want is to cause your utils package to be on your Python search path (sys.path) so that you can import that package as a separate entity (no relative paths). If you don't want to configure your environment to place utils on the Python search path, you can do it in your Twitter.py file. Here's how that looks:
import os
import sys
here = os.path.dirname(__file__)
sys.path.append(os.path.join(here, '..'))
import utils
utils.MagicEden.hello()
utils.OpenSea.hello()
utils.Tools.hello()
The first line of code computes the full path to the directory containing Twitter.py. The second line then computes the path to the parent directory of that directory and adds that path to sys.path, the Python search path. After doing this, import utils will work, giving you access to that package and everything imported in that package's __init__.py file.
I created three small files for magic_eden.py, open_sea.py, and tools.py, each containing something that looks like this:
class MagicEden:
#staticmethod
def hello():
print("Hello from MagicEden!")
I can then run Twitter.py, and I get the following result with no additional configuration:
Hello from MagicEden!
Hello from OpenSea!
Hello from Tools!
There's nothing wrong with using the above solution during early development. But you will likely at some point want to remove the code that is hacking sys.path and instead install your module in a more official way. There's a way to do this from the start so that you never have to change code on either side when you want to install your module in the official way...
What I do in situations like this is create a setup.py file in my package that lets me build it as an installable package. There are many docs and tutorials on the net that explain how to do this. Here's just one of them: https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html
Once you've done this, what you can do is install your package in "development mode" (pip install -e <package file>). What this does is install the package so that your system can find it (adds it to sys.path as a package), but installs links to the original sources rather than installing copies of them. In this mode, when you make changes to that package's source files, the changes take immediate effect in the installed module. So now you have the best of both worlds. Your package is installed in the official way such that you don't need to do anything special in the code that imports that package (no need to modify sys.path), and yet you can still make changes directly to the sources for that package and not have to keep reinstalling the package to see those changes take affect.
When you're done messing with a package in this way, you can reinstall it in the regular mode (pip install <package file>) and isolate changes to the sources for that package from uses of the package elsewhere on your system. At this point, you need to rebuild and reinstall the package to see any changes you've made to its sources.

You're attempting to import from a "sibling" folder.
To do this, you'll need to add __init__.py to your Twitter/ and parent src/ folders.
Then you'll need to import the path as Twitter.py doesn't know about any parent structure in this setup. You can then import from the utils module which is in the path.
import sys
sys.path.append('..')
from utils.magic_eden import MagicEden
assuming MagicEden is a class in your magic_eden.py file.

Adding python module from different folder

I have my application folder with my main.py and a lib folder with some modules:
LeapMotionInterpreter --> DataExtractor.py
lib --> Leap.py
I've already looked at how to do this and they all say to use sys.path.insert or sys.path.append but nothing seems to work, python doesn't even seem to know there is a lib folder there even though printing the sys.path does show the new added path to lib.
Here is the code I have tried:
1.
import sys
sys.path.insert(0, 'C:/Users/winba/Desktop/TYP/LeapMotionInterpreter/lib')
print (sys.path)
outputs:
['C:/Users/winba/Desktop/TYP/LeapMotionInterpreter/lib', 'C:\\Users\\winba\\Desktop\\TYP\\LeapMotionInterpreter'
import sys
sys.path.append('C:/Users/winba/Desktop/TYP/LeapMotionInterpreter/lib')
print (sys.path)
This one just outputs the lib path at the end of the array, I have also tried writing the paths with single \ or double \ instead of / but all of these just give me a 'No module named 'Leap' error if I try to import it and doesn't even acknowledge the existence of the lib folder. Also, putting my module in the same file seems to work as python is able to find the Leap.py file instantly however this is going to be a fairly big project and I wanted to keep it neat using folders so any help would be appreciated. Thank you.

If I'm understanding correctly, the directory you're targeting is a sub-directory of the current directory. You can add the file to the Python path at runtime by giving it it's relative path:
# DataExtractor.py
import sys
sys.path.insert(1, 'lib')
import Leap
This should solve the problem

Importing python libraries from outside directory

I am trying to import python files which have been developed. Initially, when all files were in the same directory then I there was no problems with them. However, I decided to split projects to dedicated directories. After that, I am unable to import files anymore.
My file structure is similar to the following:
dirFoo\
mainFoo1.py
dirFoo\
mainFoo2.py
dirFooCommon\
commonFoo.py
commonFoo2.py
Initially, I was trying to change the import path of my mainFoo1.py and do:
from dirFooCommon import commonFoo. However, that approach gave me the following error:
ModuleNotFoundError: No module named 'common'.
Apart from that, I tried to use imp.load_source which seems that is working. Unfortunately, this is generating so additional problems. For example what if some of the commonFoo.py libraries must include others? In this case, each of them needs to use an absolute project path, which will be a problem when I will try to use common libraries in any other place.
Do you have an idea what should I do to include my common libraries to my main projects?

You can use this folder structure that will allow you to call each package and module properly.
dirFoo\ <=== This is a package know
__init__.py
mainFoo1.py
dirFoo2\ <==== Change the name or you will have namespace issue
__init__.py
mainFoo2.py
dirFooCommon\ <=== This is a package know
__init__.py
commonFoo.py
commonFoo2.py
So in mainFoo1.pyyou can call commonFoo.pylike this
import sys
sys.path.append("/path/to/dirFooCommon")
from dirFooCommon import commonFoo
To replace sys.path.appendyou can also add the folder in your PYTHONPATH.

How Do I Import the Redis Python Module?

I ran my project and received the following error:
File "/home/nguyentv/schoollink/web/views/apis.py", line 10, in <module>
from util.redis.redis_client import Redis
ImportError: No module named util.redis.redis_client
How do I properly import this library?

The Module Search Path
When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:
the directory containing the input script (or the current directory). PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
the installation-dependent default.
After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended. See section Standard Modules for more information.
Basically, the interpreter is going to perform a lookup in your current working directory, then it will search through the system defined library directories.
The issue you are facing could be that your code is looking for a module that does not exist, you are calling the script from an incorrect directory, or the sys.path is setup incorrectly.
I could help more if you showed how you instantiated the interpreter, pwd output, and tree output.

You are trying to import Redis from a package named util. Unless this package is part of your application, it does not exist.
According to python-redis' documentation, here is how to import it:
import redis
# then use redis.Redis(...)
or, equivalently:
from redis import Redis
# then use Redis(...)