Short:
How can I import a module if a different module with the same name has already been imported? I don't need the old module anymore, but I can't import the new one under a different name.
importlib.reload() doesn't have the desired effect and apparently you really shouldn't mess with sys.modules
Context:
I'm automating a testing workflow by writing a script (A). The test script is provided and can not be altered.
The target of the Testing is a class in a script (B) of which there are different versions in subfolders. Unfortunately Script B and the class always have the same name, nothing I can do about that.
main_folder
├──script_a.py
├──test_script.py
│
├──version_1
│ ├──script_b.py
│
├──version_2
│ ├──script_b.py
│
├──version_3
│ ├──script_b.py
The test script imports the object and runs tests on it.
# test_script
from script_b import my_class
# creat instance of my_class
# and test it
script_a iterates over the version folders and runs the test script on script_b.
In each iteration one subfolder is added to the import path so the test script will find the corresponding script_b. The path is removed after the iteration.
If sys.modules already contains a version of test_script,
# script_a
import sys
import os
import importlib
folders = ['version_1', 'version_2', 'version_3']
folders = [os.getcwd() + '/' + folder for folder in folders]
for folder in folders:
sys.path.append(folder)
if 'test_script' in sys.modules:
importlib.reload(sys.modules['test_script'])
else:
importlib.import_module('test_script')
sys.path.remove(folder)
Issue:
It seems reload has no effect on script_b, which is imported by test_script. So although I change the import path to different subfolders test_script always runs on version 1.
How can I make test_script use the different versions of script_b without altering test_script itself?
Follow up:
Although the original question is answered, I was wondering, how is this solution from a design perspective? Is there a better / more elegant way to automate this testing process?
From what I found, it isn't considered good practice to reload modules.
Figured it out while writing the question. Hope it will help someone some day
Even if you can't alter test_script (or script_b) there's a work around.
Because an import statement does nothing if the relevant module is already imported we can reload script_b from the path we want directly in script_a. Since it then is replaced with the new version in sys.modules the import statement in test_script will cause no problem.
Updated code:
# script_a
import sys
import os
import importlib
folders = ['version_1', 'version_2', 'version_3']
folders = [os.getcwd() + '/' + folder for folder in folders]
for folder in folders:
sys.path.append(folder)
if 'test_script' in sys.modules:
importlib.reload(sys.modules['script_b']) # this line added
importlib.reload(sys.modules['test_script'])
else:
importlib.import_module('test_script')
sys.path.remove(folder)
Related
The project has the same structure as in the picture: I'm trying to import from "mod.py " in "index.py "
from .. import mod
However, it gives the error: "ImportError: attempted relative import with no known parent package" If you use this option:
from pack1 import mod
Then error: "ModuleNotFoundError error: there is no module named 'pack1'"
enter image description here
PROJECT/
pack1/
__init__.py
mod.py
pack2/
__init__.py
index.py
What is the problem?
This is a recurring question on StackOverflow. And much of the confusion (in my opinion) comes from how Python interprets the files and folders it sees is based on where Python is run from. First, some terminology:
module: a file containing Python code.
package: a folder containing files with Python code and other folders.
When you start Python in a directory (folder), it doesn't "know" what the namespace of that directory should be. I.e., if you are working in Z:\path\to_my\project\ when you start Python:
it does NOT consider project to be a package.
any .py files you want to import from will be in their own namespace as modules.
any folders you want to import from will also be in their own namespace as packages.
What about __init__.py? Since version 3.3, Python has implicit namespace packages, which allows importing without needing to create an empty __init__.py file.
Consider #2: if you have two files: first.py and second.py:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
with these contents:
# first.py
first_var = 'hello'
# second.py
from .first import first_var
second_var = first_var + ' world'
if you try to import like this:
>>> import second
Python basically does the following:
"ok, I see second.py"
"Reading that in as a module, chief!"
"Ok, it wants to import .first
"The . means get the package (folder) that contains first.py"
"Wait, I don't have a parent package for first.py!"
"Better raise an error."
The same rules apply for #3 as well. If we add a few packages to the project like this:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
with the following contents:
# pack1/mod.py
mod_var = 1234
# pack1/other_mod.py
from .mod import mod_var
other_var = mod_var * 10
# pack2/index.py
from ..pack1 import mod
and when you try to import like this:
>>> from pack2 import index.py
The import in pack2/index.py is going to fail for the same reason second.py, Python will work its way up the import chain of dots like this:
"Reading in in index.py as a module."
"Looks like it wants to import mod from ..pack1.
"Ok, . is the pack2 parent package namespace of index.py, found that."
"So, .. is the parent package of pack2."
"But, I don't have a parent package for pack2!"
"Better raise an error."
How do we make it work? Two thing.
First, move where Python is running up one level so that all of the .py files and subfolders are considered to be part of the same package namespace, which allows the file to reference each other using relative references.
path/
to_my/
>>Python is running here now<<
project/
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
So now Python sees project as a package namespace, and all of the files within can use relative references up to that level.
This changes how you import when you are in the Python interpreter:
>>> from project.pack2 import index.py
Second, you make explicit references instead of relative references. That can make the import statements really long, but if you have several top-level modules that need to pull from one another, this is how you can do it. This is useful when you are defining your functions in one file and writing your script in another.
# first.py
first_var = 'hello'
# second.py
from first import first_var # we dropped the dot
second_var = first_var + ' world'
I hope this helps clear up some of the confusion about relative imports.
python project files hierarchy:
parent/
__init__.py
one/
__init__.py
bar.py
two/
__init__.py
foo.py
foo.py
from one import bar
I tried to run foo.py from terminal in other directory (e.g. users/user), I got the next error:
No module named one
When I trying to run foo.py, I guess it is trying to import the files from the directory that the code had been executed from, I had tried lot of ways and I couldn't find solution, finally I found a solution, the problem with this solution is that the solution is not elegant and I hope there is an elegant and better solution.
foo.py
from pathlib import Path
import sys
sys.path.append(str(Path(__file__).parent.parent))
sys.path.append("..")
from one import bar
This solution is not elegant because it preventing me to put all the imports in the start of the page.
The fact that you have an __init.py__ in the parent directory suggests that parent is part of your package structure and that its parent directory, whatever that might be, should be in the PATH. Therefore your import should really be:
from parent.one import bar
It can be useful for an application directory structure to have a single root. Then the __init.py__ in that single root package can be used to load modules from subpackages, but this is certainly not a requirement. If that was not your intention, then you should probably delete the __init__.py that is in parent as it is serving no purpose (and is confusing) and ensure that directory parent is in your PATH.
HOWEVER: As long as the current directory you are in when you run your program is the parent directory of the root(s) of your package structure, Python should be able to find your packages with no special action on your part because the current directory is automatically added to the path. If that is inconvenient, you can set environment variable PYTHONPATH.
So, determine whether you should be changing your import statement or not based on which directories are part of your package structure. Then you should arrange for Python to find your packages either by setting the current directory, PYTHONPATH, or sys.path to the required directory -- but do this once. If you have to set sys.path, I would do this in your main program at startup before it needs to include anything:
If foo.py is your main program, then at the top of the program I would have:
if __name__ == '__main__':
from pathlib import Path
import sys
# if your import statement is: from parent.one import bar, then:
sys.path.insert(0, str(Path(__file__).parent.parent))
"""
# if your import statement is: from one import bar, then:
sys.path.insert(0, str(Path(__file__).parent))
"""
Why don’t you let the parent act like a path provider to the child, by creating a path dictionary ? like this way :
class parent:
...
def createPathDict(self):
self.path_dict = {}
self.path_dict ['parent'] = self.parentPath
self.path_dict ['one'] = os.path.join(self.parentPath, 'one')
self.path_dict ['two'] = os.path.join(self.parentPath, 'two')
# self.path_dict ['three'] = …
# ...
From child ‘two’ you import the dictionary like this (I assume you use classes) :
class foo:
def __init__(self, parent):
self.parent = parent
def addPathsToPythDirs(self):
sys.path.insert(1, self.parent.path_dict ['one']) # better
# sys.path.insert(0, self.parent.path_dict [key])
...
In that way you could keep your imports in foo.py
Why use sys.path.append(path) instead of sys.path.insert(1, path)?
From today I have funny problem. Everything was working just fine for months and today I want to start my program as usual and during import it just exits. There is no error or nothing. I tried over console and in pycharm but result is the same. It seems problem is inside init.py but during import from projects.
import of class looks like this:
from engine.database import Database
if I try directly to use Database class from the database.py it works. What has changed, I'm using python 3.6?
EDIT:
in engine directory I have init.py like this:
# Database imports
from engine.database import Database
# Tor imports
from projects.Common.tor_network import TorBuild
from projects.Common.tor_network import Check
from projects.Common.tor_network import kill_all
if I comment imports from project it works.
Because imports from projects doesn't work. Here is other init.py files
init.py in projects looks like this:
from projects.Common import *
and inside projects/Common init.py looks like this:
from os import listdir
from os.path import abspath, dirname, isfile, join
# get location of __init__.py
init_path = abspath(__file__)
# get folder name of __init__.py
init_dir = dirname(init_path)
# get all python files
py_files = [file_name.replace(".py", "") for file_name in listdir(init_dir) \
if isfile(join(init_dir, file_name)) and ".py" in file_name and not ".pyc" in file_name]
# remove this __init__ file from the list
py_files.remove("__init__")
__all__ = py_files
EDIT2:
So source of the problem is located in projects/Common init.py.
If I replace the that code only with __all__ = ['tor_network'] it will work. But that would mean I have to install manualy everytime I add program. That is really funny thing. I worked for moths now it doesnt.
EDIT3:
Ok I found the problem... One of program inside Common was runing function (I forgot to comment later) that exits after run. I forgot on that one.
Thanks for help!
Currently have the following file hierarchy:
\package
__init__.py
run_everything.py
\subpackage
__init__.py
work.py
work1.py
work2.py
\test
__init__.py
test_work.py
test_work1.py
My first question is regarding relative imports. Suppose in \subpackage\work.py I have a function called custom_function(), and I would like to test that function in test_work.py. For some reason I can not figure out how to make this import from one module to another. Trying from .. subpackage.work1 import custom_function() does not seem to work, and yields the error Attempted relative import in non-package Is there any way to resolve this?
2)
I would like to run all test files from run_everything.py with one function, would adding a suite() function in each test_work*.py file, which adds each unit_testing class to suite.addTest(unittest.makeSuite(TestClass)), and finally importing them into the top-level run_everything.py be the most conventional way in Python2.7?
Here is a hack*
Insert the path's to "subpackage" and "test" to your python path in run_everything using:
import sys
sys.path.insert(0, '/path/to/package/subpackage')
sys.path.insert(0, '/path/to/package/test')
And then, you can import all your files using vanilla imports in run_everything:
import work, work1, work2
import test_work, test_work1
*This won't permanently affect your PYTHONPATH.
I have a file structure like this:
dir_a
__init__.py
mod_1.py
mod_2.py
dir_b
__init__.py
my_mod.py
I want to dynamically import every module in dir_a in my_mod.py, and in order to do that, I have the following inside the __init__.py of dir_a:
import os
import glob
__all__ = [os.path.basename(f)[:-3] for f in glob.glob(os.path.dirname(os.path.abspath(__file__)) + "/*.py")]
But when I do:
from dir_a import *
I get ImportError: No module named dir_a
I wonder what causes this, is it because the parent directory containing dir_a and dir_b has to be in PYTHONPATH? In addition, the above approach is dynamic only in the sense that my_mod.py picks up everything when it is run, but if a new module is added to dir_a during its run this new module won't be picked up. So is it possible to implement a truly dynamic importing mechanism?
The answer to the first part of your question is a trivial yes: you cannot import dir_a when the directory containing dir_a isn't in the search path.
For the second part, I don't believe that is possible in general - you could listen for 'file created' OS signals in dir_a, see whether the created file is a .py, and add it to dir_a.__all__ - but that is complicated, probably expensive, and doesn't retroactively add the new thing to the global namespace, since from foo import * only sees what is in foo.__all__ at import time. And changing that would be error-prone - it allows your namespace to change at any time based on an unpredictable external event. Say you do this:
from dir_a import *
bar = 5
And then a bar.py gets added to dir_a. You can't write a "truly dynamic importer" without considering that situation, and deciding what you want to have happen.