Develop Reference

How can I redirect module imports with modern Python?

I am maintaining a python package in which I did some restructuring. Now, I want to support clients who still do from my_package.old_subpackage.foo import Foo instead of the new from my_package.new_subpackage.foo import Foo, without explicitly reintroducing many files that do the forwarding. (old_subpackage still exists, but no longer contains foo.py.)
I have learned that there are "loaders" and "finders", and my impression was that I should implement a loader for my purpose, but I only managed to implement a finder so far:
RENAMED_PACKAGES = {
'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}
# TODO: ideally, we would not just implement a "finder", but also a "loader"
# (using the importlib.util.module_for_loader decorator); this would enable us
# to get module contents that also pass identity checks
class RenamedFinder:
#classmethod
def find_spec(cls, fullname, path, target=None):
renamed = RENAMED_PACKAGES.get(fullname)
if renamed is not None:
sys.stderr.write(
f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
return importlib.util.find_spec(renamed)
return None
sys.meta_path.append(RenamedFinder())
https://docs.python.org/3.5/library/importlib.html#importlib.util.module_for_loader and related functionality, however, seem to be deprecated. I know it's not a very pythonic thing I am trying to achieve, but I would be glad to learn that it's achievable.

On import of your package's __init__.py, you can place whatever objects you want into sys.modules, the values you put in there will be returned by import statements:
from . import new_package
from .new_package import module1, module2
import sys
sys.modules["my_lib.old_package"] = new_package
sys.modules["my_lib.old_package.module1"] = module1
sys.modules["my_lib.old_package.module2"] = module2
If someone now uses import my_lib.old_package or import my_lib.old_package.module1 they will obtain a reference to my_lib.new_package.module1. Since the import machinery already finds the keys in the sys.modules dictionary, it never even begins looking for the old files.
If you want to avoid importing all the submodules immediately, you can emulate a bit of lazy loading by placing a module with a __getattr__ in sys.modules:
from types import ModuleType
import importlib
import sys
class LazyModule(ModuleType):
def __init__(self, name, mod_name):
super().__init__(name)
self.__mod_name = name
def __getattr__(self, attr):
if "_lazy_module" not in self.__dict__:
self._lazy_module = importlib.import(self.__mod_name, package="my_lib")
return self._lazy_module.__getattr__(attr)
sys.modules["my_lib.old_package"] = LazyModule("my_lib.old_package", "my_lib.new_package")

In the init file of the old module, have it import from the newer modules
Old (package.oldpkg):
foo = __import__("Path to new module")
New (package.newpkg):
class foo:
bar = "thing"
so
package.oldpkg.foo.bar is the same as package.newpkg.foo.bar
Hope this helps!

I think that this is what you are looking for:
RENAMED_PACKAGES = {
'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}
class RenamedFinder:
#classmethod
def find_spec(cls, fullname, path, target=None):
renamed = RENAMED_PACKAGES.get(fullname)
if renamed is not None:
sys.stderr.write(
f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
spec = importlib.util.find_spec(renamed)
spec.loader = cls
return spec
return None
#staticmethod
def create_module(spec):
return importlib.import_module(spec.name)
#staticmethod
def exec_module(module):
pass
sys.meta_path.append(RenamedFinder())
Still, IMO the approach that manipulates sys.modules is preferable as it is more readable, more explicit, and provides you much more control. It might become useful especially in further versions of your package when my_package.new_subpackage.foo starts to diverge from my_package.old_subpackage.foo while you would still need to provide the old one for backward compatibility. For that reason, you would maybe need to preserve the code of both anyway.

Consolidate all the old package names into my_package.
Old packages (old_package):
image_processing (class) Will be deleted and replaced by better_image_processing
text_recognition (class) Will be deleted and replaced by better_text_recognition
foo (variable) Will be moved to better_text_recognition
still_there (class) Will not move
New packages:
super_image_processing
better_text_recognition
Redirector (class of my_package):
class old_package:
image_processing = super_image_processing # Will be replaced
text_recognition = better_text_recognition # Will be replaced
Your main new module (my_package):
#imports here
class super_image_processing:
def its(gets,even,better):
pass
class better_text_recognition:
def now(better,than,ever):
pass
class old_package:
#Links
image_processing = super_image_processing
text_recognition = better_text_recognition
still_there = __import__("path to unchanged module")
This allows you to delete some files and keep the rest. If you want to redirect variables you would do:
class super_image_processing:
def its(gets,even,better):
pass
class better_text_recognition:
def now(better,than,ever):
pass
class old_package:
#Links
image_processing = super_image_processing
text_recognition = better_text_recognition
foo = text_recognition.foo
still_there = __import__("path to unchanged module")
Would this work?

Python how to access parent's modules and functions from child module

I have three files that all contain classes with the same names but slightly different definitions. Some methods in these classes are identical across all three files, so I abstracted them out to another file, utils.py, where they are defined within a "template" version of the original class. The problem is that these methods invoke functions and modules that exist in the original files but not this new one.
My original approach was to use multiple class inheritance, which would initialize the template class within the scope of the parent class, allowing access to all the functions and modules it requires. However, I was instructed to avoid multiple class inheritance and to simply import the utils file.
Importing does not apply the same scoping logic as mentioned above with inheritance. So here arises my problem. I have created a small example to show what I mean. I am using a module called datajoint. You don't need to know much about it except that a schema is basically a table or collection of tables in a database.
schemas.py
import datajoint as dj
from datetime import datetime
import utils
dj.conn()
schema = dj.Schema('adib_example1')
schema.drop()
schema = dj.Schema('adib_example1')
def test_print():
print("test")
#schema
class Subject(dj.Lookup):
definition = """
subject_id: int
"""
contents = [dict(subject_id=1)]
#schema
class Session(dj.Computed):
definition = """
-> Subject
time: varchar(30)
"""
def make(self, key):
utils.SessionTemplate.make(self,key)
Session.populate() # invokes Session's make(), passing Subject's primary key
Approach 1
Import scoping not working like inheritance
utils.py
class SessionTemplate():
#staticmethod
def make(table, key):
test_print() # parent function usage example
table.time = f"{datetime.now()}" # parent module usage example
new_entry = dict(**key, time=table.time)
table.insert1(new_entry)
error
Traceback (most recent call last):
File "/home/.anaconda/imported_make/schemas.py", line 30, in <module>
Session.populate() # invokes Session's make(), passing Subject's primary key
File "/opt/conda/lib/python3.9/site-packages/datajoint/autopopulate.py", line 153, in populate
make(dict(key))
File "/home/.anaconda/imported_make/schemas.py", line 28, in make
utils.SessionTemplate.make(self,key)
File "/home/.anaconda/imported_make/utils.py", line 5, in make
test_print() # parent function usage example
NameError: name 'test_print' is not defined
Approach 2
Importing schemas.py into utils.py works, but requires including schemas. before every imported function and module, which is not practical in my case.
utils.py
import schemas
class SessionTemplate():
#staticmethod
def make(table, key):
schemas.test_print() # parent function usage example
table.time = f"{schemas.datetime.now()}" # parent module usage example
new_entry = dict(**key, time=table.time)
table.insert1(new_entry)
Approach 3
Import using * to avoid having to add schemas. before each parent function/module somehow does not provide access to the parents modules and functions.
from schemas import *
class SessionTemplate():
#staticmethod
def make(table, key):
test_print() # parent function usage example
table.time = f"{datetime.now()}" # parent module usage example
new_entry = dict(**key, time=table.time)
table.insert1(new_entry)
error
Traceback (most recent call last):
File "/home/.anaconda/imported_make/run.py", line 1, in <module>
import schemas
File "/home/.anaconda/imported_make/schemas.py", line 30, in <module>
Session.populate() # invokes Session's make(), passing Subject's primary key
File "/opt/conda/lib/python3.9/site-packages/datajoint/autopopulate.py", line 153, in populate
make(dict(key))
File "/home/.anaconda/imported_make/schemas.py", line 28, in make
utils.SessionTemplate().make(self,key)
File "/home/.anaconda/imported_make/utils.py", line 7, in make
test_print() # parent function usage example
NameError: name 'test_print' is not defined
I know import * is bad practice, but it would have been fine in this instance if it worked, and I'm not sure why it doesn't.

boss.py
class tasks():
def job1(input):
// do something
return output
def job2(input):
// do something
return output
worker.py
import boss.tasks
from boss.tasks import job1, job2
input_value = "xyz"
output1 = boss.tasks().job1(input_value)
output2 = boss.tasks().job2(input_value)

Is it possible to mock os.scandir and its attributes?

for entry in os.scandir(document_dir)
if os.path.isdir(entry):
# some code goes here
else:
# else the file needs to be in a folder
file_path = entry.path.replace(os.sep, '/')
I am having trouble mocking os.scandir and the path attribute within the else statement. I am not able to mock the mock object's property I created in my unit tests.
with patch("os.scandir") as mock_scandir:
# mock_scandir.return_value = ["docs.json", ]
# mock_scandir.side_effect = ["docs.json", ]
# mock_scandir.return_value.path = PropertyMock(return_value="docs.json")
These are all the options I've tried. Any help is greatly appreciated.

It depends on what you realy need to mock. The problem is that os.scandir returns entries of type os.DirEntry. One possibility is to use your own mock DirEntry and implement only the methods that you need (in your example, only path). For your example, you also have to mock os.path.isdir. Here is a self-contained example for how you can do this:
import os
from unittest.mock import patch
def get_paths(document_dir):
# example function containing your code
paths = []
for entry in os.scandir(document_dir):
if os.path.isdir(entry):
pass
else:
# else the file needs to be in a folder
file_path = entry.path.replace(os.sep, '/')
paths.append(file_path)
return paths
class DirEntry:
def __init__(self, path):
self.path = path
def path(self):
return self.path
#patch("os.scandir")
#patch("os.path.isdir")
def test_sut(mock_isdir, mock_scandir):
mock_isdir.return_value = False
mock_scandir.return_value = [DirEntry("docs.json")]
assert get_paths("anydir") == ["docs.json"]
Depending on your actual code, you may have to do more.
If you want to patch more file system functions, you may consider to use pyfakefs instead, which patches the whole file system. This will be overkill for a single test, but can be handy for a test suite relying on file system functions.
Disclaimer: I'm a contributor to pyfakefs.

Sharing Variables across new object instances

Background:
I am writing a module in order to set-up an embedded system. In this context I need to load some modules and perform some system settings.
Context:
I have a parent class holding some general code (load the config file, build ssh connection etc.) used for several child classes. One of them is the module class that sets up the module and therefore uses among otherthings the ssh connection and the configuration file.
My goal is to share the configuration file and the connection with the next module, that will be setup. For the connection its just a waste to build and destroy it all the time, but for the configuration file, changes during setup can lead to undefined behavior.
Research/ approaches:
I tried using class variables, however they aren't passed when initaiting
a new module object.
Futher, I tried using global variables, but since the parent class and the
child classes are in different files, This won't work (Yes, i can put them
all in one file but this will be a mess) Also using a getter function from
the file where I defined the global variable didn't work.
I am aware of the 'builtin' solution from
How to make a cross-module variable?
variable, but feel this would be a bit of an overkill...
Finally, I can keep the config file and and the connection in a central
script and pass them to each of the instances, but this will lead to loads
of dependencies and I don't think it's a good solution.
So here is a bit of code with an example method, to get some file paths.
The code is set up according to approach 1 (class vaiables)
An example config file:
Files:
Core:
Backend:
- 'file_1'
- 'file_2'
Local:
File_path:
Backend: '/the/path/to'
The Parent class in setup_class.py
import os
import abc
import yaml
class setup(object):
__metaclass__ = abc.ABCMeta
configuration = []
def check_for_configuration(self, config_file):
if not self.configuration:
with open(config_file, "r") as config:
self.configuration = yaml.safe_load(config)
def get_configuration(self):
return self.configuration
def _make_file_list(self, path, names):
full_file_path = []
for file_name in names:
temp_path = path + '/' + file_name
temp_path = temp_path.split('/')
full_file_path.append(os.path.join(*temp_path))
return full_file_path
#abc.abstractmethod
def install(self):
raise NotImplementedError
The module class in module_class.py
from setup_class import setup
class module(setup):
def __init__(self, module_name, config_file = ''):
self.name = module_name
self.check_for_configuration(config_file)
def install(self):
self._backend()
def _backend(self):
files = self._make_file_list(
self.configuration['Local']['File_path']['Backend'],
self.configuration['Files'][self.name]['Backend'])
if files:
print files
And finally a test script:
from module_class import module
Analysis = module('Analysis',"./example.yml")
Core = module('Core', "./example.yml")
Core.install()
Now, when running the code, the config file is loaded everytime, a new module object is initaiated. I would like to avoid this. Are there approaches I have not considdered? What's the neatest way to achive this?

Save your global values in a global dict, and refer to that inside your module.
cache = {}
class Cache(object):
def __init__(self):
global cache
self.cache = cache

How do I read conf.py setting in Sphinx extension node?

from docutils.parsers.rst.directives.images import Figure
class MyFigure(Figure):
def run(self):
# here I need to read the 'thumbnails_folder' setting
pass
def setup(app):
app.add_config_value('thumbnails_folder', '_thumbnails', 'env')
How can I access the config value in .run()? I read sources of Sphinx-contrib, but did not see things done in my way, so they accessed conf.py the way I can't. Or should I do it in a different manner?
All I want to do is translate this
.. figure:: image/image.jpg
into this:
.. image:: image/thumbnails/image.jpg
:target: image/image.jpg
Here's the extension code
(the thumbnail is generated with PIL). And also put the :target: into downloadable files (As I see, only builder instances can do this).

The build environment holds a reference to the Config object. Configuration variables can be retrieved from this object:
def run(self):
env = self.state.document.settings.env # sphinx.environment.BuildEnvironment
config = env.config # sphinx.config.Config
folder = config["thumbnails_folder"]
...