I have a circular import problem in my python code. The decorator from one module need to call a function in another module. And this function is using the decorator which is imported from previous module. The error message shows can't import name xxxx
my code structure:
__init__.py
worker1.py
worker2.py
...
and there is a decorator in init.py
def defer():
def wrapper(func):
def wrapper_f(*args, **kargs):
some logic here...
new_func = eval("%s.apply_async" % func.func_name) # I use celery to do async call here
new_func(args = args, kwargs = kargs)
wrapper_f.__name__ = func.func_name # This is to allow celery to load different task based on function name
return wrapper_f
return wrapper
In worker1.py
#celery.task
#defer
def task1(*args, **kargs):
some logic
It will have circular import in __init__.py and worker1.py, since eval("%s.apply_async" % func.func_name) need to load function in worker1, and worker1 need to load decorator in __init__. How can I solve this circular import problem?
Btw, I can't move the decorator function into worker1.py, bcz it is needed by other workers in the same folder.
Related
I have classes which require dependencies in order to be instantiated but are otherwise optional. I'd like to lazily import the dependencies and fail to instantiate the class if they aren't available. Note that these dependencies are not required at the package level (otherwise they'd be mandatory via setuptools). I currently have something like this:
class Foo:
def __init__(self):
try:
import module
except ImportError:
raise ModuleNotFoundError("...")
def foo(self):
import module
Because this try/except pattern is common, I'd like to abstract it into a lazy importer. Ideally if module is available, I won't need to import it again in Foo.foo so I'd like module to be available once it's been imported in __init__. I've tried the following, which populates globals() and fails to instantiate the class if numpy isn't available, but it pollutes the global namespace.
def lazy_import(name, as_=None):
# Doesn't handle error_msg well yet
import importlib
mod = importlib.import_module(name)
if as_ is not None:
name = as_
# yuck...
globals()[name] = mod
class NeedsNumpyFoo:
def __init__(self):
lazy_import("numpy", as_="np")
def foo(self):
return np.array([1,2,])
I could instantiate the module outside the class and point to the imported module if import doesn't fail, but that is the same as the globals() approach. Alternatively lazy_import could return the mod and I could call it whenever the module is needed, but this is tantamount to just importing it everywhere as before.
Is there a better way to handle this?
Pandas actually has a function import_optional_dependency which may make a good example (link GitHub) as used in SQLAlchemyEngine (link GitHub)
However, this is only used during class __init__ to get a meaningful error (raise ImportError(...) by default!) or warn about absence or old dependencies (which is likely a more practical use of it, as older or newer dependencies may import correctly anywhere if they exist, but not work or be explicitly tested against or even be an accidental local import)
I'd consider doing similarly, and either not bother to have special handling or only do it in the __init__ (and then perhaps only for a few cases where you're interested in the version, etc.) and otherwise simply import where needed
class Foo():
def __init__(self, ...):
import bar # only tests for existence
def usebar(self, value):
import bar
bar.baz(value)
Plausibly you could assign to a property of the class, but this may cause some trouble or confusion (as the import should already be available in globals once imported)
class Foo():
def __init__(self, ...):
import bar
self.bar = bar
def usebar(self, value):
self.bar.baz(value)
Gave it a quick test with a wrapper, seems to work fine:
def requires_math(fn):
def wrapper(*args, **kwargs):
global math
try:
math
except NameError:
import math
return fn(*args, **kwargs)
return wrapper
#requires_math
def func():
return math.ceil(5.5)
print(func())
Edit: More advanced one that works with any module, and ensures it is a module in case it's been set to something else.
from types import ModuleType
def requires_import(*mods):
def decorator(fn):
def wrapper(*args, **kwargs):
for mod in mods:
if mod not in globals() or not isinstance(globals()[mod], ModuleType):
globals()[mod] = __import__(mod)
return fn(*args, **kwargs)
return wrapper
return decorator
#requires_import('math', 'random')
def func():
return math.ceil(random.uniform(0, 10))
print(func())
I'm trying to find a way to dynamically add methods to a class through decorator.
The decorator i have look like:
def deco(target):
def decorator(function):
#wraps(function)
def wrapper(self, *args, **kwargs):
return function(*args, id=self.id, **kwargs)
setattr(target, function.__name__, wrapper)
return function
return decorator
class A:
pass
# in another module
#deco(A)
def compute(id: str):
return do_compute(id)
# in another module
#deco(A)
def compute2(id: str):
return do_compute2(id)
# **in another module**
a = A()
a.compute() # this should work
a.compute2() # this should work
My hope is the decorator should add the compute() function to class A, any object of A should have the compute() method.
However, in my test, this only works if i explicitly import compute into where an object of A is created. I think i'm missing something obvious, but don't know how to fix it. appreciate any help!
I think this will be quite simpler using a decorator implemented as a class:
class deco:
def __init__(self, cls):
self.cls = cls
def __call__(self, f):
setattr(self.cls, f.__name__, f)
return self.cls
class A:
def __init__(self, val):
self.val = val
#deco(A)
def compute(a_instance):
print(a_instance.val)
A(1).compute()
A(2).compute()
outputs
1
2
But just because you can do it does not mean you should. This can become a debugging nightmare, and will probably give a hard time to any static code analyser or linter (PyCharm for example "complains" with Unresolved attribute reference 'compute' for class 'A')
Why doesn't it work out of the box when we split it to different modules (more specifically, when compute is defined in another module)?
Assume the following:
a.py
print('importing deco and A')
class deco:
def __init__(self, cls):
self.cls = cls
def __call__(self, f):
setattr(self.cls, f.__name__, f)
return self.cls
class A:
def __init__(self, val):
self.val = val
b.py
print('defining compute')
from a import A, deco
#deco(A)
def compute(a_instance):
print(a_instance.val)
main.py
from a import A
print('running main')
A(1).compute()
A(2).compute()
If we execute main.py we get the following:
importing deco and A
running main
Traceback (most recent call last):
A(1).compute()
AttributeError: 'A' object has no attribute 'compute'
Something is missing. defining compute is not outputted. Even worse, compute is never defined, let alone getting bound to A.
Why? because nothing triggered the execution of b.py. Just because it sits there does not mean it gets executed.
We can force its execution by importing it. Feels kind of abusive to me, but it works because importing a file has a side-effect: it executes every piece of code that is not guarded by if __name__ == '__main__, much like importing a module executes its __init__.py file.
main.py
from a import A
import b
print('running main')
A(1).compute()
A(2).compute()
outputs
importing deco and A
defining compute
running main
1
2
I have a library that can use two modules; one is fast but available only on Linux and macOS, the other is slower but it's multi-platform. My solution was to make the library compatible with both and have something like the following:
try:
import fastmodule
except ImportError:
import slowmodule
Now I want to compare the timing of the library when using either module. Is there any way of masking the fastmodule without changing the source code (i.e. within a Jupyter Notebook), in an environment where both modules are installed, so that the slowmodule is used?
This is a bit of a hack, but here it goes:
You can write your own importer and register it (note that this is Python 3-specific, Python 2 had another API for this):
import sut
import functools
import importlib
import sys
def use_slow(f):
#functools.wraps(f)
def wrapped(*args, **kwargs):
ImportRaiser.use_slow = True
if 'fastmodule' in sys.modules:
del sys.modules['fastmodule'] # otherwise it will remain cached
importlib.reload(sut)
f(*args, **kwargs)
return wrapped
def use_fast(f):
#functools.wraps(f)
def wrapped(*args, **kwargs):
ImportRaiser.use_slow = False
importlib.reload(sut)
f(*args, **kwargs)
return wrapped
class ImportRaiser:
use_slow = False
def find_spec(self, fullname, path, target=None):
if fullname == 'fastmodule':
if self.use_slow:
raise ImportError()
sys.meta_path.insert(0, ImportRaiser())
#use_fast
def test_fast():
# test code
#use_slow
def test_slow():
# test code
Here, sut is your module under test, which you have to reload in order to change the behavior. I added decorators for readabilty, but this can be done by some function or in the test setup, of course.
If you use the slow version, fastmodule will raise ImportError on import and slowmodule will be used instead. In "fast" case, all works as usual.
i try to figure how to mock a function in a helper.py that use in several methods for a unit test.
I try with patch, #patch('project.helpers.function_0', new=lambda: True) but didn't work.
How is the correct way to do this?
Thank you.
Update
I have 1 function and 1 decorator that i need to override for all test set.
helpers.py
def myfunction(asd):
# ...
return asd
decorators.py
def mydecorator(func):
#wraps(func)
def _wrapped_func(asd, *args, **kwargs):
# ...
return func(asd, *args, **kwargs)
return _wrapped_func
How i resolved
I want to know how do this with mock, thank you!
test_base.py
import project.decorators
import project.helpers
def myfunction_mock(asd):
# ...
return asd
helpers.myfunction = myfunction_mock
def mydecorator_mock(func):
# ...
decorators.mydecorator = mydecorator_mock
Some key things going on here.
If possible, you should be using unittest.mock (or mock installed via pip).
You are likely patching in the wrong location.
When using mock.patch you probably want to use the new_callable keyword argument, rather than new
Suppose your production code looks something like this....
#production_module.py
from helpers import myfunction
def some_function(a):
result = myfunction(a) + 1
return result
If you wanted to test some_function from your production code, mocking myfunction your test code should patch production_module.myfunction not helpers.myfunction
So your test code may look like this
import mock
from production_module import some_function
def mock_myfunction(*args, **kwargs):
return 1
#mock.patch('production_module.myfunction', new_callable=mock_myfunction)
def test_some_function(mock_func):
result = some_function(1) # call the production function
assert mock_func.called # make sure the mock was actually used
assert result == 2
Another way of using mock.patch is as a context manager. So the patch will only apply in that context.
with mock.patch(...) as mock_func:
some_function(1) # mock_func is called here
some_function(1) # the mock is no longer in place here
I'm writing a test suite, and the code I'm testing makes excessive use of delayed module imports. So it's possible that with 5 different inputs to the same method, this may end up importing 5 additional modules. What I'd like to be able to do is set up tests so that I can assert that running the method with one input causes one import, and doesn't cause the other 4.
I had a few ideas of how to start on this, but none so far have been successful. I already have a custom importer, and I can put the logging code in the importer. But this doesn't work, because the import statements only run once. I need the log statement to be executed regardless of if the module has been previously imported. Just running del sys.modules['modname'] also doesn't work, because that runs in the test code, and I can't reload the module in the code being tested.
The next thing I tried was subclassing dict to do the monitoring, and replace sys.modules with this subclass. This subclass has a reimplemented __getitem__ method, but calling import module doesn't seem to trigger the __getitem__ call in the subclass. I also can't assign directly to sys.modules.__getitem__, because it is read-only.
Is what I'm trying to do even possible?
UPDATE
nneonneo's answer seems to only work if the implementation of logImports() is in the same module as where it is used. If I make a base test class containing this functionality, it has problems. The first is that it can't find just __import__, erroring with:
# old_import = __import__
# UnboundLocalError: local variable '__import__' referenced before assignment
When I change that to __builtin__.__import__, I another error:
myunittest.py:
import unittest
class TestCase(unittest.TestCase):
def logImports(self):
old_import = __builtins__.__import__
def __import__(*args, **kwargs):
print args, kwargs
return old_import(*args, **kwargs)
__builtins__.__import__ = __import__
test.py:
import myunittest
import unittest
class RealTest(myunittest.TestCase):
def setUp(self):
self.logImports()
def testSomething(self):
import unittest
self.assertTrue(True)
unittest.main()
# old_import = __builtins__.__import__
# AttributeError: 'dict' object has no attribute '__import__'
Try
old_import = __import__
def __import__(*args, **kwargs):
print args, kwargs
return old_import(*args, **kwargs)
__builtins__.__import__ = __import__
This overrides __import__ completely, allowing you to monitor every invocation of import.
Building on the previous answer, in Python 3, I've had success with the following.
import builtins
old_import = __import__
def importWithLog(*args, **kwargs):
print(args[0]) # This is the module name
print(args, kwargs)
return old_import(*args, **kwargs)
builtins.__import__ = importWithLog
# Rest of the code goes here.
import time
import myModule
...