Is there a way to import a package twice in the same python session, under the same name, but at different scope, in a multi-threaded environment ?
I would like to import the package, then override some of its functions to change its behavior only when used in specific class.
For instance, is it possible to achieve something like this ?
import mod
class MyClass:
mod = __import__('mod')
def __init__():
mod.function = new_function # override module function
def method():
mod.function() # call new_function
mod.function() # call original function
It might seem weird, but in this case the user deriving the class wouldn't have to change his code to use the improved package.
To import a module as a copy:
def freshimport(name):
import sys, importlib
if name in sys.modules:
del sys.modules[name]
mod = importlib.import_module(name)
sys.modules[name] = mod
return mod
Test:
import mymodule as m1
m2 = freshimport('mymodule')
assert m1.func is not m2.func
Note:
importlib.reload will not do the job, as it always "thoughtfully" updates the old module:
import importlib
import mymodule as m1
print(id(m1.func))
m2 = importlib.reload(m1)
print(id(m1.func))
print(id(m2.func))
Sample output:
139681606300944
139681606050680
139681606050680
It looks like a job for a context manager
import modul
def newfunc():
print('newfunc')
class MyClass:
def __enter__(self):
self._f = modul.func
modul.func = newfunc
return self
def __exit__(self, type, value, tb):
modul.func = self._f
def method(self):
modul.func()
modul.func()
with MyClass() as obj:
obj.method()
modul.func()
modul.func()
outputs
func
newfunc
newfunc
func
where modul.py contains
def func():
print('func')
NOTE: this solution suits single-threaded applications only (unspecified in the OP)
Related
I have classes which require dependencies in order to be instantiated but are otherwise optional. I'd like to lazily import the dependencies and fail to instantiate the class if they aren't available. Note that these dependencies are not required at the package level (otherwise they'd be mandatory via setuptools). I currently have something like this:
class Foo:
def __init__(self):
try:
import module
except ImportError:
raise ModuleNotFoundError("...")
def foo(self):
import module
Because this try/except pattern is common, I'd like to abstract it into a lazy importer. Ideally if module is available, I won't need to import it again in Foo.foo so I'd like module to be available once it's been imported in __init__. I've tried the following, which populates globals() and fails to instantiate the class if numpy isn't available, but it pollutes the global namespace.
def lazy_import(name, as_=None):
# Doesn't handle error_msg well yet
import importlib
mod = importlib.import_module(name)
if as_ is not None:
name = as_
# yuck...
globals()[name] = mod
class NeedsNumpyFoo:
def __init__(self):
lazy_import("numpy", as_="np")
def foo(self):
return np.array([1,2,])
I could instantiate the module outside the class and point to the imported module if import doesn't fail, but that is the same as the globals() approach. Alternatively lazy_import could return the mod and I could call it whenever the module is needed, but this is tantamount to just importing it everywhere as before.
Is there a better way to handle this?
Pandas actually has a function import_optional_dependency which may make a good example (link GitHub) as used in SQLAlchemyEngine (link GitHub)
However, this is only used during class __init__ to get a meaningful error (raise ImportError(...) by default!) or warn about absence or old dependencies (which is likely a more practical use of it, as older or newer dependencies may import correctly anywhere if they exist, but not work or be explicitly tested against or even be an accidental local import)
I'd consider doing similarly, and either not bother to have special handling or only do it in the __init__ (and then perhaps only for a few cases where you're interested in the version, etc.) and otherwise simply import where needed
class Foo():
def __init__(self, ...):
import bar # only tests for existence
def usebar(self, value):
import bar
bar.baz(value)
Plausibly you could assign to a property of the class, but this may cause some trouble or confusion (as the import should already be available in globals once imported)
class Foo():
def __init__(self, ...):
import bar
self.bar = bar
def usebar(self, value):
self.bar.baz(value)
Gave it a quick test with a wrapper, seems to work fine:
def requires_math(fn):
def wrapper(*args, **kwargs):
global math
try:
math
except NameError:
import math
return fn(*args, **kwargs)
return wrapper
#requires_math
def func():
return math.ceil(5.5)
print(func())
Edit: More advanced one that works with any module, and ensures it is a module in case it's been set to something else.
from types import ModuleType
def requires_import(*mods):
def decorator(fn):
def wrapper(*args, **kwargs):
for mod in mods:
if mod not in globals() or not isinstance(globals()[mod], ModuleType):
globals()[mod] = __import__(mod)
return fn(*args, **kwargs)
return wrapper
return decorator
#requires_import('math', 'random')
def func():
return math.ceil(random.uniform(0, 10))
print(func())
As part of parallellizing some existing code (with multiprocessing), I run into the situation that something similar to the class below needs to be pickled.
Starting from:
import pickle
from functools import lru_cache
class Test:
def __init__(self):
self.func = lru_cache(maxsize=None)(self._inner_func)
def _inner_func(self, x):
# In reality this will be slow-running
return x
calling
t = Test()
pickle.dumps(t)
returns
_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000190454A7AC8>: it's not the same object as __main__.Test._inner_func
which I don't really understand. By the way, I also tried a variation where the name of _inner_func was func as well, that didn't change things.
If anybody is interested, this can be solved by using getstate and setstate like this:
from functools import lru_cache
from copy import copy
class Test:
def __init__(self):
self.func = lru_cache(maxsize=None)(self._inner_func)
def _inner_func(self, x):
# In reality this will be slow-running
return x
def __getstate__(self):
result = copy(self.__dict__)
result["func"] = None
return result
def __setstate__(self, state):
self.__dict__ = state
self.func = lru_cache(maxsize=None)(self._inner_func)
As detailled in the comments, the pickle module has issues when dealing with decorators. See this question for more details:
Pickle and decorated classes (PicklingError: not the same object)
Use methodtools.lru_cache not to create a new cache function in __init__
import pickle
from methodtools import lru_cache
class Test:
#lru_cache(maxsize=None)
def func(self, x):
# In reality this will be slow-running
return x
if __name__ == '__main__':
t = Test()
print(pickle.dumps(t))
It requires to install methodtools via pypi:
pip install methodtools
I have a module in some path. I want to create a class with the same attributes as the module (so that it can be used the same way as the module) but perform some custom actions before accessing the attributes - such as reloading the module.
def get_method_with_extra(method_name, module):
def method_with_extra(self, *args):
imp.reload(module)
func_to_call = getattr(module, method_name)
func_to_call(*args)
return method_with_extra
class tester():
def __init__(self, module_path):
self.module = imp.load_source('module', module_path)
method_list = [func for func in dir(self.module) if
callable(getattr(self.module, func))]
for method_name in method_list:
method_with_extra = get_method_with_extra(method_name,
self.module)
setattr(type(self), method_name, method_with_extra)
So if for example the module has a method named "Parse", I would like an instance of tester - tess - to have it as well, and for me to be able to call tess.parse() which should reload the inner module and then call the module's parse(). Instead, I get this error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 4, in __init__
AttributeError: attribute '__call__' of 'namespace#' object is read-only
If you are allowed to change the source of the target module, and it is small enough, I think the cleanest solution here is rewriting it as a class. And then import the class, inherit from it and customize.
Also be aware that reloading modules in Python has lots of caveats, it's more for playing in Python shell than for production code.
Based on what you said in the comment, I changed your code just a little bit. I used importlib because module imp is deprecated. Note also that "monkey patching" (that's what this kind of technique is called, you want to make a runtime patch in the code) is always tightly coupled with the target code. It there are changes, your patch code can break easily.
I wrote two files module.py and test_module.py:
#-----------
# module.py
a = 100
b = 200
# returns something
def sum3(x,y):
return x + y + 3
# does something
def print_a_b():
global a
print(a,b)
a = a + 1 # test module reloads. If ok, "a" remains 100
#----------------
# test_module.py
import module
import importlib as imp
def get_method_with_extra(method_name, module):
def method_with_extra(self, *args):
imp.reload(module) # comment to see that "a" increases
func_to_call = getattr(module, method_name)
if args: # function may not have args
return func_to_call(*args)
else: # function may not have args
return func_to_call()
return method_with_extra
class tester():
def __init__(self, module_path):
self.module = imp.import_module('module', module_path)
method_list = [func for func in dir(self.module)
if callable(getattr(self.module, func))]
for method_name in method_list:
#print(method_name)
method_with_extra = \
get_method_with_extra(method_name, self.module)
setattr(type(self), method_name, method_with_extra)
t = tester('.')
print(t.sum3(1,2))
t.print_a_b()
t.print_a_b() # checking for the reload, "a" should be 100
I have a conditional import in a self-initialized instance of a superclass, but subclass cannot see the module (python 2.7):
class A(object):
def __init__(self, arg1):
self.attr1 = self.method1(arg1)
def method1(self, arg1):
if arg1 == 'foo':
import amodule
return amodule.method1()
else:
return 'not a dependency on foo'
class B(A):
def __init__(self, arg1):
super(B, self).__init__(arg1)
if arg1 == 'foo':
self.attr2 = self.method2(self.attr1)
def method2(self, attr1):
return amodule.method2()
if __name__=='__main__':
b = B("foo")
print b.attr2
This throws NameError: global name 'amodule' is not defined. a = A("foo") works just fine
Shouldn't the super call have executed import amodule in this case? (And using import should have put the module into globals?)
Doesn't import add /amodule/ to the global namespace of the currently
executing module? (__main__)?
No, the module is added to sys.modules but if it was imported locally then you won't have any references to it anymore. i.e the name amodule is now gone.
You could still access the module using sys.modules:
def method2(self, attr1):
import sys
return sys.modules['amodule'].method2()
Or you could import it using import amodule again and it will be picked up from sys.modules.
# Here b.py contains
# print('Module b was imported')
def func1():
print('inside func1')
import b
def func2():
print('inside func2')
import sys
print(sys.modules['b'])
import b
def func3():
print('inside func3')
import b
import sys
print('Deleted b')
del sys.modules['b']
import b
func1()
print()
func2()
print()
func3()
Demo:
inside func1
Module b was imported
inside func2
<module 'b' from '/Users/ashwini/py/b.py'>
inside func3
Deleted b
Module b was imported
Try putting import amodule on the first line of the program.
The reason is that amodule is imported in method1, method2 does not have access.
If you follow your code, you would see you don't reach method1().
When you create the object
b = B(foo)
You go through A.init() becuase the call for super(). However, your init of class A() doesn't include any import statements. Then the A.__init__ part is done, and you continue with B.__init__(). The next command is a call to amodule object, which wasn't imported at all.
You can add an helper method, which checks if the arg equals 'Foo' and if so import the module. Then add a call to this function in A.__init__() function.
On another note, __init__() job is to initialize variables. It shouldn't return anything.
This works in a script to recognise if a is of class myproject.aa.RefClass
isinstance(a, myproject.aa.RefClass)
But how could I do it so I do not have to specify the full namespace ? I would like to be able to type:
isinstance(a, RefClass)
How is this done in Python ?
EDIT: let me give more details.
In module aa.referencedatatable.py:
class ReferenceDataTable(object):
def __init__(self, name):
self.name = name
def __call__(self, f):
self._myfn = f
return self
def referencedatatable_from_tag(tag):
import definitions
defn_lst = [definitions]
for defn in defn_lst:
referencedatatable_instance_lst = [getattr(defn, a) for a in dir(defn) if isinstance(getattr(defn, a), ReferenceDataTable)]
for referencedatatable_instance in referencedatatable_instance_lst
if referencedatatable_instance.name == tag
return referencedatatable_instance
raise("could not find")
def main()
referencedata_from_tag("Example")
In module aa.definitions.py:
from aa.referencedatatable import ReferenceDataTable
#ReferenceDataTable("Example")
def EXAMPLE():
raise NotImplementedError("not written")
For some reason calling the main from aa.referencedatatable.py will throw as it will not be able to recognise the instance of the class. But if I copy this main in another module it will work:
import aa.referencedatatable
a = aa.referencedatatable.referencedatatable_from_tag("Example")
print a
This second example works, for some reason calling this function inside the same module where the class is declared does not.
The 'namespace' is just a module object, and so is the class. You can always assign the class to a different name:
RefClass = myproject.aa.RefClass
or better yet, import it directly into your own namespace:
from myproject.aa import RefClass
Either way, now you have a global name RefClass that references the class object, so you can do:
isinstance(a, RefClass)