Develop Reference

How can I lazily import a module in python?

I have classes which require dependencies in order to be instantiated but are otherwise optional. I'd like to lazily import the dependencies and fail to instantiate the class if they aren't available. Note that these dependencies are not required at the package level (otherwise they'd be mandatory via setuptools). I currently have something like this:
class Foo:
def __init__(self):
try:
import module
except ImportError:
raise ModuleNotFoundError("...")
def foo(self):
import module
Because this try/except pattern is common, I'd like to abstract it into a lazy importer. Ideally if module is available, I won't need to import it again in Foo.foo so I'd like module to be available once it's been imported in __init__. I've tried the following, which populates globals() and fails to instantiate the class if numpy isn't available, but it pollutes the global namespace.
def lazy_import(name, as_=None):
# Doesn't handle error_msg well yet
import importlib
mod = importlib.import_module(name)
if as_ is not None:
name = as_
# yuck...
globals()[name] = mod
class NeedsNumpyFoo:
def __init__(self):
lazy_import("numpy", as_="np")
def foo(self):
return np.array([1,2,])
I could instantiate the module outside the class and point to the imported module if import doesn't fail, but that is the same as the globals() approach. Alternatively lazy_import could return the mod and I could call it whenever the module is needed, but this is tantamount to just importing it everywhere as before.
Is there a better way to handle this?

Pandas actually has a function import_optional_dependency which may make a good example (link GitHub) as used in SQLAlchemyEngine (link GitHub)
However, this is only used during class __init__ to get a meaningful error (raise ImportError(...) by default!) or warn about absence or old dependencies (which is likely a more practical use of it, as older or newer dependencies may import correctly anywhere if they exist, but not work or be explicitly tested against or even be an accidental local import)
I'd consider doing similarly, and either not bother to have special handling or only do it in the __init__ (and then perhaps only for a few cases where you're interested in the version, etc.) and otherwise simply import where needed
class Foo():
def __init__(self, ...):
import bar # only tests for existence
def usebar(self, value):
import bar
bar.baz(value)
Plausibly you could assign to a property of the class, but this may cause some trouble or confusion (as the import should already be available in globals once imported)
class Foo():
def __init__(self, ...):
import bar
self.bar = bar
def usebar(self, value):
self.bar.baz(value)

Gave it a quick test with a wrapper, seems to work fine:
def requires_math(fn):
def wrapper(*args, **kwargs):
global math
try:
math
except NameError:
import math
return fn(*args, **kwargs)
return wrapper
#requires_math
def func():
return math.ceil(5.5)
print(func())
Edit: More advanced one that works with any module, and ensures it is a module in case it's been set to something else.
from types import ModuleType
def requires_import(*mods):
def decorator(fn):
def wrapper(*args, **kwargs):
for mod in mods:
if mod not in globals() or not isinstance(globals()[mod], ModuleType):
globals()[mod] = __import__(mod)
return fn(*args, **kwargs)
return wrapper
return decorator
#requires_import('math', 'random')
def func():
return math.ceil(random.uniform(0, 10))
print(func())

Python unittest: Unable to mock imported functions so that conditional evaluates to False

I'm encountering a problem with unit testing in Python. Specifically, when I try to mock a function my code imports, variables assigned to the output of that function get assigned to a MagicMock object instead of the mock-function's return_value. I've been digging through the docs for python's unittest library, but am not having any luck.
The following is the code I want to test:
from production_class import function_A, function_B, function_M
class MyClass:
def do_something(self):
variable = functionB()
if variable:
do_other_stuff()
else:
do_something_else
this is what I've tried:
#mock.patch(path.to.MyClass.functionB)
#mock.patch(<other dependencies in MyClass>)
def test_do_something(self, functionB_mock):
functionB_mock.return_value = None # or False, or 'foo' or whatever.
myClass = MyClass()
myClass.do_something()
self.assertTrue(else_block_was_executed)
The issue I have is that when the test gets to variable = functionB in MyClass, the variable doesn't get set to my return value; it gets set to a MagicMock object (and so the if-statement always evaluates to True). How do I mock an imported function such that when executed, variables actually get set to the return value and not the MagicMock object itself?

We'd have to see what import path you're actually using with path.to.MyClass.functionB. When mocking objects, you don't necessarily use the path directly to where the object is located, but the one that the intepreter sees when recursively importing modules.
For example, if your test imports MyClass from myclass.py, and that file imports functionB from production_class.py, the mock path would be myclass.functionB, instead of production_class.functionB.
Then there's the issue that you need additional mocks of MyClass.do_other_stuff and MyClass.do_something_else in to check whether MyClass called the correct downstream method, based on the return value of functionB.
Here's a working example that tests both possible return values of functionB, and whether they call the correct downstream method:
myclass.py
from production_class import functionA, functionB, functionM
class MyClass:
def do_something(self):
variable = functionB()
if variable:
self.do_other_stuff()
else:
self.do_something_else()
def do_other_stuff(self):
pass
def do_something_else(self):
pass
production_class.py
import random
def functionA():
pass
def functionB():
return random.choice([True, False])
def functionM():
pass
test_myclass.py
import unittest
from unittest.mock import patch
from myclass import MyClass
class MyTest(unittest.TestCase):
#patch('myclass.functionB')
#patch('myclass.MyClass.do_something_else')
def test_do_something_calls_do_something_else(self, do_something_else_mock, functionB_mock):
functionB_mock.return_value = False
instance = MyClass()
instance.do_something()
do_something_else_mock.assert_called()
#patch('myclass.functionB')
#patch('myclass.MyClass.do_other_stuff')
def test_do_something_calls_do_other_stuff(self, do_other_stuff_mock, functionB_mock):
functionB_mock.return_value = True
instance = MyClass()
instance.do_something()
do_other_stuff_mock.assert_called()
if __name__ == '__main__':
unittest.main()
calling python test_myclass.py results in:
..
----------------------------------------------------------------------
Ran 2 tests in 0.002s
OK

What I wound up doing was changing the import statements in MyClass to import the object instead of the individual methods. I was then able to mock the object without any trouble.
More explicitly I changed MyClass to look like this:
import production_class as production_class
class MyClass:
def do_something(self):
variable = production_class.functionB()
if variable:
do_other_stuff()
else:
do_something_else
and changed my test to
#mock.patch(path.to.MyClass.production_class)
def test_do_something(self, prod_class_mock):
prod_class_mock.functionB.return_value = None
myClass = MyClass()
myClass.do_something()
self.assertTrue(else_block_was_executed)

Sharing a global object across files with circular dependencies

I have a global object which uses a class, and that class uses the global object. How can i put the imports correct in a clean way
I now have:
run.py (The file i run)
from global_class import Global_class
global_object = Global_class()
global_object.create_some_object()
global_class.py
from some_class import Some_class
class Global_class:
def __init__(self):
self.name = 'my_name'
def create_some_object(self):
self.some_object = Some_class()
some_class.py
class Some_class:
def __init__(self):
print(global_object.name)
How can i now access global_object in Some_class? If i put:
from run import global_object
It creates a circular dependency, and it crashes. A possible method i thought of was putting the some_class import in the Global_class::create_some_object() method, but it seems to me as unclean code. Is there any better way

Any python import module or from module import Class statement runs corresponding module line by line and loads all the objects in the module namespace in the memory. However each of the names in the modules reside separately (that is the purpose of the modules after all). So a global_object in some_class.py is completely separate from the global_object in run.py. When the interpreter sees this name in some_class.py it will look in the local and global namespace (using the LEGB rule which stands for local, enclosed, global and builtins). But there is no reference to global_object exists here, it does in the calling module. Your suggestion of putting the some_class import statement inside the method create_object() will also not work for the same reason. As you have found out you cannot import global_object in some_class as it will again need to run the run.py thus creating a loop.
If you want to maintain this setup then one option would be to explicitly pass the global_object which is self in Global_class to the Some_class() constructor like below
#run.py
from global_class import Global_class
global_object = Global_class()
global_object.create_some_object()
#global_class.py
from some_class import Some_class
class Global_class:
def __init__(self):
self.name = 'my_name'
def create_some_object(self):
self.some_object = Some_class(self) #self -> global_object
#some_class.py
class Some_class:
def __init__(self, global_object):
print(global_object.name)
$ python run.py
my_name

Use only class name without namespace in isinstance

This works in a script to recognise if a is of class myproject.aa.RefClass
isinstance(a, myproject.aa.RefClass)
But how could I do it so I do not have to specify the full namespace ? I would like to be able to type:
isinstance(a, RefClass)
How is this done in Python ?
EDIT: let me give more details.
In module aa.referencedatatable.py:
class ReferenceDataTable(object):
def __init__(self, name):
self.name = name
def __call__(self, f):
self._myfn = f
return self
def referencedatatable_from_tag(tag):
import definitions
defn_lst = [definitions]
for defn in defn_lst:
referencedatatable_instance_lst = [getattr(defn, a) for a in dir(defn) if isinstance(getattr(defn, a), ReferenceDataTable)]
for referencedatatable_instance in referencedatatable_instance_lst
if referencedatatable_instance.name == tag
return referencedatatable_instance
raise("could not find")
def main()
referencedata_from_tag("Example")
In module aa.definitions.py:
from aa.referencedatatable import ReferenceDataTable
#ReferenceDataTable("Example")
def EXAMPLE():
raise NotImplementedError("not written")
For some reason calling the main from aa.referencedatatable.py will throw as it will not be able to recognise the instance of the class. But if I copy this main in another module it will work:
import aa.referencedatatable
a = aa.referencedatatable.referencedatatable_from_tag("Example")
print a
This second example works, for some reason calling this function inside the same module where the class is declared does not.

The 'namespace' is just a module object, and so is the class. You can always assign the class to a different name:
RefClass = myproject.aa.RefClass
or better yet, import it directly into your own namespace:
from myproject.aa import RefClass
Either way, now you have a global name RefClass that references the class object, so you can do:
isinstance(a, RefClass)

Correct use of static methods

I have following simplified class named Password.py in folder1:
import random
CHARS = "ABC"
class PasswordHelper(object):
#staticmethod
def generateChars(length):
return ''.join(random.choice(CHARS) for x in range(length))
Now I have another class TestClass.py in folder2:
sys.path.append('../folder1/')
import Password
class Tester:
def whatever(self):
print Password.generateChars(3)
def main():
x = Tester()
x.whatever()
# call main method
main()
When calling python TestClass.py I get the following error: AttributeError: 'module' object has no attribute 'generateChars'. Both folders are on the same level. Is there a problem with the way I import the class files or with the static method declaration itself?

Python is not Java.
Firstly, there is absolutely no point to either the Tester or the Password classes. If you're not storing state, then don't define a class. Make both whatever and generateChars into normal standalone functions.
However, assuming you're doing this just to learn about Python classes, you have not understood that a class does not equal a module in Python. Since you've imported the Password module, you still need to refer to the PasswordHelper class:
Password.PasswordHelper.generateChars(3)
Alternatively, you can import the PasswordHelper class:
from Password import PasswordHelper
...
PasswordHelper.generateChars(3)
Finally, please follow PEP8 for your module, class and function names.

You defined the function in a class, so you need to reference it with the classname too:
print Password.PasswordHelper.generateChars(3)
Alternatively, move the function out of the class definition, at which point you do not need to use #staticmethod at all:
import random
CHARS = "ABC"
def generateChars(length):
return ''.join(random.choice(CHARS) for x in range(length))
In Python, functions do not have to be part of a class definition.