Develop Reference

How does the 'from' style import treat the imported module's dependencies?

I'm trying to figure out the python import system aliases the dependencies of a module imported using the 'from' keyword.
In particular, my use-case is writing unittest.mock.patch statements for mocking attributes of an object that is imported using "from", and that object uses something imported from another module. However, more broadly I just want to understand how the import system works.
My question may make more sense with an example:
Assume we have some class A in some module a, that uses a class from module b
a.py:
from b import B
class A:
def __init__(self):
B.use_some_static_method()
# OR
self.b_instance = B()
Then, for example, in some test code we want to mock B when testing our A object. How would I write a patch statement?
test_a.py:
from a import A
def test_a(mocker):
# Mock with pytest-mock's mocker
mock_b = mocker.patch('<some path>')
a = A()
a.do_something_with_B()
What path would I put in place of ? I realize that I could simply use import a instead, and then mock a.B, but I'm more interested in understanding the import system and how this works and why it works that way.

unify python script keeping specific parts

I want to unify two slightly different scripts.
My idea was to keep the common part in a file (modX.py) and create two other files to gives two different entry points (A.py and B.py). The common part will be called through an 'import'.
from modX import *
Now, I don't see how I can have specific parts in modX. One idea was to test a variable having different values in A.py and B.py.
In modX.py :
if 'is_A' in globals():
my_string = "spam"
else:
my_string = "eggs"
A.py :
is_A = True
from modX import *
print("I love {}".format(my_string))
How can my_string get "spam" ?
Altough putting all the common part in a function can be more pythonic, I would avoid refactoring modX.py too much if I can.

I've never figured out a way to pass arguments to a module on import — although it would be very useful. However there are ways to work around the limitation which make use of the fact that modules objects are cached in the sys.modules dictionary when they're first imported and can be replaced with an instance of a class. Note that attributes assigned to the class instance (self) effectively become the module's attributes after it's stored in sys.modules.
Here's how that could be used in your example:
modX.py
import sys
class MyModule(object):
def __init__(self, arg=None):
if arg == 'is_A':
self.my_string = 'spam'
else:
self.my_string = 'eggs'
def called_by(arg): # nested function - no self
import sys
# Replace module entry with new instance of MyModule
sys.modules[__name__] = MyModule(arg)
self.called_by = called_by
# Replace module entry in sys.modules[__name__] with a default instance of
# MyModule (and create an additional reference to original module so it's not
# deleted)
_ref, sys.modules[__name__] = sys.modules[__name__], MyModule()
del sys # clean-up namespace (optional)
A.py
from modX import *
called_by('is_A') # changes modX
from modX import * # do it again to get modified version
print("I love {}".format(my_string)) # -> I love spam
B.py
from modX import *
print("I love {}".format(my_string)) # -> I love eggs

The reason that it is not work is because modX.globals() is different object the A.globals()
You do it like that:
In modX.py :
def foo():
if 'is_A' in globals():
return "spam"
else:
return "eggs"
A.py :
from modX import *
modX.is_A = True
print("I love {}".format(foo()))
You can also try to use at inspect module, maybe you find the way with that. Tell us if it help you.

Why can't Python's import work like C's #include?

I've literally been trying to understand Python imports for about a year now, and I've all but given up programming in Python because it just seems too obfuscated. I come from a C background, and I assumed that import worked like #include, yet if I try to import something, I invariably get errors.
If I have two files like this:
foo.py:
a = 1
bar.py:
import foo
print foo.a
input()
WHY do I need to reference the module name? Why not just be able to write import foo, print a? What is the point of this confusion? Why not just run the code and have stuff defined for you as if you wrote it in one big file? Why can't it work like C's #include directive where it basically copies and pastes your code? I don't have import problems in C.

To do what you want, you can use (not recommended, read further for explanation):
from foo import *
This will import everything to your current namespace, and you will be able to call print a.
However, the issue with this approach is the following. Consider the case when you have two modules, moduleA and moduleB, each having a function named GetSomeValue().
When you do:
from moduleA import *
from moduleB import *
you have a namespace resolution issue*, because what function are you actually calling with GetSomeValue(), the moduleA.GetSomeValue() or the moduleB.GetSomeValue()?
In addition to this, you can use the Import As feature:
from moduleA import GetSomeValue as AGetSomeValue
from moduleB import GetSomeValue as BGetSomeValue
Or
import moduleA.GetSomeValue as AGetSomeValue
import moduleB.GetSomeValue as BGetSomeValue
This approach resolves the conflict manually.
I am sure you can appreciate from these examples the need for explicit referencing.
* Python has its namespace resolution mechanisms, this is just a simplification for the purpose of the explanation.

Imagine you have your a function in your module which chooses some object from a list:
def choice(somelist):
...
Now imagine further that, either in that function or elsewhere in your module, you are using randint from the random library:
a = randint(1, x)
Therefore we
import random
You suggestion, that this does what is now accessed by from random import *, means that we now have two different functions called choice, as random includes one too. Only one will be accessible, but you have introduced ambiguity as to what choice() actually refers to elsewhere in your code.
This is why it is bad practice to import everything; either import what you need:
from random import randint
...
a = randint(1, x)
or the whole module:
import random
...
a = random.randint(1, x)
This has two benefits:
You minimise the risks of overlapping names (now and in future additions to your imported modules); and
When someone else reads your code, they can easily see where external functions come from.

There are a few good reasons. The module provides a sort of namespace for the objects in it, which allows you to use simple names without fear of collisions -- coming from a C background you have surely seen libraries with long, ugly function names to avoid colliding with anybody else.
Also, modules themselves are also objects. When a module is imported in more than one place in a python program, each actually gets the same reference. That way, changing foo.a changes it for everybody, not just the local module. This is in contrast to C where including a header is basically a copy+paste operation into the source file (obviously you can still share variables, but the mechanism is a bit different).
As mentioned, you can say from foo import * or better from foo import a, but understand that the underlying behavior is actually different, because you are taking a and binding it to your local module.
If you use something often, you can always use the from syntax to import it directly, or you can rename the module to something shorter, for example
import itertools as it

When you do import foo, a new module is created inside the current namespace named foo.
So, to use anything inside foo; you have to address it via the module.
However, if you use from from foo import something, you don't have use to prepend the module name, since it will load something from the module and assign to it the name something. (Not a recommended practice)

import importlib
# works like C's #include, you always call it with include(<path>, __name__)
def include(file, module_name):
spec = importlib.util.spec_from_file_location(module_name, file)
mod = importlib.util.module_from_spec(spec)
# spec.loader.exec_module(mod)
o = spec.loader.get_code(module_name)
exec(o, globals())
For example:
#### file a.py ####
a = 1
#### file b.py ####
b = 2
if __name__ == "__main__":
print("Hi, this is b.py")
#### file main.py ####
# assuming you have `include` in scope
include("a.py", __name__)
print(a)
include("b.py", __name__)
print(b)
the output will be:
1
Hi, this is b.py
2

How to mock an import

Module A includes import B at its top. However under test conditions I'd like to mock B in A (mock A.B) and completely refrain from importing B.
In fact, B isn't installed in the test environment on purpose.
A is the unit under test. I have to import A with all its functionality. B is the module I need to mock. But how can I mock B within A and stop A from importing the real B, if the first thing A does is import B?
(The reason B isn't installed is that I use pypy for quick testing and unfortunately B isn't compatible with pypy yet.)
How could this be done?

You can assign to sys.modules['B'] before importing A to get what you want:
test.py:
import sys
sys.modules['B'] = __import__('mock_B')
import A
print(A.B.__name__)
A.py:
import B
Note B.py does not exist, but when running test.py no error is returned and print(A.B.__name__) prints mock_B. You still have to create a mock_B.py where you mock B's actual functions/variables/etc. Or you can just assign a Mock() directly:
test.py:
import sys
sys.modules['B'] = Mock()
import A

The builtin __import__ can be mocked with the 'mock' library for more control:
# Store original __import__
orig_import = __import__
# This will be the B module
b_mock = mock.Mock()
def import_mock(name, *args):
if name == 'B':
return b_mock
return orig_import(name, *args)
with mock.patch('__builtin__.__import__', side_effect=import_mock):
import A
Say A looks like:
import B
def a():
return B.func()
A.a() returns b_mock.func() which can be mocked also.
b_mock.func.return_value = 'spam'
A.a() # returns 'spam'
Note for Python 3:
As stated in the changelog for 3.0, __builtin__ is now named builtins:
Renamed module __builtin__ to builtins (removing the underscores, adding an ‘s’).
The code in this answer works fine if you replace __builtin__ by builtins for Python 3.

How to mock an import, (mock A.B)?
Module A includes import B at its top.
Easy, just mock the library in sys.modules before it gets imported:
if wrong_platform():
sys.modules['B'] = mock.MagicMock()
and then, so long as A doesn't rely on specific types of data being returned from B's objects:
import A
should just work.
You can also mock import A.B:
This works even if you have submodules, but you'll want to mock each module. Say you have this:
from foo import This, That, andTheOtherThing
from foo.bar import Yada, YadaYada
from foo.baz import Blah, getBlah, boink
To mock, simply do the below before the module that contains the above is imported:
sys.modules['foo'] = MagicMock()
sys.modules['foo.bar'] = MagicMock()
sys.modules['foo.baz'] = MagicMock()
(My experience: I had a dependency that works on one platform, Windows, but didn't work on Linux, where we run our daily tests.
So I needed to mock the dependency for our tests. Luckily it was a black box, so I didn't need to set up a lot of interaction.)
Mocking Side Effects
Addendum: Actually, I needed to simulate a side-effect that took some time. So I needed an object's method to sleep for a second. That would work like this:
sys.modules['foo'] = MagicMock()
sys.modules['foo.bar'] = MagicMock()
sys.modules['foo.baz'] = MagicMock()
# setup the side-effect:
from time import sleep
def sleep_one(*args):
sleep(1)
# this gives us the mock objects that will be used
from foo.bar import MyObject
my_instance = MyObject()
# mock the method!
my_instance.method_that_takes_time = mock.MagicMock(side_effect=sleep_one)
And then the code takes some time to run, just like the real method.

Aaron Hall's answer works for me.
Just want to mention one important thing,
if in A.py you do
from B.C.D import E
then in test.py you must mock every module along the path, otherwise you get ImportError
sys.modules['B'] = mock.MagicMock()
sys.modules['B.C'] = mock.MagicMock()
sys.modules['B.C.D'] = mock.MagicMock()

I realize I'm a bit late to the party here, but here's a somewhat insane way to automate this with the mock library:
(here's an example usage)
import contextlib
import collections
import mock
import sys
def fake_module(**args):
return (collections.namedtuple('module', args.keys())(**args))
def get_patch_dict(dotted_module_path, module):
patch_dict = {}
module_splits = dotted_module_path.split('.')
# Add our module to the patch dict
patch_dict[dotted_module_path] = module
# We add the rest of the fake modules in backwards
while module_splits:
# This adds the next level up into the patch dict which is a fake
# module that points at the next level down
patch_dict['.'.join(module_splits[:-1])] = fake_module(
**{module_splits[-1]: patch_dict['.'.join(module_splits)]}
)
module_splits = module_splits[:-1]
return patch_dict
with mock.patch.dict(
sys.modules,
get_patch_dict('herp.derp', fake_module(foo='bar'))
):
import herp.derp
# prints bar
print herp.derp.foo
The reason this is so ridiculously complicated is when an import occurs python basically does this (take for example from herp.derp import foo)
Does sys.modules['herp'] exist? Else import it. If still not ImportError
Does sys.modules['herp.derp'] exist? Else import it. If still not ImportError
Get attribute foo of sys.modules['herp.derp']. Else ImportError
foo = sys.modules['herp.derp'].foo
There are some downsides to this hacked together solution: If something else relies on other stuff in the module path this kind of screws it over. Also this only works for stuff that is being imported inline such as
def foo():
import herp.derp
or
def foo():
__import__('herp.derp')

I found fine way to mock the imports in Python. It's Eric's Zaadi solution found here which I just use inside my Django application.
I've got class SeatInterface which is interface to Seat model class.
So inside my seat_interface module I have such an import:
from ..models import Seat
class SeatInterface(object):
(...)
I wanted to create isolated tests for SeatInterface class with mocked Seat class as FakeSeat. The problem was - how tu run tests offline, where Django application is down. I had below error:
ImproperlyConfigured: Requested setting BASE_DIR, but settings are not
configured. You must either define the environment variable
DJANGO_SETTINGS_MODULE or call settings.configure() before accessing
settings.
Ran 1 test in 0.078s
FAILED (errors=1)
The solution was:
import unittest
from mock import MagicMock, patch
class FakeSeat(object):
pass
class TestSeatInterface(unittest.TestCase):
def setUp(self):
models_mock = MagicMock()
models_mock.Seat.return_value = FakeSeat
modules = {'app.app.models': models_mock}
patch.dict('sys.modules', modules).start()
def test1(self):
from app.app.models_interface.seat_interface import SeatInterface
And then test magically runs OK :)
.
Ran 1 test in 0.002s
OK

If you do an import ModuleB you are really calling the builtin method __import__ as:
ModuleB = __import__('ModuleB', globals(), locals(), [], -1)
You could overwrite this method by importing the __builtin__ module and make a wrapper around the __builtin__.__import__method. Or you could play with the NullImporter hook from the imp module. Catching the exception and Mock your module/class in the except-block.
Pointer to the relevant docs:
docs.python.org: __import__
Accessing Import internals with the imp Module
I hope this helps. Be HIGHLY adviced that you step into the more arcane perimeters of python programming and that a) solid understanding what you really want to achieve and b)thorough understanding of the implications is important.

I know this is a fairly old question, but I have found myself returning to it a few times recently, and wanted to share a concise solution to this.
import sys
from unittest import mock
def mock_module_import(module):
"""Source: https://stackoverflow.com/a/63584866/3972558"""
def _outer_wrapper(func):
def _inner_wrapper(*args, **kwargs):
orig = sys.modules.get(module) # get the original module, if present
sys.modules[module] = mock.MagicMock() # patch it
try:
return func(*args, **kwargs)
finally:
if orig is not None: # if the module was installed, restore patch
sys.modules[module] = orig
else: # if the module never existed, remove the key
del sys.modules[module]
return _inner_wrapper
return _outer_wrapper
It works by temporarily patching the key for the module in sys.modules, and then restoring the original module after the decorated function is called. This can be used in scenarios where a package may not be installed in the testing environment, or a more complex scenario where the patched module might actually perform some of its own internal monkey-patching (which was the case I was facing).
Here's an example of use:
#mock_module_import("some_module")
def test_foo():
# use something that relies upon "some_module" here
assert True

I found myself facing a similar problem today, and I've decided to solve it a bit differently. Rather than hacking on top of Python's import machinery, you can simply add the mocked module into sys.path, and have Python prefer it over the original module.
Create the replacement module in a subdirectory, e.g.:
mkdir -p test/mocked-lib
${EDITOR} test/mocked-lib/B.py
Before A is imported, insert this directory to sys.path. I'm using pytest, so in my test/conftest.py, I've simply done:
import os.path
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "mocked-lib"))
Now, when the test suite is run, the mocked-lib subdirectory is prepended into sys.path and import A uses B from mocked-lib.

Mock Y of (from X import Y) in doctest (python)

I'm trying to create a doctest with mock of function that resides in a separate module
and that is imported as bellow
from foomodule import foo
def bar():
"""
>>> from minimock import mock
>>> mock('foo', nsdicts=(bar.func_globals,), returns=5)
>>> bar()
Called foo()
10
"""
return foo() * 2
import doctest
doctest.testmod()
foomodule.py:
def foo():
raise ValueError, "Don't call me during testing!"
This fails.
If I change import to import foomodule
and use foomodule.foo everywhere
Then it works.
But is there any solution for mocking function imported the way above?

You've just met one of the many reasons that make it best to never import object from "within" modules -- only modules themselves (possibly from within packages). We've made this rule part of our style guidelines at Google (published here) and I heartily recommend it to every Python programmer.
That being said, what you need to do is to take the foomodule.foo that you've just replaced with a mock and stick it in the current module. I don't recall enough of doctest's internal to confirm whether
>>> import foomodule
>>> foo = foomodule.foo
will suffice for that -- give it a try, and if it doesn't work, do instead
>>> import foomodule
>>> import sys
>>> sys.modules[__name__].foo = foomodule.foo
yeah, it's a mess, but the cause of that mess is that innocent-looking from foomodule import foo -- eschew that, and your life will be simpler and more productive;-).

Finally, found out that this was rather an issue of trunk version of MiniMock.
Old stable one performs as expected.