I am attempting to unit test some Python 3 code that imports a module. Unfortunately, the way the module is written, simply importing it has unpleasant side effects, which are not important for the tests. I'm trying to use unitest.mock.patch to get around it, but not having much luck.
Here is the structure of an illustrative sample:
.
└── work
├── __init__.py
├── test_work.py
├── work.py
└── work_caller.py
__init__.py is an empty file
work.py
import os
def work_on():
path = os.getcwd()
print(f"Working on {path}")
return path
def unpleasant_side_effect():
print("I am an unpleasant side effect of importing this module")
# Note that this is called simply by importing this file
unpleasant_side_effect()
work_caller.py
from work.work import work_on
class WorkCaller:
def call_work(self):
# Do important stuff that I want to test here
# This call I don't care about in the test, but it needs to be called
work_on()
test_work.py
from unittest import TestCase, mock
from work.work_caller import WorkCaller
class TestWorkMockingModule(TestCase):
def test_workcaller(self):
with mock.patch("work.work.unpleasant_side_effect") as mocked_function:
sut = WorkCaller()
sut.call_work()
In work_caller.py I only want to test the beginning code, not the call to work_on(). When I run the test, I get the following output:
paul-> python -m unittest
I am an unpleasant side effect of importing this module
Working on /Users/paul/src/patch-test
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
I was expecting that the line I am an unpleasant side effect of importing this module would not be printed because the function unpleasant_side_effect would be mocked. Where might I be going wrong?
The unpleasant_side_effect is run for two reasons. First because the imports are handled before the test case is started and is therefore not mocked when importing is happening. Secondly, because the mocking itself imports work.py and thus runs unpleasant_side_effect even if work_caller.py was not imported.
The import problem can be solved by mocking the module work.py itself. This can either be done globally in the test module or in the testcase itself. Here I assigned it a MagicMock, which can be imported, called etc.
test_work.py
from unittest import TestCase, mock
class TestWorkMockingModule(TestCase):
def test_workcaller(self):
import sys
sys.modules['work.work'] = mock.MagicMock()
from work.work_caller import WorkCaller
sut = WorkCaller()
sut.call_work()
The downside is that work_on is also mocked, which I am not sure whether is a problem in your case.
It is not possible to not run the entire module when it is imported, since functions and classes are also statements, thus the module execution has to finish before returning to the caller, where one want to alter the imported module.
In case you asked partially about the best practice.
You should always split your code to library used by every other code and side-effect lines. And probably eliminate side-effects by calling the side-effecting code from you def main(): But if you want to keep side-effects anyway, then you could do:
work_lib.py:
...no side-effects...
work.py
from work_lib import ...
...side-effects....
test_work.py
from work_lib import ...
Another solution is to put this line ahead of any code that you don't want to run on import:
if __name__ == "__main__":
If the code is at the highest/outermost level of a module, the name will be "main" when running directly or will be the module name when being imported. So in your example, if you put that line ahead of your call to unpleasant_side_effect(), the function wouldn't get called when the module is imported.
Related
Im trying to unittests methods in fileA.py. fileA.py contains imports to firebasehandler.py where Im setting a connection to firebase. The methods I'm trying to test have no relation or need at all with anything from firebasehandler.py, but when running the tests I don't want to go through the credentials checking phase. What can I do to skip that import when running the unittests?
I recommend taking a look at unittest mocking:
https://docs.python.org/3/library/unittest.mock.html
You could for example do this in your test code:
from unittest.mock import MagicMock
sys.modules['firebasehandler'] = MagicMock()
import fileA
This should prevent loading of the actual firebasehandler module AND you can see if the mocked version was accessed etc if you wish.
I guess you can mock the imported object or method from fileA.py in your UT.
I've read this post about circular imports in Python. It describes the following scenario and argues that this raises an error when run:
# module1
import module2
def function1():
module2.function2()
def function3():
print('Goodbye, World!')
# module2
import module1
def function2():
print('Hello, World!')
module1.function3()
# __init__.py
import module1
module1.function1()
But when I run this (Python 3.95), it runs perfectly fine. The post is pretty old and it doesn't specify the Python version it uses. Maybe there was some change in latter Pythons that support this?
Here's a simplified sequence of events that happen in the code in Python 3:
__init__.py starts running
An empty __main__ module is added to sys.modules
import module1 starts loading module1.py
An empty module1 module is added to sys.modules
import module2 starts loading module2.py
An empty module2 module is added to sys.modules
module2.function2 is created and added to module2.__dict__
The fact that function2 references names in module1 does not affect the creation of the function object in any way
module2 is fully loaded and execution returns to module1
module1.function1 and module1.function3 are created and added to module1.__dict__
Again, it does not matter what names the functions reference because they are not being called. AttributeError and NameError can be raised at runtime if necessary.
module1 is fully loaded and execution returns to __main__
module1.function runs successfully, since all the names it references are resolvable.
As you can see, there are no circular import issues in this particular sequence of imports because module1 and module2 do not attempt to call each other's functions. The current import system allows both modules to load before the functions are called.
The post you mention is from 2017, and must be using a version of python from before 3.0. A hint is found in the link in the following quote, which links to the python-2.x docs:
This approach doesn't contradict Python syntax, as the Python documentation says: "It is customary but not required to place all import statements at the beginning of a module (or script, for that matter)".
The paragraph after that is a bit misleading by the way:
The Python documentation also says that it is advisable to use import X, instead of other statements, such as from module import *, or from module import a,b,c.
While star imports are certainly discouraged, specific-name imports of the form from module import a,b,c are generally very much encouraged with few exceptions.
I have two scripts:
script1:
from test import script2
if __name__ == '__main__':
foo()
script2:
def foo():
print 'hello'
In the structure:
test:
│ A
│-----B:
│ script2.py
│script1.py
I am trying to call the function foo() from script2.py in script1.py.
I received the error:
This inspection detect names that should resolve but don't. Due to
dynamic dispatch and duck typing, this is possible in a limited but
useful number of cases. Top-level and class-level itesm are supported
better than instance items.
I read number of similar cases but they didn't help me:
Call a function from another file in Python
How to import a function from parent folder in python?
Import Script from a Parent Directory
For Python packages to work, you need __init__.py files in the folders. These can be empty.
Then, you are importing from a subfolder, so you need import script2 from A.B
At that point, you may use script2.foo()
Also worth mentioning that Python3 should be targeted for any new projects .
I have an issue accessing part of imported module from the pytest.
Here is branch with code referenced below: https://github.com/asvc/snapshotr/tree/develop
In particular, when running this test, it works as expected for test_correct_installation() but test_script_name_checking() fails with AttributeError.
import main as ss
import os
class TestInit:
def test_correct_installation(self):
assert os.path.exists(ss.snapr_path)
assert os.path.isfile(ss.snapr_path + "/main/markup.py")
assert os.path.isfile(ss.snapr_path + "/main/scandir.py")
def test_script_name_checking(self):
assert ss.ssPanel.check_script('blah') is None # Here it fails
Link to the main which is being tested
What I'm trying to do is to "extract" isolated piece of code, run it with known data and compare result to some reference. Seems like extraction part doesn't work quite well, best practises for such cases would be greatly appreciated.
Traceback:
AttributeError: 'module' object has no attribute 'ssPanel'
I have tried a small hack in the test_init.py:
class dummy():
pass
nuke = dummy()
nuke.GUI = True
But it (obviously) doesn't work as nuke.GUI is being redefined in __init__.py upon every launch.
This is a quite complex situation. When you import main in test_init.py, it will import main/__init__.py and execute all the code. This will cause nuke being imported and also, if nuke.GUI is False, there will not be ssPanel, as you can see.
The problem is that, you can't fake a dummy nuke in the test script. It won't work. Because before the test is running, the real nuke was already imported.
My suggestion would be seperate ssPanel into another python file. Then in __init__.py we can do:
if nuke.GUI:
from sspanel import ssPanel
And in test scripts, we can also easily import it using:
from main.sspanel import ssPanel
Summary: when a certain python module is imported, I want to be able to intercept this action, and instead of loading the required class, I want to load another class of my choice.
Reason: I am working on some legacy code. I need to write some unit test code before I start some enhancement/refactoring. The code imports a certain module which will fail in a unit test setting, however. (Because of database server dependency)
Pseduo Code:
from LegacyDataLoader import load_me_data
...
def do_something():
data = load_me_data()
So, ideally, when python excutes the import line above in a unit test, an alternative class, says MockDataLoader, is loaded instead.
I am still using 2.4.3. I suppose there is an import hook I can manipulate
Edit
Thanks a lot for the answers so far. They are all very helpful.
One particular type of suggestion is about manipulation of PYTHONPATH. It does not work in my case. So I will elaborate my particular situation here.
The original codebase is organised in this way
./dir1/myapp/database/LegacyDataLoader.py
./dir1/myapp/database/Other.py
./dir1/myapp/database/__init__.py
./dir1/myapp/__init__.py
My goal is to enhance the Other class in the Other module. But since it is legacy code, I do not feel comfortable working on it without strapping a test suite around it first.
Now I introduce this unit test code
./unit_test/test.py
The content is simply:
from myapp.database.Other import Other
def test1():
o = Other()
o.do_something()
if __name__ == "__main__":
test1()
When the CI server runs the above test, the test fails. It is because class Other uses LegacyDataLoader, and LegacydataLoader cannot establish database connection to the db server from the CI box.
Now let's add a fake class as suggested:
./unit_test_fake/myapp/database/LegacyDataLoader.py
./unit_test_fake/myapp/database/__init__.py
./unit_test_fake/myapp/__init__.py
Modify the PYTHONPATH to
export PYTHONPATH=unit_test_fake:dir1:unit_test
Now the test fails for another reason
File "unit_test/test.py", line 1, in <module>
from myapp.database.Other import Other
ImportError: No module named Other
It has something to do with the way python resolves classes/attributes in a module
You can intercept import and from ... import statements by defining your own __import__ function and assigning it to __builtin__.__import__ (make sure to save the previous value, since your override will no doubt want to delegate to it; and you'll need to import __builtin__ to get the builtin-objects module).
For example (Py2.4 specific, since that's what you're asking about), save in aim.py the following:
import __builtin__
realimp = __builtin__.__import__
def my_import(name, globals={}, locals={}, fromlist=[]):
print 'importing', name, fromlist
return realimp(name, globals, locals, fromlist)
__builtin__.__import__ = my_import
from os import path
and now:
$ python2.4 aim.py
importing os ('path',)
So this lets you intercept any specific import request you want, and alter the imported module[s] as you wish before you return them -- see the specs here. This is the kind of "hook" you're looking for, right?
There are cleaner ways to do this, but I'll assume that you can't modify the file containing from LegacyDataLoader import load_me_data.
The simplest thing to do is probably to create a new directory called testing_shims, and create LegacyDataLoader.py file in it. In that file, define whatever fake load_me_data you like. When running the unit tests, put testing_shims into your PYTHONPATH environment variable as the first directory. Alternately, you can modify your test runner to insert testing_shims as the first value in sys.path.
This way, your file will be found when importing LegacyDataLoader, and your code will be loaded instead of the real code.
The import statement just grabs stuff from sys.modules if a matching name is found there, so the simplest thing is to make sure you insert your own module into sys.modules under the target name before anything else tries to import the real thing.
# in test code
import sys
import MockDataLoader
sys.modules['LegacyDataLoader'] = MockDataLoader
import module_under_test
There are a handful of variations on the theme, but that basic approach should work fine to do what you describe in the question. A slightly simpler approach would be this, using just a mock function to replace the one in question:
# in test code
import module_under_test
def mock_load_me_data():
# do mock stuff here
module_under_test.load_me_data = mock_load_me_data
That simply replaces the appropriate name right in the module itself, so when you invoke the code under test, presumably do_something() in your question, it calls your mock routine.
Well, if the import fails by raising an exception, you could put it in a try...except loop:
try:
from LegacyDataLoader import load_me_data
except: # put error that occurs here, so as not to mask actual problems
from MockDataLoader import load_me_data
Is that what you're looking for? If it fails, but doesn't raise an exception, you could have it run the unit test with a special command line tag, like --unittest, like this:
import sys
if "--unittest" in sys.argv:
from MockDataLoader import load_me_data
else:
from LegacyDataLoader import load_me_data