I have the following my_func.py with create_config function.
*my_func.py
from fabric.state import env
def create_config(node_name):
config = {
"log_level": "INFO",
"addr1": "127.0.0.1",
}
config["addr2"] = env.host
return config
I tried the following approach to mock env.host variable where env is an import from fabric.state.
*test.py
import unittest
import my_func
import mock
class MyTestCase(unittest.TestCase):
def setUp(self):
self.master_config = {
"log_level": "INFO",
"addr2": "0.0.0.0",
"addr1": "127.0.0.1",
}
#mock.patch('env.host')
def test_create_consul_config(self, mock_host):
mock_host.return_value = "0.0.0.0"
result = my_func.create_config('master')
self.assertDictEqual(self.master_config, result)
if __name__ == '__main__':
unittest.main()
I am getting import error with 'env'. What is the best way to mock variable within a function with python mock.
ImportError: No module named env
mock variable env.host?
get the type of env first
In [6]: from fabric.state import env
In [7]: type(env)
Out[7]: fabric.utils._AttributeDict
env.host is instance variable of a class, the mock is a little different, mock_env is object(AttributeDict), the assignment of instance_variable host is direct assignment, not with a return_value
#mock.patch('my_func.env')
def test_create_consul_config(self, mock_env):
mock_env.host = 'xxx'
result = my_func.create_config('master')
self.assertDictEqual(self.master_config, result)
From the unittest.mock documentation on patch (note target is the first argument of patch):
target should be a string in the form 'package.module.ClassName'. The
target is imported and the specified object replaced with the new
object, so the target must be importable from the environment you are
calling patch() from. The target is imported when the decorated
function is executed, not at decoration time.
So you need to include the full path to the function you are patching. Note also in where to patch that the target should be the path to where the function/object is used, not where it is defined.
So changing your patch call to:
#mock.patch("my_func.env.host")
should fix the ImportError.
Related
I need to mock a global variable. This variable is globally initialized in package_a, but trying to mock it imports the method and I can't get it to.
Package_a works fine but when called from test/ the path used by package_a no longer works. Instead of being /assest/file it is /test/assets/file and therefore does not find the file and returns the error 'FileNotFoundError: [Errno 2] No such file or directory:'. So I can't import package_a without first simulating the "path_to_instance" variable.
path_to_instance = 'last_path'
variable_a = package_b.instance_variable_a(path_to_instance)
class Testimporter(unittest.TestCase):
#patch('package_a.path_to_instance', 'new_path')
def test_case_00(self):
import package_a
I get that when the package_a is imported the path_to_instance variable has the value "last_path" and not "new_path".
What am I doing wrong?
Thanks in advance.
I tried:
class TestMapper(unittest.TestCase):
def setUp(self):
self.patcher = patch('package_a.path_to_instance')
self.mock_object = self.patcher.start()
and other changes that didn't work. I never get to "change" the value of the variable "path_to_instance "
It seems that when #patch('package_a.path_to_instance') is called it imports the module and fails.
I am having what I believe to be a common issue in using mock patching in that I can not figure out the right thing to patch.
I have two questions that I am hoping for help with.
Thoughts on how to fix the specific issue in the below example
And possibly most-importantly pro-tips/pointers/thoughts/suggestions on how to best troubleshoot the "which thing do I patch" question. The problem I'm having is, without a full understanding of how patching works, I really dont even know what I should be looking for and find myself playing a guessing game.
An example using pyarrow that is currently causing me pain:
mymodule.py
import pyarrow
class HdfsSearch:
def __init__(self):
self.fs = self._connect()
def _connect(self) -> object:
return pyarrow.hdfs.connect(driver="libhdfs")
def search(self, path: str):
return self.fs.ls(path=path)
test_module.py
import pyarrow
import pytest
from mymodule import HdfsSearch
#pytest.fixture()
def hdfs_connection_fixture(mocker):
mocker.patch("pyarrow.hdfs.connect")
yield HdfsSearch()
def test_hdfs_connection(hdfs_connection_fixture):
pyarrow.hdfs.connect.assert_called_once() # <-- succeeds
def test_hdfs_search(hdfs_connection_fixture):
hdfs_connection_fixture.search(".")
pyarrow.hdfs.HadoopFileSystem.ls.assert_called_once() # <-- fails
pytest output:
$ python -m pytest --verbose test_module.py
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/bbaur/miniconda3/envs/dev/bin/python
cachedir: .pytest_cache
rootdir: /home/user1/work/app
plugins: cov-2.7.1, mock-1.10.4
collected 2 items
test_module.py::test_hdfs_connection PASSED [ 50%]
test_module.py::test_hdfs_search FAILED [100%]
================================================================================================================= FAILURES =================================================================================================================
_____________________________________________________________________________________________________________ test_hdfs_search _____________________________________________________________________________________________________________
hdfs_connection_fixture = <mymodule.HdfsSearch object at 0x7fdb4ec2a610>
def test_hdfs_search(hdfs_connection_fixture):
hdfs_connection_fixture.search(".")
> pyarrow.hdfs.HadoopFileSystem.ls.assert_called_once()
E AttributeError: 'function' object has no attribute 'assert_called_once'
test_module.py:16: AttributeError
You're not calling the assert on the Mock object, this is the correct assert:
hdfs_connection_fixture.fs.ls.assert_called_once()
Explanation:
When you access any attribute in a Mock object it will return another Mock object.
Since you patched "pyarrow.hdfs.connect" you've replaced it with a Mock, let's call it Mock A. Your _connect method will return that Mock A and you'll assign it to self.fs.
Now let's break down what's happening in the search method when you call self.fs.ls.
self.fs returns your Mock A object, then the .ls will return a different Mock object, let's call it Mock B. In this Mock B object you're doing the call passing (path=path).
In your assert you're trying to access pyarrow.hdfs.HadoopFileSystem, but it was never patched. You'll need do the assert on the Mock B object, which is at hdfs_connection_fixture.fs.ls
What to Patch
If you change your import in mymodule.py to this from pyarrow.hdfs import connect your patch will stop working.
Why is that?
When you patch something you're changing what a name points to, not the actual object.
Your current patch is patching the name pyarrow.hdfs.connect and in mymodule you're using the same name pyarrow.hdfs.connect so everything is fine.
However, if you use from pyarrow.hdfs import connect mymodule will have imported the real pyarrow.hdfs.connect and created a reference for it with the name mymodule.connect.
So when you call connect inside mymodule you're accessing the name mymodule.connect, which is not patched.
That is why you would need to patch mymodule.connect when using from import.
I'd recommend using from x import y when doing this kind of patching. It makes it more explicit what you're trying to mock and the patch will be limited to that module only, which can prevent unforeseen side-effects.
Source, this section in the Python documentation: Where to patch
To understand how patching works in python let's first understand the import statement.
When we use import pyarrow in a module (mymodule.py in this case) it does two operations :
It searches for the pyarrow module in sys.modules
It binds the results of that search to a name(pyarrow) in the local scope.
By doing something like: pyarrow = sys.modules['pyarrow']
NOTE: import statements in python doesn't execute code. The import statement brings a name into local scope. The execution of code happens as a side-effect only when python can't find a module in sys.modules
So, to patch pyarrow imported in mymodule.py we need to patch the pyarrow name present in the local scope of mymodule.py
patch('mymodule.pyarrow', autospec=True)
test_module.py
import pytest
from mock import Mock, sentinel
from pyarrow import hdfs
from mymodule import HdfsSearch
class TestHdfsSearch(object):
#pytest.fixture(autouse=True, scope='function')
def setup(self, mocker):
self.hdfs_mock = Mock(name='HadoopFileSystem', spec=hdfs.HadoopFileSystem)
self.connect_mock = mocker.patch("mymodule.pyarrow.hdfs.connect", return_value=self.hdfs_mock)
def test_initialize_HdfsSearch_should_connect_pyarrow_hdfs_file_system(self):
HdfsSearch()
self.connect_mock.assert_called_once_with(driver="libhdfs")
def test_initialize_HdfsSearch_should_set_pyarrow_hdfs_as_file_system(self):
hdfs_search = HdfsSearch()
assert self.hdfs_mock == hdfs_search.fs
def test_search_should_retrieve_directory_contents(self):
hdfs_search = HdfsSearch()
self.hdfs_mock.ls.return_value = sentinel.contents
result = hdfs_search.search(".")
self.hdfs_mock.ls.assert_called_once_with(path=".")
assert sentinel.contents == result
Use context managers to patch built-ins
def test_patch_built_ins():
with patch('os.curdir') as curdir_mock: # curdir_mock lives only inside with block. Doesn't lives outside
assert curdir_mock == os.curdir
assert os.curdir == '.'
I have a class with a method which uses shutil.rmtree to remove some files if a param is passed as true, How to mock this behavior so as other tests don't break which needs these files.
My class looks like this -
class FileConverter(object):
def __init__(self, path_to_files):
self._path_to_files = path_to_files
def convert_files(self, rmv_src=False):
doStuff()
if rmv_src:
shutil.rmtree(self.__path_to_files)
def doStuff():
# does some stuff
Now my tests look like -
class TestFileConverter(unittest.TestCase):
def test_convert_success(self):
input_dir = 'resources/files'
file_converter = FileConverter(input_dir)
file_converter.convert_files()
# assert the things from doStuff
#mock.patch('shutil.rmtree')
def test_convert_with_rmv(self, rm_mock):
input_dir = 'resources/files'
file_converter = FileConverter(input_dir)
file_converter.convert_files(True)
self.assertEquals(rm_mock, [call(input_dir)])
Now when I run this testsuite the test with rmv gives me assertionError
<MagicMock name='rmtree' id='139676276822984'> != [call('resources/images')]
and the first test gives me file not found error since the mock did not work and the rmv source test removed the file
FileNotFoundError: [Errno 2] No such file or directory: 'resources/images'
If I comment out the second test with rmv_source true then my first test works fine.
What am I doing wrong here?
Your module has already imported shutil.rmtree so mocking it later in the test suite won't do anything.
You need to mock the module when you import FileConverter, not afterwards.
import sys
from mock import MagicMock
sys.modules['shutil'] = MagicMock()
# and/or
sys.modules['shutil.rmtree'] = MagicMock()
import FileConverter
If you still need to use shutil in your test code, then first import it using an alias,and use that when you need the 'real' module:
import sys
from mock import MagicMock
import shutil as shutil_orig
sys.modules['shutil'] = MagicMock()
import shutil
print(type(shutil_orig.rmtree))
# <class 'function'>
print(type(shutil.rmtree))
# <class 'mock.mock.MagicMock'>
The original post should work except the call(input_dir) did not work for me
#mock.patch('shutil.rmtree')
def test_convert_with_rmv(self, rm_mock):
input_dir = 'resources/files'
rm_mock.return_value = 'REMOVED'
file_converter = FileConverter(input_dir)
file_converter.convert_files(True)
rm_mock.assert_called_with(input_dir)
self.assertEqual(rm_mock.return_value, 'REMOVED')
The test_convert_with_rmv has no way removed the input_dir, it probably never created in the first place. You could assert this statement in each test before and after convert_files called:
self.asserTrue(os.path.isdir(input_dir))
I have a class method that I am trying to test that requires two patched methods, ConfigB.__init__ and listdir:
from os import listdir
from config.ConfigB import ConfigB
class FileRunner(object):
def runProcess(self, cfgA)
cfgB = ConfigB(cfgA)
print(listdir())
I have the following test set up:
import unittest
import unittest.mock imort MagicMock
import mock
from FileRunner import FileRunner
class TestFileRunner(unittest.TestCase):
#mock.patch('ConfigB.ConfigB.__init__')
#mock.patch('os.listdir')
def test_methodscalled(self, osListDir, cfgB):
cfgA = MagicMock()
fileRunner = FileRunner()
cfgB.return_value = None
osListDir.return_value = None
fileRunner.runProcess(cfgA)
Now the patched mock and return value works for ConfigB.ConfigB, but it does not work for os.listdir. When the print(listdir()) method runs I get a list of file in the current directory, not a value of None as I specified in the patched return value. Any idea what is going wrong?
You need to patch your relative path to your code. patch('os.listdir') doesn't works because you need to patch this:
#mock.patch("path.to.your.pythonfile.listdir")
Try with that.
I'm using Python for a weeks now and i'm confronted to an issue with dynamic import.
I have a file Test.py that in which a class is defined. I would like to use this class after the dynamic import of Test.py from another file.
My final goal is more complex but I simplified it but i still get the same problem.
File : Test.py
class Test :
def __init__ ( self ) :
print ( "instance" )
File : Main.py
def allImports ( ) :
__import__ ( "Test" )
What i get :
>>> import Main
>>> Main.allImports()
>>> myInstance = Test ()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'Test' is not defined
I cannot specify in the fromlist which element from Test.py i have to import because i'm not supposed to know them.
What should i do ?
For a solution closer to your intent:
import importlib
def allImports(globals):
mod = importlib.import_module('Test', globals['__name__'])
try:
keys = mod.__all__
except AttributeError:
keys = dir(mod)
for key in keys:
if not key.startswith('_'):
globals[key] = getattr(mod, key)
# …
allImports(globals())
Test # should work, you can also look into dir(Test) to find the class.
If your module doesn't have an __all__ the code above /will/ clobber your namespace something fierce. Either make sure you define __all__, or modify allImports() to only import the things you want. (E.g. only classes, or only classes defined in the module. This part really depends on your use case.)
When using __import__() to load a module, you have to look it up in sys.modules:
>>> import sys
>>> import Main
>>> Main.allImports()
>>> myInstance = sys.modules['Test'].Test()
instance
>>>
More information in the documentation and here, here, and here.
this code makes __import__ ( "Test" ) a local variable, so you can't access it outside the function.
def allImports ( ) :
__import__ ( "Test" )
try:
def allImports ( ) :
test= __import__ ( "Test" )
return test #return the module
>>> import Main
>>> x=Main.allImports() #store the returned module in x
>>> myInstance = x.Test ()
instance
>>>myInstance
<Test.Test instance at 0x011D7F80>
__import__ doesn't modify magically neither global nor local namespaces.
Modules and classes are first class citizens in Python i.e., you can use them as any other object in Python (bind to a name, pass as a parameter to a function, return as a value from a function).
def allImports():
return __import__("Test")
Test_module = allImports()
Test = Test_module.Test # class
test_instance = Test()
If the above code is inside a function then to put Test into global namespace: globals()['Test'] = Test. Note most probably you don't need it and there are better ways to do whatever you want without modifying global namespace inside a function.
Usage of __import__() is discouraged use importlib.import_module() instead.
If the name of the module and the class are known you could just write at the module level:
from Test import Test