I am trying to mock the call to a function and still have the effect of the function apply. I found the solution using Python wraps, but all examples I found are applied to mocking the member method of a class. In my case, I have a pure function (not defined in a class). This seems to not work with the usual examples, as they require you to instantiate the class first I guess to obtain the real version of the method to pass into wraps.
Can this be done with pure functions?
https://wesmckinney.com/blog/spying-with-python-mocks/
My code:
<this_module.py>
def my_function:
does something important
def test_my_function:
with patch.object("this_module", "my_function", wraps="this_module.my_function")
Related
I have a complex function that calls many other 3rd party methods. I monkeypatched them out one by one:
import ThirdParty as tp
def my_method():
tp.func_3rd_party_1()
...
tp.func_3rd_party_5()
return "some_value"
In my test:
import pytest
def test_my_method(monkeypatch):
monkeypatch.setattr(ThirdParty, 'func_3rd_party_1', some_mock_1())
...
monkeypatch.setattr(ThirdParty, 'func_3rd_party_5', some_mock_5())
return_value = my_method()
assert return value
This runs just fine but the test feels too implicit for me in this form. I'd like to explicitly state that the monkeypatched methods were indeed called.
For the record, my mocked methods are not using any inbuilt Mock library resource. They are just redefined methods (smart stubs).
Is there any way to assert for that?
So the pytest monkeypatching fixture is specifically provided so you can change some global attributes like environment variables, stuff in third party libraries, etc, to provide some controlled and easy behavior for your test.
The Mock objects, on the other hand, are meant to provide all sorts of tracking and inspection on the object.
The two go hand in hand: You use patching to replace some third party function with a Mock object, then execute your code, and then ask the Mock object if it has indeed been invoked with the right arguments, for the right number of times.
Note that even though the mock module is part of unittest, it works perfectly fine with pytest.
Now as for the patching itself, it's up to your personal preference, and depends a bit on what exactly you want to patch, whether using unittest.mock.patch is more compact or pytest's monkeypatch fixture.
import pytest
from unittest.mock import Mock
def test_my_method(monkeypatch):
# refer to the mock module documentation for more complex
# set ups, where the mock object _also_ exhibits some behavior.
# as is, calling the function doesn't actually _do_ anything.
some_mock_1 = Mock()
...
some_mock_5 = Mock(return_value=66)
monkeypatch.setattr(ThirdParty, 'func_3rd_party_1', some_mock_1)
...
monkeypatch.setattr(ThirdParty, 'func_3rd_party_5', some_mock_5)
some_mock_1.assert_called_once()
some_mock_5.assert_called_with(42)
...
Now a note on this type of testing: Don't go overboard! It can quite easily lead to what's called brittle tests: Tests that break with the slightest change to your code. It can make refactoring an impossible neightmare.
These types of assertions are best when you use them in a message-focused object-oriented approach. If the whole point of the class or method under test is to invoke, in a particular way, the method or class of another object, then Mock away. If the calls to third party functions on the other hand are merely a means to an end, then go a level higher with your test and test for the desired behavior instead.
I'm a beginner programmer, I've been stuck for the past week trying to write unit tests. I read through the unit test docs and watched two long tutorials on implementing unit testing with Mock. The docs refer to mocking classes extensively, but for functions, I'm not sure if I should be using #patch/with patch, patch.dict{}, side_effect, or some other option to mock a function, specifically the argument to a function.
mymodule.py
def regex():
'''Runs a regex, creates a dict 'data' and then calls scraper(data)'''
def scraper(data):
'''scrapes a website and then calls a function which submits data to a db'''
I would like to create a test that passes in test data to the function scraper . Thank you in advance.
Yes, you can also do unit test using mock for non-object_oriented code.
See example below:
from unittest.mock import MagicMock
def a():
return 10
def b():
print(a())
b()
a = MagicMock(return_value=3)
b()
And the output is:
10
3
In the previous example mock is used to fake/mock the function a(), so you can test function b() in isolation, b() is your SUT, without calling a() real implementation. This can be useful for more complex code, specially when function a() relies on data that might not be available in the unit test level.
You haven't given enough examples to really help, but a couple of observations:
Generally, it's better to structure your code to avoid the need for mocking in tests; to the extent possible, each function should be a self-contained piece of code, which can be called separately.
There's no need to mock arguments; simply pass the test value in.
I'm not sure what's intended with this code:
bar = {'key': 'value'}
def foo(bar):
pass
The bar defined at the outer level is a completely separate variable to the bar used as an argument in the function definition. It is confusing to give them the same name...
A function can be mocked using patch or patch.object with the return_value=... option; often, though, it suggests that the code needs to be refactored to reduce the dependency between the two functions.
I have an architecture, where I use wrapper for calling functions from package module. Inside the module there is a function that calls another three. I need to override one of them in run-time. Exactly I need to change parameters that are forwarded to another set of functions being called.
Here is a case sample:
a.py
import b_wrapper as wrapper
def foo():
if wrapper.bar(parameter):
"""some more code goes here"""
b_wrapper.py
import some.package.module as module
def bar(parameter):
return module.baz(veryImportantParameter, parameter)
file.py
def functionThree(par): # needs to be overwritten
"""more functions called forwarding par as a parameter"""
def baz(veryImportantParameter, parameter)
functionOne(veryImportantParameter, otherParameters)
functionTwo(veryImportantParameter, someMoreParameters)
functionThree(veryImportantParameter, parameterToChange, evenMoreParameters)
What I tried to do is overriding in wrapper file, didn't work out, as other functions are interfering with it. As reference used this post.
I'm not quite sure that this is doable, because of unique functions that are called inside this module, also looking for alternatives that won't require overriding portion of module.
Edit: mixing up arguments and parameters is intentional for demonstration purpose only.
I am trying to understand, what is monkey patching or a monkey patch?
Is that something like methods/operators overloading or delegating?
Does it have anything common with these things?
No, it's not like any of those things. It's simply the dynamic replacement of attributes at runtime.
For instance, consider a class that has a method get_data. This method does an external lookup (on a database or web API, for example), and various other methods in the class call it. However, in a unit test, you don't want to depend on the external data source - so you dynamically replace the get_data method with a stub that returns some fixed data.
Because Python classes are mutable, and methods are just attributes of the class, you can do this as much as you like - and, in fact, you can even replace classes and functions in a module in exactly the same way.
But, as a commenter pointed out, use caution when monkeypatching:
If anything else besides your test logic calls get_data as well, it will also call your monkey-patched replacement rather than the original -- which can be good or bad. Just beware.
If some variable or attribute exists that also points to the get_data function by the time you replace it, this alias will not change its meaning and will continue to point to the original get_data. (Why? Python just rebinds the name get_data in your class to some other function object; other name bindings are not impacted at all.)
A MonkeyPatch is a piece of Python code which extends or modifies
other code at runtime (typically at startup).
A simple example looks like this:
from SomeOtherProduct.SomeModule import SomeClass
def speak(self):
return "ook ook eee eee eee!"
SomeClass.speak = speak
Source: MonkeyPatch page on Zope wiki.
What is a monkey patch?
Simply put, monkey patching is making changes to a module or class while the program is running.
Example in usage
There's an example of monkey-patching in the Pandas documentation:
import pandas as pd
def just_foo_cols(self):
"""Get a list of column names containing the string 'foo'
"""
return [x for x in self.columns if 'foo' in x]
pd.DataFrame.just_foo_cols = just_foo_cols # monkey-patch the DataFrame class
df = pd.DataFrame([list(range(4))], columns=["A","foo","foozball","bar"])
df.just_foo_cols()
del pd.DataFrame.just_foo_cols # you can also remove the new method
To break this down, first we import our module:
import pandas as pd
Next we create a method definition, which exists unbound and free outside the scope of any class definitions (since the distinction is fairly meaningless between a function and an unbound method, Python 3 does away with the unbound method):
def just_foo_cols(self):
"""Get a list of column names containing the string 'foo'
"""
return [x for x in self.columns if 'foo' in x]
Next we simply attach that method to the class we want to use it on:
pd.DataFrame.just_foo_cols = just_foo_cols # monkey-patch the DataFrame class
And then we can use the method on an instance of the class, and delete the method when we're done:
df = pd.DataFrame([list(range(4))], columns=["A","foo","foozball","bar"])
df.just_foo_cols()
del pd.DataFrame.just_foo_cols # you can also remove the new method
Caveat for name-mangling
If you're using name-mangling (prefixing attributes with a double-underscore, which alters the name, and which I don't recommend) you'll have to name-mangle manually if you do this. Since I don't recommend name-mangling, I will not demonstrate it here.
Testing Example
How can we use this knowledge, for example, in testing?
Say we need to simulate a data retrieval call to an outside data source that results in an error, because we want to ensure correct behavior in such a case. We can monkey patch the data structure to ensure this behavior. (So using a similar method name as suggested by Daniel Roseman:)
import datasource
def get_data(self):
'''monkey patch datasource.Structure with this to simulate error'''
raise datasource.DataRetrievalError
datasource.Structure.get_data = get_data
And when we test it for behavior that relies on this method raising an error, if correctly implemented, we'll get that behavior in the test results.
Just doing the above will alter the Structure object for the life of the process, so you'll want to use setups and teardowns in your unittests to avoid doing that, e.g.:
def setUp(self):
# retain a pointer to the actual real method:
self.real_get_data = datasource.Structure.get_data
# monkey patch it:
datasource.Structure.get_data = get_data
def tearDown(self):
# give the real method back to the Structure object:
datasource.Structure.get_data = self.real_get_data
(While the above is fine, it would probably be a better idea to use the mock library to patch the code. mock's patch decorator would be less error prone than doing the above, which would require more lines of code and thus more opportunities to introduce errors. I have yet to review the code in mock but I imagine it uses monkey-patching in a similar way.)
According to Wikipedia:
In Python, the term monkey patch only
refers to dynamic modifications of a
class or module at runtime, motivated
by the intent to patch existing
third-party code as a workaround to a
bug or feature which does not act as
you desire.
First: monkey patching is an evil hack (in my opinion).
It is often used to replace a method on the module or class level with a custom implementation.
The most common usecase is adding a workaround for a bug in a module or class when you can't replace the original code. In this case you replace the "wrong" code through monkey patching with an implementation inside your own module/package.
Monkey patching can only be done in dynamic languages, of which python is a good example. Changing a method at runtime instead of updating the object definition is one example;similarly, adding attributes (whether methods or variables) at runtime is considered monkey patching. These are often done when working with modules you don't have the source for, such that the object definitions can't be easily changed.
This is considered bad because it means that an object's definition does not completely or accurately describe how it actually behaves.
Monkey patching is reopening the existing classes or methods in class at runtime and changing the behavior, which should be used cautiously, or you should use it only when you really need to.
As Python is a dynamic programming language, Classes are mutable so you can reopen them and modify or even replace them.
What is monkey patching? Monkey patching is a technique used to dynamically update the behavior of a piece of code at run-time.
Why use monkey patching? It allows us to modify or extend the behavior of libraries, modules, classes or methods at runtime without
actually modifying the source code
Conclusion Monkey patching is a cool technique and now we have learned how to do that in Python. However, as we discussed, it has its
own drawbacks and should be used carefully.
I need to create a function to rotate a given matrix (list of lists) clockwise, and I need to use it in my Table class. Where should I put this utility function (called rotateMatrixClockwise) so I can call it easily from within a function in my Table class?
Make it a static function...
add the #staticmethod decorator
don't include 'self' as the first argument
Your definition would be:
#staticmethod
def rotateMatrixClockwise():
# enter code here...
Which will make it callable everywhere you imported 'table' by calling:
table.rotateMatrixClockwise()
The decorator is only necessary to tell python that no implicit first argument is expected. If you wanted to make method definitions act like C#/Java where self is always implicit you could also use the '#classmethod' decorator.
Here's the documentation for this coming directly from the python manual.
Note: I'd recommend using Utility classes only where their code can't be coupled directly to a module because they generally violate the 'Single Responsibility Principle' of OOP. It's almost always best to tie the functionality of a class as a method/member to the class.
If you don't want to make it a member of the Table class you could put it into a utilities module.