Is possible to mock the declaration import pyarrow.parquet as pq?
I'm trying to mock it as:
#patch("twilio_events.workers.EngineDWH")
#patch("twilio_events.workers.pyarrow")
#patch("twilio_events.workers.s3fs")
def test_process_batch(s3fs, py, e):
pass;
But it throws me an exception Attribute Error....<workspace/myproject/twilio_events/workers.py'> does not have the attribute 'pyarrow'>.
Other mocks are fine, only this one is throwing this exception :(.
Any help?
The target of a patch is the name that needs to be replaced. In this case, if paraquet is imported as pq, you have to replace pq.
Try something like
#patch("twilio_events.workers.pq")
This is explained a bit in the docs:
patch() works by (temporarily) changing the object that a name points to with another one. There can be many names pointing to any individual object, so for patching to work you must ensure that you patch the name used by the system under test.
The basic principle is that you patch where an object is looked up, which is not necessarily the same place as where it is defined.
Related
I'm using pytest to patch the os.makedirs method for a test. In a particular test I wanted to add a side effect of an exception.
So I import the os object that I've imported in my script under test, patch it, and then set the side effect in my test:
from infrastructure.scripts.src.server_administrator import os
def mock_makedirs(self, mocker):
mock = MagicMock()
mocker.patch.object(os, "makedirs", return_value=mock)
return mock
def test_if_directory_exist_exception_is_not_raised(self, administrator, mock_makedirs):
mock_makedirs.side_effect = Exception("Directory already exists.")
with pytest.raises(Exception) as exception:
administrator.initialize_server()
assert exception.value == "Directory already exists."
The problem I ran into was that when the mock gets called in my script under test, the side effect no longer existed. While troubleshooting I stopped the tests in the debugger to look at the ID values for the mock I created and the mock that the patch should have set as the return value and found that they are different instances:
I'm still relatively new to some of the testing tools in python, so this may be me missing something in the documentation, but shouldn't the returned mock patched in here be the mock I created?? Am I patching it wrong?
UPDATE
I even adjusted the import style to grab makedirs directly to patch it:
def mock_makedirs(self, mocker):
mock = MagicMock()
mocker.patch("infrastructure.scripts.src.server_administrator.makedirs", return_value=mock)
return mock
And I still run into the same "different mocks" issue.
:facepalm:
I was patching incorrectly. I'm considering just deleting the whole question/answer, but I figured I'd leave it here in case someone runs into the same situation.
I'm defining the patch like this:
mocker.patch.object(os, "makedirs", return_value=mock)
Which would be a valid structure if I was patching the result a function/method. That is, what this patch is saying is "when you call the makedirs, return this.
What I actually want to do is return a mock in place of the method. In it's current form it makes sense that I see two different mocks because the patch logic is currently "replace makedirs with a new mock and then when that mock is called, return this other mock (the mock I made)"
What I really want is just:
mocker.patch.object(os, "makedirs", mock)
Where my third argument (in the patch.object form) is the mock module parameter (vs the named return_value parameter).
In retrospect, it's pretty obvious when I think about it which is why I'm considering deleting the question, but it's an easy enough trip-up that I'm going to leave it live for now.
The below prints False. Is this not how mocking works?
I tried changing the path to the function, but it errors out, so the path seems correct. What am I missing?
import pytest
from deals.services.services import is_user_valid
class TestApi:
def test_api(self, mocker):
mocker.patch('deals.services.services.is_user_valid', return_value=True)
print(is_user_valid("sdfds", "sdfsdf"))
The issue here is that you're essentially doing the following:
from deals.services.services import is_user_valid
import deals.services.services
deals.services.services.is_user_valid = Mock(return_value=True)
# call local is_user_valid
By importing the "terminal" symbol itself you've shorted any possibility of mocking, it's now a local reference, and so updating the "remote" reference will have no effect on the local version. Meaning you should keep a handle on the module itself, such that the relevant symbol gets resolved on each access:
from deals.services import services
def test_api(mocker):
mocker.patch('deals.services.services.is_user_valid', return_value=True)
print(services.is_user_valid("sdfds", "sdfsdf"))
should work better.
This is also an issue with any module using such imports, they requiring patching the point of use rather than the point of definition because by the time the mock runs chances are the user module already has their copy.
See the documentation for some more details.
My actual problem was more complicated than the minimal version above. The mocked function was used in a different file.
What helped was to patch the function where it's imported/used, NOT where it's defined. So provide mocker.patch the path to the file where the function is imported.
Using Django 1.10 and python 3.5.1.
I'm trying to mock 'call_command' function to throw an exception. The problem is that seems like the moment it gets the 'side_effect' function - it keeps to it also for other tests. What am I doing wrong or how can I 'revert' the side_effect from that function?
In this example, after running one of the tests, all other tests that run afterwards will throw the same exception even if it's not supposed to throw exception in that test.
def test_run_migrations_raise_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=Exception('e message')):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
def test_run_migrations_raise_flow_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=FlowException(500, 'fe message', {'a': 1})):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
You should not patch a function that is in your module-local (i.e. Python's "global" - which is actually "module") namespace.
When in Python you do
from module.that import this
this becomes a variable on the module that contains the import statement. Any changes to "module.that.this" will affect the object pointed in the other module, but using only this will still reefer to the original object.
Perhaps your code is not exactly as you show us, or maybe "mock.pacth" can find out that the module-local call_command is pointing to django.core.management.call_command in the other module when it makes the patch - but not when reversing the patch. The fact is your module-local name call_command is being changed.
You can fix that by simply changing your code to not bind a module variable directly to the function you want to change:
from django.core import management
def test_run_migrations_raise_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=Exception('e message')):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
management.call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
I hope you can understand that and solve this problem. Now, that said, this use of mock makes no sense at all: the idea of using mock is that some callable used indirectly by code you call within the code-block that applies the patch does not have the original effect - so the intermetiate code can run and be tested. You are calling directly the mock object - so it will have none of the original code - calling call_command('run_data_migrations') runs no code on your code base at all, and thus, there is nothing there to test. It just calls the mocked instance, and it will not change the status of anything that could be detected with check_migrations_called.
I am attempting to mock a method and running into issues with mock actually overwriting it.
app/tests/test_file.py <- contains the the unit test, currently using:
#mock.patch('app.method', return_value='foo')
def test(self, thing):
...
do some stuff with app/main/server.py
and get its response, assert a few values
...
assert 'foo' is in return value of some stuff
The method being mocked is being called by another file that server.py is calling.
app/main/server.py <- what the unit test is actually interacting with
app/main/route.py <- where method being mocked is called
app/main/thing.py <- contains method to be mocked
This is with python 2.7 and each package has an init file. The parent folder (app) contains imports for every class and method. I've tried app.method which doesn't give problems, but doesnt work. I've tried thing.method, throws an error. I've tried app.main.thing.method which does nothing.
I've had success in this same test suite mocking an object and one of its methods, but that object is created and used directly in the server.py file. I'm wondering if it's because the method being called is so far down the chain. Mocking is pretty magical to me, especially in Python.
After more digging finally figured it out, will leave it up for any others that have problems (especially since it's not easily Google'able).
As #Gang specified the full path should work, however the module needs to be the module where the method is called, not the module where it is located, as this person points out. . Using #Gang example:
#mock.patch('app.main.route.method')
def test(self, mock_method):
mock_method.return_value = 'foo'
# call and assert
I've been searching on internet to find an example of using flexmock on python modules, but all doc's seem to be for object/class. I'm wondering if it's possible to mock some variables returned by a module. What if that module calls another module?
ex.
def function_inside_function(id, some_string):
test_log = {"id": id, "definition": some_string}
return test_log
def function1(id):
some_string = 'blah' + id # i am totally bs-ing here
log = function_inside_function(id, some_string)
return log
so now I want to test each function separately by using flexmock to mock some values
back then when doing the same thing with an object, I could do (say the object is assigned to be test_object)
flexmock(test_object).should_receive('some_func').and_return('some_value')
where some_func is being called inside that object
but when I try to do the same with a module, I kept getting
FlexmockError: <function function1 at some_address> does not have attribute function_inside_function
I want to know if it's possible to use flexmock on modules, and, if yes. how?
After a lot of research and trial n error, it turns out that I have to use sys.modules
say my module is imported from path.to.module, then the syntax would be
flexmock(sys.modules['path.to.module']).should_receive('function.of.object').and_return(response)
function.of.object is the function being called. For example, requests.get. using only get will not work.
response is the response you try to mock. in the requests.get example, the response would be a requests.Response(), and then you can use setattr to set the attributes if flexmock complains about it. (Is there a better way to do it?)