Develop Reference

Object is enumerable but not indexable?

Problem summary and question
I'm trying to look at some of the data inside an object that can be enumerated over but not indexed. I'm still newish to python, but I don't understand how this is possible.
If you can enumerate it, why can't you access the index through the same way enumerate does? And if not, is there a way to access the items individually?
The actual example
import tensorflow_datasets as tfds
train_validation_split = tfds.Split.TRAIN.subsplit([6, 4])
(train_data, validation_data), test_data = tfds.load(
name="imdb_reviews",
split=(train_validation_split, tfds.Split.TEST),
as_supervised=True)
Take a select subset of the dataset
foo = train_data.take(5)
I can iterate over foo with enumerate:
[In] for i, x in enumerate(foo):
print(i)
which generates the expected output:
0
1
2
3
4
But then, when I try to index into it foo[0] I get this error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-44-2acbea6d9862> in <module>
----> 1 foo[0]
TypeError: 'TakeDataset' object does not support indexing

Python only allows these things if the class has methods for them:
__getitem__ is required for the [] syntax.
__iter__ and __next__1 are required to iterate.
Any class can define one without defining the other. __getattr__ is usually not defined if it would be inefficient.
1 __next__ is required on the class returned by __iter__.

This is a result of foo being iterable, but not having a __getitem__ function. You can use itertools.isslice to get the nth element of an iterable like so
import itertools
def nth(iterable, n, default=None):
"Returns the nth item or a default value"
return next(itertools.islice(iterable, n, None), default)

In Python, instances of custom classes can implement enumeration through special (or "dunder") __iter__ method. Perhaps this class implements __iter__ but not __getitem__.
Dunder overview: https://dbader.org/blog/python-dunder-methods
Specs for an __iter__ method: https://docs.python.org/3/library/stdtypes.html#typeiter

How to get the arguments from a method call on a mocked object?

I have a unit test where the setup mocks a client like so:
def setUp(self):
self.mock_client = mock.patch.object(module_name, 'ClientClassName', autospec=True).start()
Then in my test I have a faked return value:
def myTest(self):
self.mock_client.my_method.return_value = ...
Now I want to get the arguments that my_method was called with, however I've been tearing my hair out trying to access them. It seems that I can't just do:
mock_args, mock_kwargs = self.mock_client.my_method.call_args
This gives me back First off why doesn't this work? I did make a little headway and found that:
self.mock_client.method_calls[0]
will give me back a call object that looks like call().my_method(...the arguments), but I have been trying for hours to get access to the individual arguments and cant seem to do it. Where am I going wrong?

Call args are just accessed with subscription on the mock.call object, i.e. __getitem__.
>>> from unittest.mock import MagicMock
>>> m = MagicMock()
>>> m(123, xyz="hello")
<MagicMock name='mock()' id='140736989479888'>
>>> m("another call")
<MagicMock name='mock()' id='140736989479888'>
>>> m.call_args_list
[call(123, xyz='hello'), call('another call')]
>>> m.call_args_list[0][0]
(123,)
>>> m.call_args_list[0][1]
{'xyz': 'hello'}
Item 0 will be a tuple of args, and item 1 will be a dict of kwargs. Attribute access also works, like a namedtuple (item 0 is attribute "args", and item 1 is attribute "kwargs"). If you only need to access the most recent call, you can use call_args instead of call_args_list.
Note that accessing the call args items directly is usually not required, you can use an assertion against another call instance in the tests:
>>> from unittest.mock import call
>>> m(k=123)
<MagicMock name='mock()' id='140736989479888'>
>>> assert m.call_args == call(k=123) # pass
>>> assert m.call_args == call(k=124) # fail
AssertionError
Or an even higher level, you can use m.assert_has_calls on the mock directly.
When mocking methods, whether the mock calls have self or not can be influenced by autospec:
>>> from unittest import mock
>>> class A(object):
... def f(self, *args, **kwargs):
... pass
...
>>> with mock.patch("__main__.A.f") as m:
... a = A()
... a.f('without autospec', n=1)
...
>>> m.call_args
call('without autospec', n=1)
>>> with mock.patch("__main__.A.f", autospec=True) as m:
... a = A()
... a.f('with autospec', n=2)
...
>>> m.call_args
call(<__main__.A object at 0x7fffe3d4e6a0>, 'with autospec', n=2)
This is discussed in more detail in the docs here.

In addition to wim's answer, you can actually dig down fairly deeply into this arguments, although sometimes you seem to find a string instead of a real object.
The main point to understand is that when you iterate through my_mock.call_args_list you get objects of type unittest.mock._Call. These can indeed be compared to call(...) objects which you have created yourself. But that's not all.
unittest.mock._Call is itself iterable, and consists of 2 elements: one is a tuple, the other is a dict. These are none other than the *args and **kwargs passed to the mock method.
Given that this is, as far as I can make out, completely undocumented in the docs, I suppose it is not beyond the bounds of possibility that this could break one day. It does often prove handy to know though, in my experience.

List method in python returns empty list when we pass magic mock object

I have a python code which needs to be run in both Python 2 and 3.
In python 2 dict.values() returns list but in python 3 it returns dict_val object. So to make it compatible I have put list(dict.values()). It works fine. But when I am doing a Unit test using python mock there is one error. I am mocking dict.values() and it gives output like this <MagicMock name='mock().values()' id='1099587993168'>, but when I use a list, it makes this empty list. Below is the example.
function file:
class abc():
def get_dict(self, key):#i want to mock this as its depends on other method also
dic = {'key': {'smaplekey': 'samplevalue'}}# its sample -
return dic['key']
def run_method(self, val):
print val
def a(self,key):
print 'before list'
print self.get_dict(key).values()
print list(self.get_dict(key).values())
b = list(self.get_dict(key).values())[0]
print 'after list'
self.run_method(b)
test file:
import unittest
from mock import Mock, patch, MagicMock, ANY
import function_file
class TestA(unittest.TestCase):
#patch('function_file.abc.run_method')
#patch('function_file.abc.get_dict',MagicMock(return_vlaue={'key': {}}))
def test_a(self, mock_run_method):
manager = function_file.abc()
result = manager.a('key')
mock_run_method.assert_called_once_with(ANY, create=True)
if __name__ == '__main__':
unittest.main()
Here list method make magic mock object empty list so it is failing.
And below is the python error
before list
<MagicMock name='mock().values()' id='1099803145040'>
[]
E
======================================================================
ERROR: test_a (__main__.TestA)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "test_manager.py", line 9, in test_a
result = manager.a('key')
File "/usr/lib/python2.7/site-packages/abcd/function_file.py", line 13, in a
b = list(self.get_dict(key).values())[0]
IndexError: list index out of range
----------------------------------------------------------------------
Ran 1 test in 0.004s
FAILED (errors=1)
Here it is going inside the 'a' method and printing also, but list method make mock object empty list. List method should not make mock object empty list

change:
#patch('get_dict',MagicMock(return_vlaue={'key': {}}))
to
#patch('__main__.get_dict', MagicMock(return_value={'key': 1}))
patch target should in the form "module.object_name", you missed namespace, remain mistakes are typos.

Try #patch(__name__ + '.get_dict', MagicMock(return_value={'key': {})
to give a proper target to your mock.

Function with fewer arguments and optional arguments

I often write code that looks like this and am looking for a better suggestion. Basically, I usually create some general function myfuc_general that handles all the cases I need with parameters, usually with optional parameters. However, often I usually run 2 (possibly more) specific functions. Everything is the same except one of the arguments is different, in this case a. I run them so often that I actually prefer to just have two additional functions so I don't have to remember what the optional parameter needs to be.
So for myfunct_specific1, I am running a=10 and for myfunct_specific2, a=20. Is there something better to do than this? This seems pretty sloppy and it has the disadvantage in the event that I need to change the myfuct_general call, then I have to change all the other functions.
def myfunc_general(constant, a=1,b=2):
return constant+a+b
def myfunct_specific1(constant,b=2):
a=10
return myfunc_general(constant,a,b=2)
def myfunct_specific2(constant,b=2):
a=20
return myfunc_general(constant,a,b=2)
print myfunct_specific1(3) #15
print myfunct_specific2(3) #25
edit (addition):
iCodez thank you for suggestion. I have this particular situation and it is throwing me an error. Help? Thanks again
def myfunc_general(constant, constant2, a=0,b=2):
return constant+constant2+b+a
import functools
myfunct_specific=functools.partial(myfunc_general,constant2=30)
print myfunct_specific
print myfunct_specific(3,5,b=3)
Traceback (most recent call last):
File "C:/Python27/test", line 8, in <module>
print myfunct_specific(3,5,b=3)
TypeError: myfunc_general() got multiple values for keyword argument 'constant2'

You can use functools.partial to make this a lot easier:
from functools import partial
def myfunc_general(constant, a=1, b=2):
return constant+a+b
myfunct_specific1 = partial(myfunc_general, a=10)
myfunct_specific2 = partial(myfunc_general, a=20)
Below is a demonstration:
>>> from functools import partial
>>>
>>> def myfunc_general(constant, a=1, b=2):
... return constant+a+b
...
>>> myfunct_specific1 = partial(myfunc_general, a=10)
>>> myfunct_specific2 = partial(myfunc_general, a=20)
>>>
>>> print myfunct_specific1(3)
15
>>> print myfunct_specific2(3)
25
>>>

Asserting successive calls to a mock method

Mock has a helpful assert_called_with() method. However, as far as I understand this only checks the last call to a method.
If I have code that calls the mocked method 3 times successively, each time with different parameters, how can I assert these 3 calls with their specific parameters?

assert_has_calls is another approach to this problem.
From the docs:
assert_has_calls (calls, any_order=False)
assert the mock has been
called with the specified calls. The mock_calls list is checked for
the calls.
If any_order is False (the default) then the calls must be sequential.
There can be extra calls before or after the specified calls.
If any_order is True then the calls can be in any order, but they must
all appear in mock_calls.
Example:
>>> from unittest.mock import call, Mock
>>> mock = Mock(return_value=None)
>>> mock(1)
>>> mock(2)
>>> mock(3)
>>> mock(4)
>>> calls = [call(2), call(3)]
>>> mock.assert_has_calls(calls)
>>> calls = [call(4), call(2), call(3)]
>>> mock.assert_has_calls(calls, any_order=True)
Source: https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls

Usually, I don't care about the order of the calls, only that they happened. In that case, I combine assert_any_call with an assertion about call_count.
>>> import mock
>>> m = mock.Mock()
>>> m(1)
<Mock name='mock()' id='37578160'>
>>> m(2)
<Mock name='mock()' id='37578160'>
>>> m(3)
<Mock name='mock()' id='37578160'>
>>> m.assert_any_call(1)
>>> m.assert_any_call(2)
>>> m.assert_any_call(3)
>>> assert 3 == m.call_count
>>> m.assert_any_call(4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "[python path]\lib\site-packages\mock.py", line 891, in assert_any_call
'%s call not found' % expected_string
AssertionError: mock(4) call not found
I find doing it this way to be easier to read and understand than a large list of calls passed into a single method.
If you do care about order or you expect multiple identical calls, assert_has_calls might be more appropriate.
Edit
Since I posted this answer, I've rethought my approach to testing in general. I think it's worth mentioning that if your test is getting this complicated, you may be testing inappropriately or have a design problem. Mocks are designed for testing inter-object communication in an object oriented design. If your design is not objected oriented (as in more procedural or functional), the mock may be totally inappropriate. You may also have too much going on inside the method, or you might be testing internal details that are best left unmocked. I developed the strategy mentioned in this method when my code was not very object oriented, and I believe I was also testing internal details that would have been best left unmocked.

You can use the Mock.call_args_list attribute to compare parameters to previous method calls. That in conjunction with Mock.call_count attribute should give you full control.

I always have to look this one up time and time again, so here is my answer.
Asserting multiple method calls on different objects of the same class
Suppose we have a heavy duty class (which we want to mock):
In [1]: class HeavyDuty(object):
...: def __init__(self):
...: import time
...: time.sleep(2) # <- Spends a lot of time here
...:
...: def do_work(self, arg1, arg2):
...: print("Called with %r and %r" % (arg1, arg2))
...:
here is some code that uses two instances of the HeavyDuty class:
In [2]: def heavy_work():
...: hd1 = HeavyDuty()
...: hd1.do_work(13, 17)
...: hd2 = HeavyDuty()
...: hd2.do_work(23, 29)
...:
Now, here is a test case for the heavy_work function:
In [3]: from unittest.mock import patch, call
...: def test_heavy_work():
...: expected_calls = [call.do_work(13, 17),call.do_work(23, 29)]
...:
...: with patch('__main__.HeavyDuty') as MockHeavyDuty:
...: heavy_work()
...: MockHeavyDuty.return_value.assert_has_calls(expected_calls)
...:
We are mocking the HeavyDuty class with MockHeavyDuty. To assert method calls coming from every HeavyDuty instance we have to refer to MockHeavyDuty.return_value.assert_has_calls, instead of MockHeavyDuty.assert_has_calls. In addition, in the list of expected_calls we have to specify which method name we are interested in asserting calls for. So our list is made of calls to call.do_work, as opposed to simply call.
Exercising the test case shows us it is successful:
In [4]: print(test_heavy_work())
None
If we modify the heavy_work function, the test fails and produces a helpful error message:
In [5]: def heavy_work():
...: hd1 = HeavyDuty()
...: hd1.do_work(113, 117) # <- call args are different
...: hd2 = HeavyDuty()
...: hd2.do_work(123, 129) # <- call args are different
...:
In [6]: print(test_heavy_work())
---------------------------------------------------------------------------
(traceback omitted for clarity)
AssertionError: Calls not found.
Expected: [call.do_work(13, 17), call.do_work(23, 29)]
Actual: [call.do_work(113, 117), call.do_work(123, 129)]
Asserting multiple calls to a function
To contrast with the above, here is an example that shows how to mock multiple calls to a function:
In [7]: def work_function(arg1, arg2):
...: print("Called with args %r and %r" % (arg1, arg2))
In [8]: from unittest.mock import patch, call
...: def test_work_function():
...: expected_calls = [call(13, 17), call(23, 29)]
...: with patch('__main__.work_function') as mock_work_function:
...: work_function(13, 17)
...: work_function(23, 29)
...: mock_work_function.assert_has_calls(expected_calls)
...:
In [9]: print(test_work_function())
None
There are two main differences. The first one is that when mocking a function we setup our expected calls using call, instead of using call.some_method. The second one is that we call assert_has_calls on mock_work_function, instead of on mock_work_function.return_value.