Develop Reference

Can we assign an object to a callable (or to a function) in python?

Introduction:
We know that, in python, a function is a class. To some extent, We can look at it as a data type which can be called and return a value. So it is a callable. We also know that Python classes are callables. When they are called, we are actually making objects as their instances.
My implementation: In a current task, I have defined the following class with two methods:
class SomeClass():
def some_method_1():
some_code
def some_method_2():
some_code
self.some_method_1()
some_code
To describe the code above, some_method_2 is using some_method_1 inside it.
Now I am seeking to test some_method_2. In this case, I need to replace some_method_1 with a mock object and specify the mock object to return what I define:
from unittest.mock import Mock
import unittest
class TestSomeClass(unittest.TestCase):
some_object=Some_Class()
some_object.some_method_1 = Mock(return_value=foo)
self.assertEqual(an_expected_value, some_object.some_method_2())
This works totally fine and the script runs without error and the test result is also ok.
A fast Intro about Mypy: Mypy is a python type checking tool which can run and check if the script is written correctly based on variable types. It does this process by some criteria such as type annotations, variable assignments and libraries stub files. This process is done without interpreting and running the code.
What is the Problem?
When I try to check my code with mypy, it gives error for this line:
some_object.some_method_1 = Mock(return_value=foo)
The error indicates that I am not allowed to assign an object to a callable in python. Sometimes mypy does not report real errors and I doubt if this is the case. Especially because I can run my code with no problem.
Now, my question is, have I done my job wrong or just the mypy report is wrong? If I have done wrong, how can I implement the same scenario in a correct manner?

Python 'call_command' mock is used in other tests as well

Using Django 1.10 and python 3.5.1.
I'm trying to mock 'call_command' function to throw an exception. The problem is that seems like the moment it gets the 'side_effect' function - it keeps to it also for other tests. What am I doing wrong or how can I 'revert' the side_effect from that function?
In this example, after running one of the tests, all other tests that run afterwards will throw the same exception even if it's not supposed to throw exception in that test.
def test_run_migrations_raise_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=Exception('e message')):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
def test_run_migrations_raise_flow_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=FlowException(500, 'fe message', {'a': 1})):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)

You should not patch a function that is in your module-local (i.e. Python's "global" - which is actually "module") namespace.
When in Python you do
from module.that import this
this becomes a variable on the module that contains the import statement. Any changes to "module.that.this" will affect the object pointed in the other module, but using only this will still reefer to the original object.
Perhaps your code is not exactly as you show us, or maybe "mock.pacth" can find out that the module-local call_command is pointing to django.core.management.call_command in the other module when it makes the patch - but not when reversing the patch. The fact is your module-local name call_command is being changed.
You can fix that by simply changing your code to not bind a module variable directly to the function you want to change:
from django.core import management
def test_run_migrations_raise_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=Exception('e message')):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
management.call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
I hope you can understand that and solve this problem. Now, that said, this use of mock makes no sense at all: the idea of using mock is that some callable used indirectly by code you call within the code-block that applies the patch does not have the original effect - so the intermetiate code can run and be tested. You are calling directly the mock object - so it will have none of the original code - calling call_command('run_data_migrations') runs no code on your code base at all, and thus, there is nothing there to test. It just calls the mocked instance, and it will not change the status of anything that could be detected with check_migrations_called.

How to mock Python classes when nested several dependencies deep

If I have the following architecture...
Please note the edits below. It occurred to me (after some recent refactoring) that there are actually three classes in three different files. Sorry that the file/class names are getting ridiculous. I assure you those are not the real names. :)
main_class.py
class MainClass(object):
def do_some_stuff(self):
dependent_class = DependentClass()
dependent_class.py
class DependentClass(object):
def __init__():
dependent_dependent_class = DependentDependentClass()
dependent_dependent_class.do_dependent_stuff()
dependent_dependent_class.py
class DependentDependentClass(object):
def do_dependent_stuff(self):
print "I'm gonna do production stuff that I want to mock"
print "Like access a database or interact with a remote server"
class MockDependentDependentClass(object):
def do_dependent_stuff(self):
print "Respond as if the production stuff was all successful."
and I want to call main_class.do_some_stuff during testing but, during its execution, I want instances of DependentDependentClass replaced with MockDependentDependentClass how can I do that pythonically using best practices.
Currently, the best thing I could come up with is to conditionally instantiate one class or the other based on the presence/value of an environment variable. It certainly works but is pretty dirty.
I spent some time reading about the unittest.mock and mock.patch functions and they seem like they might be able to help but each description that I could wrap my head around seemed to be a little different than my actual use case.
The key is that I don't want to define mock return values or attributes but that I want the namespace changed, globally, I guess, such that when my application thinks it is instantiating DependentClass it is actually instantiating MockDependentClass.
The fact that I can't find any examples of anyone doing exactly this means one of two things:
It's because I'm doing it in a very dumb/naive way.
I'm doing something so genius no else has ever encountered it.
... I assume it's number 1...
Full disclosure, unit testing is not something with which I am skilled. It's an effort that my internal tools development team is trying to catch up to step our game up a bit. It's possible that I'm not thinking about testing correctly.
Any thoughts would be most welcome. Thank you, in advance!
SOLUTION!!!
Thanks to #de1 for the help. Given my clever architecture shown above the following accomplishes what I want.
The following code is located in main_class.py
import dependent_class
from dependent_dependent_class import MockDependentDependentClass
with patch.object(dependent_class, "DependentDependentClass", MockDependentDependentClass):
main_class = MainClass()
main_class.do_some_stuff()
The code seems to (and hell if I know how it's doing this) manipulate the namespace within the module dependent_class so that, while inside the with block (that's a context manager for anyone who is hung up on that part) anything referring to the class object DependentDependentClass will actually be referencing MockDependentDependentClass.

The mock module does indeed seem to be a good fit in this case. You can specify the mock (your mock) to use when calling the various patch methods.
If you are importing only the class rather than the module you can patch the imported DependentDependentClass in DependentClass:
import .DependentClass as dependent_class
from .DependentDependentClass import MockDependentDependentClass
with patch.object(dependent_class, 'DependentDependentClass', MockDependentDependentClass):
# do something while class is patched
Alternatively:
with patch('yourmodule.DependentClass.DependentDependentClass', MockDependentDependentClass):
# do something while class is patched
or the following will only work if you are accessing the class via a module or import it after it is being patched:
with patch('yourmodule.DependentDependentClass.DependentDependentClass', MockDependentDependentClass):
# do something while class is patched
Just bare in mind what object is being patched, when.
Note: you might find it less confusing naming your files in lower case, slightly different to the embedded class(es).
Note 2: If you need to mock a dependency of a dependency of the module under test then it might suggest that you are not testing at the right level.

Proper use of `isinstance(obj, class)`

As I write it, it seems almost surreal to me that I'm actually experiencing this problem.
I have a list of objects. Each of these objects are of instances of an Individual class that I wrote.
Thus, conventional wisdom says that isinstance(myObj, Individual) should return True. However, this was not the case. So I thought that there was a bug in my programming, and printed type(myObj), which to my surprise printed instance and myObj.__class__ gave me Individual!
>>> type(pop[0])
<type 'instance'>
>>> isinstance(pop[0], Individual) # with all the proper imports
False
>>> pop[0].__class__
Genetic.individual.Individual
I'm stumped! What gives?
EDIT: My Individual class
class Individual:
ID = count()
def __init__(self, chromosomes):
self.chromosomes = chromosomes[:] # managed as a list as order is used to identify chromosomal functions (i.e. chromosome i encodes functionality f)
self.id = self.ID.next()
# other methods

This error indicates that the Individual class somehow got created twice. You created pop[0] with one version of Instance, and are checking for instance with the other one. Although they are pretty much identical, Python doesn't know that, and isinstance fails. To verify this, check whether pop[0].__class__ is Individual evaluates to false.
Normally classes don't get created twice (unless you use reload) because modules are imported only once, and all class objects effectively remain singletons. However, using packages and relative imports can leave a trap that leads to a module being imported twice. This happens when a script (started with python bla, as opposed to being imported from another module with import bla) contains a relative import. When running the script, python doesn't know that its imports refer to the Genetic package, so it processes its imports as absolute, creating a top-level individual module with its own individual.Individual class. Another other module correctly imports the Genetic package which ends up importing Genetic.individual, which results in the creation of the doppelganger, Genetic.individual.Individual.
To fix the problem, make sure that your script only uses absolute imports, such as import Genetic.individual even if a relative import like import individual appears to work just fine. And if you want to save on typing, use import Genetic.individual as individual. Also note that despite your use of old-style classes, isinstance should still work, since it predates new-style classes. Having said that, it would be highly advisable to switch to new-style classes.

You need to use new-style classes that inherit from
class ClassName(object):
pass
From your example, you are using old-style classes that inherit from
class Classname:
pass
EDIT: As #user4815162342 said,
>>> type(pop[0])
<type 'instance'>
is caused by using an old-style class, but this is not the cause of your issues with isinstance. You should instead make sure you don't create the class in more than one place, or if you do, use distinct names. Importing it more than once should not be an issue.

is there a way to look at the code behind an __enter__() function in the python interpreter?

question pretty much says it all.
i'd like to look at the code in this fashion:
>>>f = open("x.txt")
>>>print contents of f.__enter__() #<- how can I do this?

No. (Other than looking at the Python source code.)
>>> f = open("x.txt")
>>> f.__enter__
<built-in method __enter__ of file object at 0x022E4E90>
So the implementation of __enter__ is somewhere inside Python's C code.
It's actually in Objects/fileobject.c which you can find in the Python source tree [note: I think that's the currently-latest thing on the 2.7 branch; there's probably a better way to link to it] and looking at the code you'll see that actually f.__enter__ returns f itself. Of course that's just what happens in this particular case; other objects' __enter__ methods will do entirely different things.
In this case, it happens that the __enter__ method is native code. In others it may be Python code, but you still can't generally see it from inside Python.
>>> import decimal
>>> decimal.localcontext().__enter__
<bound method _ContextManager.__enter__ of <decimal._ContextManager object at 0x02192B50>>
That's Python bytecode rather than native code. You can see the bytecode:
import dis
dis.dis(decimal.localcontext().__enter__)
but the original Python source code is not guaranteed to be available. But you can try:
import inspect
print inspect.getsource(decimal.localcontext().__enter__)
which will sometimes do what you want.

You can't, at least not from an abritary callable (or any other) object. You can try to find the source code, and there's even a function in the standard library that can do this in many cases. However, the I/O modules are propably written in C, so you'd have to go and search the repository.