Develop Reference

How to mock open differently depending on the parameters passed to open()

My question is how to mock open in python, such that it reacts differently depending on the argument open() is called with. These are some different scenario's that should be possible:
open a mocked file; read preset contents, the basic scenario.
open two mocked files and have them give back different values for the read() method. The order in which the files are opened/read from should not influence the results.
Furthermore, if I call open('actual_file.txt') to open an actual file, I want the actual file to be opened, and not a magic mock with mocked behavior. Or if I just don't want the access to a certain file mocked, but I do want other files to be mocked, this should be possible.
I know about this question: Python mock builtin 'open' in a class using two different files.
But that answer only partially answers up to the second requirement. The part about order independent results is not included and it does not specify how to mock only some calls, and allow other calls to go through to the actual files (default behavior).

A bit late, but I just recently happened upon the same need, so I'd like to share my solution, based upon this answer from the referred-to question:
import pytest
from unittest.mock import mock_open
from functools import partial
from pathlib import Path
mock_file_data = {
"file1.txt": "some text 1",
"file2.txt": "some text 2",
# ... and so on ...
}
do_not_mock: {
# If you need exact match (see note in mocked_file(),
# you should replace these with the correct Path() invocations
"notmocked1.txt",
"notmocked2.txt",
# ... and so on ...
}
# Ref: https://stackoverflow.com/a/38618056/149900
def mocked_file(m, fn, *args, **kwargs):
m.opened_file = Path(fn)
fn = Path(fn).name # If you need exact path match, remove this line
if fn in do_not_mock:
return open(fn, *args, **kwargs)
if fn not in mock_file_data:
raise FileNotFoundError
data = mock_file_data[fn]
file_obj = mock_open(read_data=data).return_value
file_obj.__iter__.return_value = data.splitlines(True)
return file_obj
def assert_opened(m, fn):
fn = Path(fn)
assert m.opened_file == fn
#pytest.fixture()
def mocked_open(mocker):
m = mocker.patch("builtins.open")
m.side_effect = partial(mocked_file, m)
m.assert_opened = partial(assert_opened, m)
return m
def test_something(mocked_open):
...
# Something that should NOT invoke open()
mocked_open.assert_not_called()
...
# Something that SHOULD invoke open()
mocked_open.assert_called_once()
mocked_open.assert_opened("file1.txt")
# Depends on how the tested unit handle "naked" filenames,
# you might have to change the arg to:
# Path.cwd() / "file1.txt"
# ... and so on ...
Do note that (1) I am using Python 3, and (2) I am using pytest.

This can be done by following the approach in the other question's accepted answer (Python mock builtin 'open' in a class using two different files) with a few alterations.
First off. Instead of just specifying a side_effect that can be popped. We need to make sure the side_effect can return the correct mocked_file depending on the parameters used with the open call.
Then if the file we wish to open is not among the files we wish to mock, we instead return the original open() of the file instead of any mocked behavior.
The code below demonstrates how this can be achieved in a clean, repeatable way. I for instance have this code inside of a file that provides some utility functions to make testing easier.
from mock import MagicMock
import __builtin__
from mock import patch
import sys
# Reference to the original open function.
g__test_utils__original_open = open
g__test_utils__file_spec = None
def create_file_mock(read_data):
# Create file_spec such as in mock.mock_open
global g__test_utils__file_spec
if g__test_utils__file_spec is None:
# set on first use
if sys.version_info[0] == 3:
import _io
g__test_utils__file_spec = list(set(dir(_io.TextIOWrapper)).union(set(dir(_io.BytesIO))))
else:
g__test_utils__file_spec = file
file_handle = MagicMock(spec=g__test_utils__file_spec)
file_handle.write.return_value = None
file_handle.__enter__.return_value = file_handle
file_handle.read.return_value = read_data
return file_handle
def flexible_mock_open(file_map):
def flexible_side_effect(file_name):
if file_name in file_map:
return file_map[file_name]
else:
global g__test_utils__original_open
return g__test_utils__original_open(file_name)
global g__test_utils__original_open
return_value = MagicMock(name='open', spec=g__test_utils__original_open)
return_value.side_effect = flexible_side_effect
return return_value
if __name__ == "__main__":
a_mock = create_file_mock(read_data="a mock - content")
b_mock = create_file_mock(read_data="b mock - different content")
mocked_files = {
'a' : a_mock,
'b' : b_mock,
}
with patch.object(__builtin__, 'open', flexible_mock_open(mocked_files)):
with open('a') as file_handle:
print file_handle.read() # prints a mock - content
with open('b') as file_handle:
print file_handle.read() # prints b mock - different content
with open('actual_file.txt') as file_handle:
print file_handle.read() # prints actual file contents
This borrows some code straight from the mock.py (python 2.7) for the creating of the file_spec.
side note: if there's any body that can help me in how to hide these globals if possible, that'd be very helpful.

How to create a function which accepts each 1 argument as either one type or the other?

I want to create a function which accepts 3 arguments, and each argument can be either something to pass to the open function, which my function will pass to open and then work with it, or it can be a file object, then my function just works with it without the need to open it.
So, it's something like this
def work_with_files(file1, file2, file3):
files_to_close = []
file_objects = []
try:
for file in (file1, file2, file3):
if not is_file_object(file):
opened_file = open(file.argument_for_open)
file_objects.append(opened_file)
files_to_close.append(opened_file)
else:
file_objects.append(file.file_object)
work_with_files(file_objects)
finally:
for file in files_to_close:
file.close()
In C++ I would use boost::variant, in haskell I would probably use Either. What's the most pythonic way to do this? What should I pass as arguments, and how should is_file_object work?

You could use contextlib.ExitStack (so fileobjs are automatically closed inside a with) and generalise your function, if it's a string - then pass it to open otherwise assume it's already a file-like object that has a suitable .close()...
from contextlib import ExitStack
def work_with_files(*args):
with ExitStack() as stack:
fileobjs = [
stack.enter_context(open(obj) if isinstance(obj, str) else obj)
for obj in args
]
# do something with fileobjs list
Where the do something might be like:
from itertools import chain
for line in chain.from_iterable(fileobjs):
print(line)

You can use isinstance function :
isinstance(file_name,file)
isinstance(object, classinfo)
Return true if the object argument is an instance of the classinfo argument, or of a (direct, indirect or virtual) subclass thereof. Also return true if classinfo is a type object (new-style class) and object is an object of that type or of a (direct, indirect or virtual) subclass thereof.
Also as :
File objects are implemented using C’s stdio package and can be created with the built-in open() function.
You can use a try-except statement to deal with file objects:
try:
open('file-name')
except IOError:
#do stuff

If you don't want to check the parameter type, you could use the try-except construct to do exactly what you want:
def work_with_files(file1, file2, file3):
# ensure that each file* variable is a file-like object
try: file1 = open(file1) # if it opens, then file1 was a filepath
except: pass # else, file1 was already a file-like object
try: file2 = open(file2)
except: pass
try: file3 = open(file3)
except: pass
do_stuff_with_files(file1, file2, file3)
for f in (file1, file2, file3): f.close() # close the files

How does open() work with and without `with`?

I'd like to write a function similar to open. I'd like to be able to call it with with, but also without with.
When I use contextlib.contextmanager, it makes my function work fine with with:
#contextmanager
def versioned(file_path, mode):
version = calculate_version(file_path, mode)
versioned_file = open(file_path, mode)
yield versioned_file
versioned_file.close()
So, I use it like this:
with versioned('file.txt', 'r') as versioned_file:
versioned_file.write(...)
How do I use it without with:
versioned_file = versioned('file.txt', 'r')
versioned_file.write(...)
versioned_file.close()
It complains:
AttributeError: 'GeneratorContextManager' object has no attribute 'write'

The problem is that contextmanager only provides exactly that; a context manager to be used in the with statement. Calling the function does not return the file object, but a special context generator which provides the __enter__ and __exit__ functions. If you want both the with statement and “normal” assignments to work, then you will have to have some object as the return value from your function that is fully usable and also provides the context functions.
You can do this pretty easily by creating your own type, and manually providing the context functions:
class MyOpener:
def __init__ (self, filename):
print('Opening {}'.format(filename))
def close (self):
print('Closing file.')
def write (self, text):
print('Writing "{}"'.format(text))
def __enter__ (self):
return self
def __exit__ (self, exc_type, exc_value, traceback):
self.close()
>>> f = MyOpener('file')
Opening file
>>> f.write('foo')
Writing "foo"
>>> f.close()
Closing file.
>>> with MyOpener('file') as f:
f.write('foo')
Opening file
Writing "foo"
Closing file.

You have this:
#contextmanager
def versioned(file_path, mode):
# some setup code
yield versioned_file
# some teardown code
Your basic problem of course is that what you yield from the context manager comes out of the with statement via as, but is not the object returned by your function. You want a function that returns something that behaves like the object open() returns. That is to say, a context manager object that yields itself.
Whether you can do that depends what you can do with the type of versioned_file. If you can't change it then you're basically out of luck. If you can change it then you need to implement the __enter__ and __exit__ functions as specified in PEP 343.
In your example code, though, it already has it, and your teardown code is the same as what it does itself on context exit already. So don't bother with contextlib at all, just return the result of open().
For other examples where you do need __enter__ and __exit__, if you like the contextlib style (and who doesn't?) you can bridge the two things. Write a function context that's decorated with #contextmanager and yields self. Then implement:
def __enter__(self):
self.context = context() # if context() is a method use a different name!
return self.context.__enter__()
def __exit__(self, *args):
return self.context.__exit__(*args)
It's basically up to you whether you find this better or worse than separating out the setup code into __enter__ and the teardown code into __exit__. I generally find it better.

Do you really need to use contextlib.contextmanager?
If you have a custom stream you would want to use Poke's solution.
But since you are just returning a file object, why go through all the hassle:
def versioned(file_path, mode):
version = calculate_version(file_path, mode)
return open(file_path, mode)
with versioned('test.conf', 'r') as stream:
print stream.read()
f = versioned('test.conf', 'r')
print f.read()
f.close()
Both will work perfectly fine :)

How do I mock an open used in a with statement (using the Mock framework in Python)?

How do I test the following code with unittest.mock:
def testme(filepath):
with open(filepath) as f:
return f.read()

Python 3
Patch builtins.open and use mock_open, which is part of the mock framework. patch used as a context manager returns the object used to replace the patched one:
from unittest.mock import patch, mock_open
with patch("builtins.open", mock_open(read_data="data")) as mock_file:
assert open("path/to/open").read() == "data"
mock_file.assert_called_with("path/to/open")
If you want to use patch as a decorator, using mock_open()'s result as the new= argument to patch can be a little bit weird. Instead, use patch's new_callable= argument and remember that every extra argument that patch doesn't use will be passed to the new_callable function, as described in the patch documentation:
patch() takes arbitrary keyword arguments. These will be passed to the Mock (or new_callable) on construction.
#patch("builtins.open", new_callable=mock_open, read_data="data")
def test_patch(mock_file):
assert open("path/to/open").read() == "data"
mock_file.assert_called_with("path/to/open")
Remember that in this case patch will pass the mocked object as an argument to your test function.
Python 2
You need to patch __builtin__.open instead of builtins.open and mock is not part of unittest, you need to pip install and import it separately:
from mock import patch, mock_open
with patch("__builtin__.open", mock_open(read_data="data")) as mock_file:
assert open("path/to/open").read() == "data"
mock_file.assert_called_with("path/to/open")

The way to do this has changed in mock 0.7.0 which finally supports mocking the python protocol methods (magic methods), particularly using the MagicMock:
http://www.voidspace.org.uk/python/mock/magicmock.html
An example of mocking open as a context manager (from the examples page in the mock documentation):
>>> open_name = '%s.open' % __name__
>>> with patch(open_name, create=True) as mock_open:
... mock_open.return_value = MagicMock(spec=file)
...
... with open('/some/path', 'w') as f:
... f.write('something')
...
<mock.Mock object at 0x...>
>>> file_handle = mock_open.return_value.__enter__.return_value
>>> file_handle.write.assert_called_with('something')

With the latest versions of mock, you can use the really useful mock_open helper:
mock_open(mock=None, read_data=None)
A helper function to create a
mock to replace the use of open. It works for open called directly or
used as a context manager.
The mock argument is the mock object to configure. If None (the
default) then a MagicMock will be created for you, with the API
limited to methods or attributes available on standard file handles.
read_data is a string for the read method of the file handle to
return. This is an empty string by default.
>>> from mock import mock_open, patch
>>> m = mock_open()
>>> with patch('{}.open'.format(__name__), m, create=True):
... with open('foo', 'w') as h:
... h.write('some stuff')
>>> m.assert_called_once_with('foo', 'w')
>>> handle = m()
>>> handle.write.assert_called_once_with('some stuff')

To use mock_open for a simple file read() (the original mock_open snippet already given on this page is geared more for write):
my_text = "some text to return when read() is called on the file object"
mocked_open_function = mock.mock_open(read_data=my_text)
with mock.patch("__builtin__.open", mocked_open_function):
with open("any_string") as f:
print f.read()
Note as per docs for mock_open, this is specifically for read(), so won't work with common patterns like for line in f, for example.
Uses python 2.6.6 / mock 1.0.1

The top answer is useful but I expanded on it a bit.
If you want to set the value of your file object (the f in as f) based on the arguments passed to open() here's one way to do it:
def save_arg_return_data(*args, **kwargs):
mm = MagicMock(spec=file)
mm.__enter__.return_value = do_something_with_data(*args, **kwargs)
return mm
m = MagicMock()
m.side_effect = save_arg_return_array_of_data
# if your open() call is in the file mymodule.animals
# use mymodule.animals as name_of_called_file
open_name = '%s.open' % name_of_called_file
with patch(open_name, m, create=True):
#do testing here
Basically, open() will return an object and with will call __enter__() on that object.
To mock properly, we must mock open() to return a mock object. That mock object should then mock the __enter__() call on it (MagicMock will do this for us) to return the mock data/file object we want (hence mm.__enter__.return_value). Doing this with 2 mocks the way above allows us to capture the arguments passed to open() and pass them to our do_something_with_data method.
I passed an entire mock file as a string to open() and my do_something_with_data looked like this:
def do_something_with_data(*args, **kwargs):
return args[0].split("\n")
This transforms the string into a list so you can do the following as you would with a normal file:
for line in file:
#do action

I might be a bit late to the game, but this worked for me when calling open in another module without having to create a new file.
test.py
import unittest
from mock import Mock, patch, mock_open
from MyObj import MyObj
class TestObj(unittest.TestCase):
open_ = mock_open()
with patch.object(__builtin__, "open", open_):
ref = MyObj()
ref.save("myfile.txt")
assert open_.call_args_list == [call("myfile.txt", "wb")]
MyObj.py
class MyObj(object):
def save(self, filename):
with open(filename, "wb") as f:
f.write("sample text")
By patching the open function inside the __builtin__ module to my mock_open(), I can mock writing to a file without creating one.
Note: If you are using a module that uses cython, or your program depends on cython in any way, you will need to import cython's __builtin__ module by including import __builtin__ at the top of your file. You will not be able to mock the universal __builtin__ if you are using cython.

If you don't need any file further, you can decorate the test method:
#patch('builtins.open', mock_open(read_data="data"))
def test_testme():
result = testeme()
assert result == "data"

To patch the built-in open() function with unittest:
This worked for a patch to read a json config.
class ObjectUnderTest:
def __init__(self, filename: str):
with open(filename, 'r') as f:
dict_content = json.load(f)
The mocked object is the io.TextIOWrapper object returned by the open() function
#patch("<src.where.object.is.used>.open",
return_value=io.TextIOWrapper(io.BufferedReader(io.BytesIO(b'{"test_key": "test_value"}'))))
def test_object_function_under_test(self, mocker):

I'm using pytest in my case, and the good news is that in Python 3 the unittest library can also be imported and used without issue.
Here is my approach. First, I create a conftest.py file with reusable pytest fixture(s):
from functools import cache
from unittest.mock import MagicMock, mock_open
import pytest
from pytest_mock import MockerFixture
class FileMock(MagicMock):
def __init__(self, mocker: MagicMock = None, **kwargs):
super().__init__(**kwargs)
if mocker:
self.__dict__ = mocker.__dict__
# configure mock object to replace the use of open(...)
# note: this is useful in scenarios where data is written out
_ = mock_open(mock=self)
#property
def read_data(self):
return self.side_effect
#read_data.setter
def read_data(self, mock_data: str):
"""set mock data to be returned when `open(...).read()` is called."""
self.side_effect = mock_open(read_data=mock_data)
#property
#cache
def write_calls(self):
"""a list of calls made to `open().write(...)`"""
handle = self.return_value
write: MagicMock = handle.write
return write.call_args_list
#property
def write_lines(self) -> str:
"""a list of written lines (as a string)"""
return ''.join([c[0][0] for c in self.write_calls])
#pytest.fixture
def mock_file_open(mocker: MockerFixture) -> FileMock:
return FileMock(mocker.patch('builtins.open'))
Where I decided to make the read_data as a property, in order to be more pythonic. It can be assigned in a test function with whatever data that open() needs to return.
In my test file, named something like test_it_works.py, I have a following test case to confirm intended functionality:
from unittest.mock import call
def test_mock_file_open_and_read(mock_file_open):
mock_file_open.read_data = 'hello\nworld!'
with open('/my/file/here', 'r') as in_file:
assert in_file.readlines() == ['hello\n', 'world!']
mock_file_open.assert_called_with('/my/file/here', 'r')
def test_mock_file_open_and_write(mock_file_open):
with open('/out/file/here', 'w') as f:
f.write('hello\n')
f.write('world!\n')
f.write('--> testing 123 :-)')
mock_file_open.assert_called_with('/out/file/here', 'w')
assert call('world!\n') in mock_file_open.write_calls
assert mock_file_open.write_lines == """\
hello
world!
--> testing 123 :-)
""".rstrip()
Check out the gist here.

Sourced from a github snippet to patch read and write functionality in python.
The source link is over here
import configparser
import pytest
simpleconfig = """[section]\nkey = value\n\n"""
def test_monkeypatch_open_read(mockopen):
filename = 'somefile.txt'
mockopen.write(filename, simpleconfig)
parser = configparser.ConfigParser()
parser.read(filename)
assert parser.sections() == ['section']
def test_monkeypatch_open_write(mockopen):
parser = configparser.ConfigParser()
parser.add_section('section')
parser.set('section', 'key', 'value')
filename = 'somefile.txt'
parser.write(open(filename, 'wb'))
assert mockopen.read(filename) == simpleconfig

SIMPLE #patch with assert
If you're wanting to use #patch. The open() is called inside the handler and is read.
#patch("builtins.open", new_callable=mock_open, read_data="data")
def test_lambda_handler(self, mock_open_file):
lambda_handler(event, {})

How to wrap built-in methods in Python? (or 'how to pass them by reference')

I want to wrap the default open method with a wrapper that should also catch exceptions. Here's a test example that works:
truemethod = open
def fn(*args, **kwargs):
try:
return truemethod(*args, **kwargs)
except (IOError, OSError):
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
open = fn
I want to make a generic method of it:
def wrap(method, exceptions = (OSError, IOError)):
truemethod = method
def fn(*args, **kwargs):
try:
return truemethod(*args, **kwargs)
except exceptions:
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
method = fn
But it doesn't work:
>>> wrap(open)
>>> open
<built-in function open>
Apparently, method is a copy of the parameter, not a reference as I expected. Any pythonic workaround?

The problem with your code is that inside wrap, your method = fn statement is simply changing the local value of method, it isn't changing the larger value of open. You'll have to assign to those names yourself:
def wrap(method, exceptions = (OSError, IOError)):
def fn(*args, **kwargs):
try:
return method(*args, **kwargs)
except exceptions:
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
return fn
open = wrap(open)
foo = wrap(foo)

Try adding global open. In the general case, you might want to look at this section of the manual:
This module provides direct access to all ‘built-in’ identifiers of Python; for example, __builtin__.open is the full name for the built-in function open(). See chapter Built-in Objects.
This module is not normally accessed explicitly by most applications, but can be useful in modules that provide objects with the same name as a built-in value, but in which the built-in of that name is also needed. For example, in a module that wants to implement an open() function that wraps the built-in open(), this module can be used directly:
import __builtin__
def open(path):
f = __builtin__.open(path, 'r')
return UpperCaser(f)
class UpperCaser:
'''Wrapper around a file that converts output to upper-case.'''
def __init__(self, f):
self._f = f
def read(self, count=-1):
return self._f.read(count).upper()
# ...
CPython implementation detail: Most modules have the name __builtins__ (note the 's') made available as part of their globals. The value of __builtins__ is normally either this module or the value of this modules’s __dict__ attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python.

you can just add return fn at the end of your wrap function and then do:
>>> open = wrap(open)
>>> open('bhla')
Traceback (most recent call last):
File "<pyshell#24>", line 1, in <module>
open('bhla')
File "<pyshell#18>", line 7, in fn
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
SystemExit: Can't open 'bhla'. Error #2: No such file or directory