Develop Reference

Mocking global variable which is a module

I am trying to write a test for a small module which makes use of boto3, there are a few global imports that I want to mock, but it seems like the act of mocking the module means the global module is loaded and fails.
I do wonder if this highlights the fact the code should be refactored but I am not totally sure on this, the code is deployed as an AWS lambda function and the global variables are created outside the handler function.
Here are the global variables, defined outside the handler:
QUEUE_NAME = os.getenv("QUEUE_NAME", None)
TABLE_NAME = os.getenv("TABLE_NAME", None)
sqs_client = boto3.resource("sqs")
dynamo_client = boto3.client("dynamodb")
queue = sqs_client.get_queue_by_name(QueueName=QUEUE_NAME)
When I try to mock the sqs_client object I get an error, similarly I get that error if I import the module into my test module. Here is the stub of the failing test:
#mock.patch.dict(os.environ, {"QUEUE_NAME": "QUEUE1", "TABLE_NAME": "TABLE1"})
#mock.patch("queue_processor.sqs_client.get_queue_by_name", return_value=None)
#mock.patch("queue_processor.dynamo_client.get_item", return_value=None)
def test_queue_entry(mocker):
The error appears to be because the mock is not being used so a call is made to AWS:
E botocore.errorfactory.QueueDoesNotExist: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the GetQueueUrl operation: The specified queue does not exist for this wsdl version.
My question is twofold, how do I avoid this error, in which the SQS client tries to call AWS rather than using the mock, and secondly, is this actually something I want to fix or would the code be better refactored to avoid global variables?

Pytest mocker failing to find Path

I am working with someone else's testing code, and they make extensive use of mocker. The problem is that I changed the underlying code so it tests for the existence of a file using Path ().is_file.
Now I need to mock Path ().is_file so it returns True. I tried this:
from pathlib import Path
#pytest.fixture(scope="function")
def mock_is_file (mocker):
# mock the AlignDir existence validation
mocker.patch ('Path.is_file')
return True
I'm getting this error:
E ModuleNotFoundError: No module named 'Path'
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/mock.py:1161: ModuleNotFoundError
What is the correct way to patch Path.is_file()?

Mock will import the object given the string, so you need to patch with a fully qualified name.
You should probably also set the return_value so that the call returns a boolean, rather than returning another generated mock. The return value in the fixture itself is not needed.
mocker.patch("pathlib.Path.is_file", return_value=True)
You also don't need to import the thing that you're mocking at the top of the test module like that, mock itself will import it when patching.

Mocking an entire function with parameters in Python

I have this Python code normally running on a distant server that I want to test locally:
from af.data import Variable
def create_file():
main_client = file.Client()
directory = main_client.create(Variable.get("DIR_NAME"))
As I am developing locally and do not have access to the remote service providing the af.data.Variable class, I'd like to mock the Variable.get(str) function, but I'd like to be able—in my own mock—to return some value based on the passed the str parameter. So far, I've found only ways to mock a function to some pre-defined static values using side_effect of unittest.
How can I do that?

after the Peter's comment, here is the solution (pretty straightforward):
from unittest.mock import Mock
from af.data import Variable
import os
def side_effect(arg):
"""Put here whatever logic you want to put in your mock, here's an example of reading a system environment variable"""
return os.getenv(arg)
mock = Mock()
mock.get.side_effect = side_effect
Variable = mock
"""Then just use the Variable.get method as you would do in production environment, it should be mocked to the side_effect function declared above"""

Testing constants declarations using pytest

We have a Python 3.7 application that has a declared constants.py file that has this form:
APP_CONSTANT_1 = os.environ.get('app-constant-1-value')
In a test.py we were hoping to test the setting of these constants using something like this (this is highly simplified but represents the core issue):
class TestConfig:
"""General config tests"""
#pytest.fixture
def mock_os_environ(self, monkeypatch):
""" """
def mock_get(*args, **kwargs):
return 'test_config_value'
monkeypatch.setattr(os.environ, "get", mock_get)
def test_mock_env_vars(self, mock_os_environ):
import constants
assert os.environ.get('app-constant-1-value') == 'test_config_value' #passes
assert constants.APP_CONSTANT_1 == 'test_config_value' #fails
The second assertion fails as constants.constants.APP_CONSTANT_1 is None. Turns out that the constants.py seems to be loaded during pytest's 'collecting' phase and thus is already set by the time the test is run.
What are we missing here? I feel like there is a simple way to resolve this in pytest but haven't yet discovered the secret. Is there some way to avoid loading the constants file prior to the tests being run? Any ideas are appreciated.

The problem is most likely that constants has been loaded before. To make sure it gets the patched value, you have to reload it:
import os
from importlib import reload
import pytest
import constants
class TestConfig:
"""General config tests"""
#pytest.fixture
def mock_os_environ(self, monkeypatch):
""" """
monkeypatch.setenv('app-constant-1-value', 'test_config_value')
reload(constants)
def test_mock_env_vars(self, mock_os_environ):
assert os.environ.get('app-constant-1-value') == 'test_config_value'
assert app.APP_CONSTANT_1 == 'test_config_value'
Note that I used monkeypatch.setenv to specifically set the variable you need. If you don't need to change all environment variables, this is easier to use.

Erm, I would avoid using constants. You can subclass os.environment for a start, and then use a mocked subclass for your unit tests, so you can have my_env.unique_env as a member variable. You can then use eg. import json to use a json configuration file without getting involved with hard coded python.
The subclass can then hold the relevant variables (or methods if you prefer)
Being able to add a facade to os.environment provides you with the abstraction you are looking for, without any of the problems.
Even is one is using a legacy/larger project, the advantage of using an adapter for access to the environment must be apparent.
Since you are writing unit tests, there is an opportunity to use an adapter class in both the tests and the functions being tested.

how to share a variable across modules for all tests in py.test

I have multiple tests run by py.test that are located in multiple classes in multiple files.
What is the simplest way to share a large dictionary - which I do not want to duplicate - with every method of every class in every file to be used by py.test?
In short, I need to make a "global variable" for every test. Outside of py.test, I have no use for this variable, so I don't want to store it in the files being tested. I made frequent use of py.test's fixtures, but this seems overkill for this need. Maybe it's the only way?

Update: pytest-namespace hook is deprecated/removed. Do not use. See #3735 for details.
You mention the obvious and least magical option: using a fixture. You can apply it to entire modules using pytestmark = pytest.mark.usefixtures('big_dict') in your module, but then it won't be in your namespace so explicitly requesting it might be best.
Alternatively you can assign things into the pytest namespace using the hook:
# conftest.py
def pytest_namespace():
return {'my_big_dict': {'foo': 'bar'}}
And now you have pytest.my_big_dict. The fixture is probably still nicer though.

There are tons of things I love about py.test, but one thing I absolutely HATE is how poorly it plays with code intelligence tools. I disagree that an autouse fixture to declare a variable is the "most clear" method in this case because not only does it completely baffle my linter, but also anyone else who is not familiar with how py.test works. There is a lot of magic there, imo.
So, one thing you can do that doesn't make your linter explode and doesn't require TestCase boilerplate is to create a module called globals. Inside this module, stub the names of the things you want global to {} or None and import the global module into your tests. Then in your conftest.py file, use the py.test hooks to set (or reset) your global variable(s) as appropriate. This has the advantage of giving you the stub to work with when building tests and the full data for the tests at runtime.
For example, you can use the pytest_configure() hook to set your dict right when py.test starts up. Or, if you wanted to make sure the data was pristine between each test, you could autouse a fixture to assign your global variable to your known state before each test.
# globals.py
my_data = {} # Create a stub for your variable
# test_module.py
import globals as gbl
def test_foo():
assert gbl.my_data['foo'] == 'bar' # The global is in the namespace when creating tests
# conftest.py
import globals as gbl
my_data = {'foo': 'bar'} # Create the master copy in conftest
#pytest.fixture(autouse=True)
def populate_globals():
gbl.my_data = my_data # Assign the master value to the global before each test
One other advantage to this approach is you can use type hinting in your globals module to give you code completion on the global objects in your test, which probably isn't necessary for a dict but I find it handy when I am using an object (such as webdriver). :)

I'm suprised no answer mentioned caching yet: since version 2.8, pytest has a powerful cache mechanism.
Usage example
#pytest.fixture(autouse=True)
def init_cache(request):
data = request.config.cache.get('my_data', None)
data = {'spam': 'eggs'}
request.config.cache.set('my_data', data)
Access the data dict in tests via builtin request fixture:
def test_spam(request):
data = request.config.cache.get('my_data')
assert data['spam'] == 'eggs'
Sharing the data between test runs
The cool thing about request.cache is that it is persisted on disk, so it can be even shared between test runs. This comes handy when you running tests distributed (pytest-xdist) or have some long-running data generation which does not change once generated:
#pytest.fixture(autouse=True)
def generate_data(request):
data = request.config.cache.get('my_data', None)
if data is None:
data = long_running_generation_function()
request.config.cache.set('my_data', data)
Now the tests won't need to recalculate the value on different test runs unless you clear the cache on disk explicitly. Take a look what's currently in the cache:
$ pytest --cache-show
...
my_data contains:
{'spam': 'eggs'}
Rerun the tests with the --cache-clear flag to delete the cache and force the data to be recalculated. Or just remove the .pytest_cache directory in the project root dir.
Where to go from here
The related section in pytest docs: Cache: working with cross-testrun state.

Having a big dictionary of globals that every test uses is probably a bad idea. If possible, I suggest refactoring your tests to avoid this sort of thing.
That said, here is how I would do it: define an autouse fixture that adds a reference to the dictionary in the global namespace of every function.
Here is some code. It's all in the same file, but you can move the fixture out to conftest.py at the top level of your tests.
import pytest
my_big_global = {'key': 'value'}
#pytest.fixture(autouse=True)
def myglobal(request):
request.function.func_globals['foo'] = my_big_global
def test_foo():
assert foo['key'] == 'value'
def test_bar():
assert foo['key'] == 'bar'
Here is the output from when I run this code:
$ py.test test_global.py -vv
======================================= test session starts =======================================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 2 items
test_global.py:9: test_foo PASSED
test_global.py:12: test_bar FAILED
============================================ FAILURES =============================================
____________________________________________ test_bar _____________________________________________
def test_bar():
> assert foo['key'] == 'bar'
E assert 'value' == 'bar'
E - value
E + bar
test_global.py:13: AssertionError
=============================== 1 failed, 1 passed in 0.01 seconds ===============================
Note that you can't use a session-scoped fixture because then you don't have access to each function object. Because of this, I'm making sure to define my big global dictionary once and use references to it -- if I defined the dictionary in that assignment statement, a new copy would be made each time.
In closing, doing anything like this is probably a bad idea. Good luck though :)

You can add your global variable as an option inside the pytest_addoption hook.
It is possible to do it explicitly with addoption or use set_defaults method if you want your attribute be determined without any inspection of the command line, docs
When option was defined, you can paste it inside any fixture with request.config.getoption and then pass it to the test explicitly or with autouse.
Alternatively, you can pass your option into almost any hook inside the config object.
#conftest.py
def pytest_addoption(parser):
parser.addoption("--my_global_var", default="foo")
parser.set_defaults(my_hidden_var="bar")
#pytest.fixture()
def my_hidden_var(request):
return request.config.getoption("my_hidden_var")
#test.py
def test_my_hidden_var(my_hidden_var):
assert my_hidden_var == "bar"