I am trying to write a test for a small module which makes use of boto3, there are a few global imports that I want to mock, but it seems like the act of mocking the module means the global module is loaded and fails.
I do wonder if this highlights the fact the code should be refactored but I am not totally sure on this, the code is deployed as an AWS lambda function and the global variables are created outside the handler function.
Here are the global variables, defined outside the handler:
QUEUE_NAME = os.getenv("QUEUE_NAME", None)
TABLE_NAME = os.getenv("TABLE_NAME", None)
sqs_client = boto3.resource("sqs")
dynamo_client = boto3.client("dynamodb")
queue = sqs_client.get_queue_by_name(QueueName=QUEUE_NAME)
When I try to mock the sqs_client object I get an error, similarly I get that error if I import the module into my test module. Here is the stub of the failing test:
#mock.patch.dict(os.environ, {"QUEUE_NAME": "QUEUE1", "TABLE_NAME": "TABLE1"})
#mock.patch("queue_processor.sqs_client.get_queue_by_name", return_value=None)
#mock.patch("queue_processor.dynamo_client.get_item", return_value=None)
def test_queue_entry(mocker):
The error appears to be because the mock is not being used so a call is made to AWS:
E botocore.errorfactory.QueueDoesNotExist: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the GetQueueUrl operation: The specified queue does not exist for this wsdl version.
My question is twofold, how do I avoid this error, in which the SQS client tries to call AWS rather than using the mock, and secondly, is this actually something I want to fix or would the code be better refactored to avoid global variables?
Related
I have a lamba handler that uses an environment variable. How can I set that value using pytest. I'm getting the error
tests/test_kinesis.py:3: in <module>
from runner import kinesis
runner/kinesis.py:6: in <module>
DATA_ENGINEERING_BUCKET = os.environ["BUCKET"]
../../../../../.pyenv/versions/3.8.8/lib/python3.8/os.py:675: in __getitem__
raise KeyError(key) from None
E KeyError: 'BUCKET'
7:03
I tried setting in the test like this
class TestHandler(unittest.TestCase):
#mock_s3
#mock_lambda
def test_handler(monkeypatch):
monkeypatch.setenv("BUCKET", "test-bucket")
actual = kinesis.handler(kinesis_stream_event, "")
expected = {"statusCode": 200, "body": "OK"}
assert actual == expected
DATA_ENGINEERING_BUCKET = os.environ["BUCKET"]
def handler(event, context):
...
You're getting the failure before your monkeypatch is able to run. The loading of the environment variable will happen when the runner module is first imported.
If this is a module you own, I'd recommend modifying the code to use a default value if DATA_ENGINEERING_BUCKET isn't set. Then you can modify it's value to whatever you want at runtime by calling module.DATA_ENGINEERING_BUCKET = "my_bucket".
DATA_ENGINEERING_BUCKET = os.environ.get("BUCKET", default="default_bucket")
If you can't modify that file then things are more complicated.
I looked into creating a global fixture that monkeypatches the environment and loads the module once, before any tests load and received a pytest error about using function level fixtures within a session level fixture. Which makes sense monkeypatch really isn't intended to fake things long term. You can stick the module load into your test after the monkeypatch but that will generate a lot of boilerplate.
What eventually worked creating a fixture that will provide the class in lieu of importing it. The fixture; sets os.environ to the desired value, loads the module, resets os.environ to it's origional value then yields the module. Any tests that need this module can request the fixture to have access to it within their scope. A word of caution, because test files are imported before fixtures are run any test files that don't use the fixture and import the module normally will raise a KeyError and cause pytest to crash before running any tests.
conftest.py
import os, pytest
#pytest.fixture(scope='session')
def kinesis():
old_environ = os.environ
os.environ = {'BUCKET': 'test-bucket'}
import kinesis
os.environ = old_environ
yield kinesis
tests.py
# Do NOT import kinesis in any test file. Rely on the fixture.
class TestHandler(unittest.TestCase):
#mock_s3
#mock_lambda
def test_handler(kinesis):
actual = kinesis.handler(kinesis_stream_event, "")
expected = {"statusCode": 200, "body": "OK"}
assert actual == expected
A potentially simpler method
os.environ is a dictionary of environment variables that is created when os first loads. If you want a single value for every test then you just need to add the value you want to it before loading any test modules. If you put os.environ['BUCKET'] = 'test-bucket' at the top of conftest.py you will set the environment variable for the rest of the test session. Then as long as the first import of the module happens afterwards you won't have a key error. The big downside to this approach is that unless you know to look in conftest.py or grep the code it will be difficult to determine where the environment variable is getting set when troubleshooting.
Using Django 1.10 and python 3.5.1.
I'm trying to mock 'call_command' function to throw an exception. The problem is that seems like the moment it gets the 'side_effect' function - it keeps to it also for other tests. What am I doing wrong or how can I 'revert' the side_effect from that function?
In this example, after running one of the tests, all other tests that run afterwards will throw the same exception even if it's not supposed to throw exception in that test.
def test_run_migrations_raise_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=Exception('e message')):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
def test_run_migrations_raise_flow_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=FlowException(500, 'fe message', {'a': 1})):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
You should not patch a function that is in your module-local (i.e. Python's "global" - which is actually "module") namespace.
When in Python you do
from module.that import this
this becomes a variable on the module that contains the import statement. Any changes to "module.that.this" will affect the object pointed in the other module, but using only this will still reefer to the original object.
Perhaps your code is not exactly as you show us, or maybe "mock.pacth" can find out that the module-local call_command is pointing to django.core.management.call_command in the other module when it makes the patch - but not when reversing the patch. The fact is your module-local name call_command is being changed.
You can fix that by simply changing your code to not bind a module variable directly to the function you want to change:
from django.core import management
def test_run_migrations_raise_exception(self):
with mock.patch('django.core.management.call_command', return_value=None, side_effect=Exception('e message')):
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
management.call_command('run_data_migrations')
self.check_migrations_called(MigrationTracker.objects.all(), data_migrations_settings_in_db)
I hope you can understand that and solve this problem. Now, that said, this use of mock makes no sense at all: the idea of using mock is that some callable used indirectly by code you call within the code-block that applies the patch does not have the original effect - so the intermetiate code can run and be tested. You are calling directly the mock object - so it will have none of the original code - calling call_command('run_data_migrations') runs no code on your code base at all, and thus, there is nothing there to test. It just calls the mocked instance, and it will not change the status of anything that could be detected with check_migrations_called.
I am getting a strange "unbound local" error in a python package that seems to defy all logic. I can't get a MWE to reproduce it, but will try to explain succinctly and hope that someone might be able to offer some explanation of what is going on.
For the sake of this example module is a package I developed, and Model is a class within module. The definition of the Model class (model.py) looks like:
import module
class Model:
def __init__(self):
print module
def run(self):
print module
Now, when I instantiate a Model in a script like this:
from model import Model
m = Model()
m.run()
module prints successfully within the __init__, but I get an unbound local error within the run function.
I attempted to diagnose this with pdb, which is where things got really weird, because if I add a pdb trace immediately prior to the print module line in the run() function, then I can successfully run print module without an unbound local error, but if I step to the next line then it throws the error. How can module be in the scope of __init__(), and in the scope of pdb, but not in the scope of run()?
I know this is not ideal since there is no MWE, but I cannot seem to reproduce this outside the context of the full code. I am hoping that someone will have an idea of what might possibly be going on and suggest some strategies to debug further.
Apparently you have a local variable named module somewhere in the function run. For example, the following code will throw UnboundLocalError
import sys
def run():
print sys
sys = None
run()
Here sys = None introduces a local name that shadows the imported sys inside run and at the time print invoked it is not yet defined, hence the error. To use the imported module inside run you have to find and rename the local variable.
More info on python scoping rules is here
I have a function called get_account(param1,param2)
in run time I need to replace this function with the function mock_get_account(param1,param2)
so when the system calls get_account(param1,param2) I need the mock_get_account(param1,param2) to be called instead.
I tried this code:
package.get_account=self.mock_get_account
package.get_account(x,y)
but still the get_account runs instead of the mock_get_account
I'm new to python and I don't know if this is even possible but I have seen the lamda function and I know that function programming is possible in python. Thanks
Edit:
if i do the following:
package.get_account=self.mock_get_account
package.get_account(x,y)
then every thing is ok, meaning the mock_get_account is called, but in mu code I the following code i do a post self.client.post(url, data=data, follow=True) that triggers the package.get_account and this is not working:
package.get_account=self.mock_get_account
package.get_account(x,y)
#the folowing call will trigger the package.get_account(x,y) function in a django url #callback
self.client.post(url, data=data, follow=True)
meaning it calls the old function, also get_account(param1,param2) is defined in side a file, and is not a child function of a class and mock_get_account(self,param1,param2) is defined in a class Test and is called inside the Test.test_account - function
This is very opinionated and does not (directly) answer your question, but hopefully solves your problem.
A better practice is to use a subclass with your mock_get_account's implementation override the parent get_account method, example below:
class A(object):
def get_account(self):
return 1
def post(self):
return self.get_account()
class B(A):
def get_account(self):
return 2 # your original mock_get_account implementation
a = A()
print(a.get_account())
b = B()
print(b.post()) # this .post will trigger the overridden implementation of get_account
My guess is that the code implementing self.client.post has access to get_account through an import statement that looks like from package import get_account.
from package import get_account will first load package if it hasn't been already imported. Then it will look for a name get_account in that module, and whatever object that was bound to will be bound in the importing package's namespace, also under the name get_account. Thereafter the two names refer to the same object, but they are not the same name.
So if your mocking code comes along after this point, it sets the name get_account in package to instead refer to mock_get_account. But that'll only affect code that reads get_account from package again; anything that's already imported that name specially won't be affected.
If the code behind self.client.post instead had access only to package through import package, and was calling package.get_account it would work, because it's then only the object representing the package module that has been bound in the importing module's namespace. package.get_account would be reading an attribute of that object, and so would get whatever the current value is. If the from package import get_account appeared at function local scope rather than module scope, then this would behave similarly.
If I'm correct and your code is structured this way, then it's unfortunately not really package.get_account you need to rebind to a mock, but the get_account name in the module where self.client.post comes from (as well as any other modules which may call it).
I'm fairly new to using mock and testing in general. This is my first attempt to mock a whole imported module. So for example I have
try:
import redis
except:
redis = None
Then later on in the code I check for redis
if redis is None:
return
How can I set a mock object or class to the redis namespace so I don't have to install redis on my CI server?
Names are just names, and you can assign anything to the 'redis' name at file/global scope, using either import or plain old assignment.
Like so:
import mock_redis as redis
...or so:
def mock_redis(): pass
BTW, your exception clause should be narrowed to handle only ImportError.