Develop Reference

mocked service call is not being used

I have the model object that looks like this
class CatalogModel(BaseModel):
#property
def custom_service(self):
return CustomService()
async def get_offers(self, catalog_name):
try:
svc_response = self.custom_service.get_offers(catalog_name=catalog_name)()
except BaseException:
raise SalesForceException()
return CustomOfferResponse().dump(svc_response)
I am trying to write a test for that get_offers function (that uses custom_services which is connecting to Salesforce)
My test is looking like this. I am using pytest, pytest vcr etc.
class TestCatalogModel:
catalog_name = 'CATALOG_1'
#freeze_time("2021-07-12")
async def test_get_offers(self, loop, offers, offers_response):
with MockUser(ident="test_model_get_offers"):
with patch(
"com.services.client.CustomService.get_offers", new=offers
):
eo_offers = await CatalogModel().get_offers(self.catalog_name)
assert offers_response == eo_offers
However when executing the test it fails with the error
E vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/test_api/recordings/2021-07-12/test_get_offers_model/salesforce/auth/client/services_oauth2_token.yaml') in your current record mode ('none').
E No match for the request (<Request (POST) https://server.salesforce.com/services/oauth2/token>) was found.
E No similar requests, that have not been played, found.
During handling of the above exception, another exception occurred:
...model.py:17: in test_get_offers
eo_offers = await CatalogModel().get_offers(self.catalog_name)
model.py:24: in get_offers
raise SalesForceException()
E ...SalesForceException: Error from Salesforce.
As far as I understand it is trying to connect to real Salesforce service, rather than using a mock. What is the problem?

I would avoid trying to partially patch methods on classes. Instead I would use inversion of control to allow mock instances to be used during tests. This avoids having to do any patching at all.
class CatalogModel(BaseModel):
def __init__(self, custom_service=None):
if custom_service is None:
custom_service = CustomService()
self.custom_service = custom_service
async def get_offers(self, catalog_name):
try:
svc_response = self.custom_service.get_offers(catalog_name=catalog_name)()
except BaseException:
raise SalesForceException()
return CustomOfferResponse().dump(svc_response)
Now writing the test becomes much simpler.
class TestCatalogModel:
catalog_name = 'CATALOG_1'
#freeze_time("2021-07-12")
async def test_get_offers(self, loop, offers, offers_response):
with MockUser(ident="test_model_get_offers"):
custom_service = mock.Mock()
custom_service.get_offers.return_value = offers
model = CatalogModel(custom_service)
eo_offers = await model.get_offers(self.catalog_name)
assert offers_response == eo_offers

TypeError: get() takes exactly 3 arguments (2 given) error in Python

I have checked few of the answers in SO, but nothing worked for me. I'm still giving the same error:
TypeError: get() takes exactly 3 arguments (2 given)
Can anybody check the following code and let me know what I have done wrong?
my 'views.py' is as follows
def get(self, request, tag):
print("Tag for tagging :")
data_loader = SvnDataLoader()
print("Two :")
ss = SubsystemRevision.get_subsystem_for_tag(tag)
print("Subsystem is %s", ss)
try:
print("inside try")
pr = subprocess.Popen(['perl', './svntasktag.pl', 'ss'], stdout=subprocess.PIPE)
data = pr.communicate()
context = {'data':data}
except TagHistoryMissing:
data = 'Tag is missing.'
except SvnException as e:
data = "Problem while trying to fetch tag-history from svn. Try again later"
#logger.error("SvnException %s while trying to fetch the tag %s" % (str(e), tag.name))
return render_to_response('pai_app/create_tag.html', {'data': data}, context_instance=RequestContext(request))

First of all you should know that self refers to the object itself. You can add another argument and you error will be solved e.g.
def get(self, request, tag, foobar):
...
But it's better to know what arguments you want to have and what you are passing to the function.
The second point is that you are using get() in your views. Usually it is not the good practice to use method names for the functions (we have Model.objects.get()). So it is better to change your function to something like get_tag() to prevent any type of conflicts during development. Don't forget to update your app modules! Good luck!

If your function not in a class,you can use
def get(request, tag):
pass
Because the params self not be used in your function.
In django function views,the first params must be a request obj instance.

Testing and mocking a threaded function within a view

I'm trying to unit test a function that I run threaded within a view. Whenever I try to mock it, it always goes to the original function, no the mocked function.
The code I'm testing, from the view module:
def restart_process(request):
batch_name = request.POST.get("batch_name", "")
if batch_name:
try:
batch = models.Batch.objects.get(num=batch_name)
except models.Batch.DoesNotExist:
logger.warning("Trying to restart a batch that does not exist: " + batch_name)
return HttpResponse(404)
else:
logger.info(batch_name + " restarted")
try:
t = threading.Thread(target=restart_from_last_completed_state, args=(batch,))
t.daemon = True
t.start()
except RuntimeError:
return HttpResponse(500, "Threading error")
return HttpResponse(200)
else:
return HttpResponse(400)
The test function:
class ThreadTestCases(TransactionTestCase):
def test_restart_process(self):
client = Client()
mock_restart_from_last_completed_state = mock.Mock()
with mock.patch("processapp.views.restart_from_last_completed_state", mock_restart_from_last_completed_state):
response = client.post('/batch/restart/', {"batch_name": "BATCH555"})
self.assertEqual(response.status_code, 200)
mock_restart_from_last_completed_state.assert_called_once()
The URL:
url(r'^batch/restart/$', views.restart_from_last_completed_state, name="restart_batch"),
I always get this error:
ValueError: The view processapp.processing.process_runner.restart_from_last_completed_state didn't return an HttpResponse object. It returned None instead.
I put a print command in the original function (restart_from_last_completed_state) and it always runs so the mocking does not take place.
The error seems to take the function as a view although it is not.
I'm not sure where the error is, the threading, testing, something else?

The URL variable was wrong. Was supposed to be views.restart_process not views.restart_from_last_completed_state
A copy/paste error as so many times...

Redis get and set decorator

Currently working on python and redis. I have Flask as my framework and working on Blueprints.
Looking into implementing cache with redis for my APIs, I have tried Flask-Cache and redis-simple-cache.
Downside are Flask-Cache saves the function even when you change the parameter of the function. It only saves it once per function.
As per redis-simple-cache, it save its keys as SimpleCache-<key name> which not advisable on my end.
So my question is, how can you create a decorator which save and retrieve or check if there is a key existing for the specific key.
I know a save decorator is possible. But is a retrieve or check decorator possible?? Please correct me if I am wrong. Thank you.

You seem to not have read the Flask-Cache documentation very closely. Caching does not ignore parameters and the cache key is customisable. The decorators the project supplies already give you the functionality you seek.
From the Caching View Functions section:
This decorator will use request.path by default for the cache_key.
So the default cache key is request.path, but you can specify a different key. Since Flask view functions get their arguments from path elements, the default request.path makes a great key.
From the #cached() decorator documentation:
cached(timeout=None, key_prefix='view/%s', unless=None)
By default the cache key is view/request.path. You are able to use this decorator with any function by changing the key_prefix. If the token %s is located within the key_prefix then it will replace that with request.path. You are able to use this decorator with any function by changing the key_prefix.
and
key_prefix – [...] Can optionally be a callable which takes no arguments but returns a string that will be used as the cache_key.
So you can set key_prefix to a function, and it'll be called (without arguments) to produce the key.
Moreover:
The returned decorated function now has three function attributes assigned to it. These attributes are readable/writable:
[...]
make_cache_key
A function used in generating the cache_key used.
This function is passed the same arguments the view function is passed. In all, this allows you to produce any cache key you want; either use key_prefix and pull out more information from the request or g or other sources, or assign to view_function.make_cache_key and access the same arguments the view function receives.
Then there is the #memoize() decorator:
memoize(timeout=None, make_name=None, unless=None)
Use this to cache the result of a function, taking its arguments into account in the cache key.
So this decorator caches return values purely based on the arguments passed into the function. It too supports a make_cache_key function.
I've used both decorators to make a Google App Engine Flask project scale to double-digit millions of views per month for a CMS-backed site, storing results in the Google memcached structure. Doing this with Redis would only require setting a configuration option.

You can use this cache decorator, the cache object you create will have to be a flask cache object instead of django one i.e. should support cache.get and cache.set methods, this is pretty flexible based on how you want to create cache keys, i.e.
based on what parameters being passed to the method
in what cases to cache/not cache the result
Use same cache even if kwarg order is changed, i.e. same cache for my_method(a=1,b=2) and my_method(b=2,a=1) call.
"
__author__ = 'Dhruv Pathak'
import cPickle
import logging
import time
from functools import wraps
from django.conf import settings
import traceback
"""following imports are from datautils.py in this repo, datautils
also contains many other useful data utility methods/classes
"""
from datautils import mDict, mList, get_nested_ordered_dict, nested_path_get
"""following import is specific to django framework, and can be altered
based on what type of caching library your code uses"""
from django.core.cache import cache, caches
logger = logging.getLogger(__name__)
def cache_result(cache_key=None, cache_kwarg_keys=None, seconds=900, cache_filter=lambda x: True, cache_setup = "default"):
def set_cache(f):
#wraps(f)
def x(*args, **kwargs):
if settings.USE_CACHE is not True:
result = f(*args, **kwargs)
return result
try:
# cache_conn should be a cache object supporting get,set methods
# can be from python-memcached, pylibmc, django, django-redis-cache, django-redis etc
cache_conn = caches[cache_setup]
except Exception, e:
result = f(*args, **kwargs)
return result
final_cache_key = generate_cache_key_for_method(f, kwargs, args, cache_kwarg_keys, cache_key)
try:
result = cache_conn.get(final_cache_key)
except Exception, e:
result = None
if settings.DEBUG is True:
raise
else:
logger.exception("Cache get failed,k::{0},ex::{1},ex::{2}".format(final_cache_key, str(e), traceback.format_exc()))
if result is not None and cache_filter(result) is False:
result = None
logger.debug("Cache had invalid result:{0},not returned".format(result))
if result is None: # result not in cache
result = f(*args, **kwargs)
if isinstance(result, (mDict, mList)):
result.ot = int(time.time())
if cache_filter(result) is True:
try:
cache_conn.set(final_cache_key, result, seconds)
except Exception, e:
if settings.DEBUG is True:
raise
else:
logger.exception("Cache set failed,k::{0},ex::{1},ex::{2},dt::{3}".format(final_cache_key, str(e), traceback.format_exc(), str(result)[0:100],))
#else:
# logger.debug("Result :{0} failed,not cached".format(result))
else: # result was from cache
if isinstance(result, (mDict, mList)):
result.src = "CACHE_{0}".format(cache_setup)
return result
return x
return set_cache
def generate_cache_key_for_method(method, method_kwargs, method_args, cache_kwarg_keys=None, cache_key=None):
if cache_key is None:
if cache_kwarg_keys is not None and len(cache_kwarg_keys) > 0:
if len(method_args) > 0:
raise Exception("cache_kwarg_keys mode needs set kwargs,args should be empty")
method_kwargs = get_nested_ordered_dict(method_kwargs)
cache_kwarg_values = [nested_path_get(method_kwargs, path_str=kwarg_key, strict=False) for kwarg_key in cache_kwarg_keys]
if any([kwarg_value is not None for kwarg_value in cache_kwarg_values]) is False:
raise Exception("all cache kwarg keys values are not set")
final_cache_key = "{0}::{1}::{2}".format(str(method.__module__), str(method.__name__), hash(cPickle.dumps(cache_kwarg_values)))
else:
final_cache_key = "{0}::{1}".format(str(method.__module__), str(method.__name__))
final_cache_key += "::{0}".format(str(hash(cPickle.dumps(method_args, 0)))) if len(method_args) > 0 else ''
final_cache_key += "::{0}".format(str(hash(cPickle.dumps(method_kwargs, 0)))) if len(method_kwargs) > 0 else ''
else:
final_cache_key = "{0}::{1}::{2}".format(method.__module__, method.__name__, cache_key)
return final_cache_key
2-3 utility methods are imported from this file in same repo, you can just put them in same file.

AttributeError when using request.function in pytest yield fixture

I have several pytest test cases that need nearly identical setup, so I would like to have them reuse a fixture to keep things DRY. The setup involves creating a new ticket in an external ticket tracking system, then the test cases interact with the ticket based on the data, and finally the fixture cleans up by closing out the ticket. The challenge here is that each test case needs slightly different data to be prepped in the ticket.
Each test case has different calls and different assertions, so I can't combine them all into a single parametrized test case with a single test fixture. Parametrizing the the fixture itself would result in every test case running every permutation of the fixture data, which ends up with a lot of irrelevant test failures.
What I would like to do is set a variable in the test case, then have the fixture use that variable to set up the test data when creating the ticket. I've tried to use request.function as specified in the pytest fixture docs but I keep getting:
=================================== ERRORS ===================================
____________________ ERROR at setup of TestMCVE.test_stuff ___________________
request = <SubRequest 'ticket' for <Function 'test_stuff'>>
#pytest.yield_fixture
def ticket(request):
> ticket_summary = getattr(request.function, "summary")
E AttributeError: 'function' object has no attribute 'summary'
tests\test_mcve.py:11: AttributeError
My code is:
import pytest
def ticket_system_api(summary):
# stub for MCVE purposes
return summary
#pytest.yield_fixture
def ticket(request):
ticket_summary = getattr(request.function, "summary")
new_ticket = ticket_system_api(summary=ticket_summary)
yield new_ticket
class TestMCVE:
def test_stuff(self, ticket):
summary = 'xyz'
# do real things here, except MCVE
assert 'xyz' == ticket
I've tried using request.node instead of request.function as well as binding the summary variable per this answer, changing summary = 'xyz' to test_stuff.summary = 'xyz' but these still fail with the same AttributeError.
How can I pass the function level data to the fixture?

You can accomplish this with indirect parametrization. The API (and the documentation) could be friendlier, but the functionality you want is there.
Your example was very close, and minor tweaks were needed. Take a look:
import pytest
def ticket_system_api(summary):
# stub for MCVE purposes
return summary
#pytest.fixture
def ticket(request):
# NOTE: This will raise `AttributeError` if the fixture
# doesn't receive a parameter.
ticket_summary = request.param
new_ticket = ticket_system_api(summary=ticket_summary)
return new_ticket
class TestMCVE:
#pytest.mark.parametrize('ticket', ('abc',), indirect=True)
def test_abc(self, ticket):
# do real things here, except MCVE
assert ticket == 'abc'
#pytest.mark.parametrize('ticket', ('xyz',), indirect=True)
def test_xyz(self, ticket):
# do real things here, except MCVE
assert ticket == 'xyz'