Develop Reference

Patch or Mock Google Cloud Logging in Python unittests

I have a program which is eventually deployed on Google Cloud Platform and I use the Logs Explorer with GCP to monitor my logs. For that, I need to use the google.cloud.logging library, which is necessary to show logs with correct severity level on GCP Logs Explorer.
In my unit-tests, I am trying to patch the the call to the google.cloud.logging library, however, they all end up with 403 error on my local, suggesting that the library has not been patched.
cloud_logger.py
import google.cloud.logging
import logging
def get_logger():
client = google.cloud.logging.Client()
client.get_default_handler()
client.setup_logging()
logger = logging.getLogger(__name__)
return logger
rss_crawler.py which utlizes the cloud_logger.get_logger
from cloud_logger import get_logger
logger = get_logger()
def crawl_rss_source(source_crawling_details):
brand_name = source_crawling_details[constants.BRAND_NAME]
source_name = source_crawling_details[constants.SOURCE_NAME]
initial_agent_settings = source_crawling_details[constants.INITIAL_AGENT_SETTINGS]
logger.info(f"Started crawling {brand_name}-{source_name} crawling")
source = source_crawling_details["source"]
entry_points_list = source[constants.ENTRY]
source_crawling_details.update({constants.ENTRY: entry_points_list})
source_crawling_details.update({constants.AGENT: initial_agent_settings})
content = get_content(source_crawling_details)
logger.info("Getting links present in rss feed entry")
entry_points = rss_entry_points(content)
source_crawling_details.update({constants.ENTRY_POINTS: entry_points})
candidate_urls = start_content_tasks(source_crawling_details)
if not candidate_urls:
raise CustomException("There are no links to scrape")
# filtered urls found with crawl rules, next step get scrape candidates based on scrape rules
scrape_rules = source[constants.SCRAPE]
scrape_candidates = get_scrape_candidates(scrape_rules, candidate_urls)
if not scrape_candidates:
raise CustomException(
f"Could not find any links for scraping, please check scrape,crawl rules, or possibly depth level for brand {brand_name} , source {source_name}"
)
return scrape_candidates
test_rss_crawler.py
#patch("start_crawl.fetch_source_crawling_fields")
#patch("rss_crawler.logger")
def test_crawl_rss_source_raises_exception(
self, mocked_logger, mocked_source_fetcher
):
mocked_logger.logger.return_value = logging.getLogger(__name__)
self.test_source[constants.SCRAPE] = {
"white_list": ["https://buffer.com/blog/(\\w|\\d|\\-)+/$"]
}
details = set_content_source_details(
self.brand_name,
self.source_name,
self.agent_args,
self.source,
**self.key_word_argument,
)
# test to see if exception is raised if scrape rule is not matching
self.assertRaises(CustomException, crawl_rss_source, details)
self.test_source[constants.CRAWL] = {"white_list": ["https://buffer.com/blog/"]}
details = set_content_source_details(
self.brand_name,
self.source_name,
self.agent_args,
self.source,
**self.key_word_argument,
)
# test to see if exception is raised if crawl rule is not matching
self.assertRaises(CustomException, crawl_rss_source, details)
However, I get these warning messages, when I run these tests, even after patching:
Traceback (most recent call last):
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/cloud/logging_v2/handlers/transports/background_thread.py", line 115, in _safely_commit_batch
batch.commit()
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 385, in commit
client.logging_api.write_entries(entries, **kwargs)
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 149, in write_entries
self._gapic_api.write_log_entries(request=request)
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py", line 592, in write_log_entries
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
return retry_target(
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/api_core/retry.py", line 189, in retry_target
return target()
File "/Users/reydon227/Concured/crawler/env/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 The caller does not have permission

not sure about how author handled that, but I had similar symptoms with one of my tests with mocked google things, and I can't see any imports section in test_rss_crawler.py code. In my code I had to import class under test inside test method, because #mock.patch(..) runs before method, but after test file import section, and code in imported module may already initialize class supposed to be patched, so patch has no effect. Mb smbd will find useful that importing modules supposed to be patched inside test method may help.
#mock.patch('some_class')
def test_smth(self, some_class):
import some_class
some_class.return_value = ....

How should I handle exceptions raised in #jwt_required decorator? (in flask-jwt-extended)

I have a function with #jwt_required decorator.
class Test(Resource):
#jwt_required
def get(self):
return {"test": "ok" }
Which works fine when the correct HTTP header is set, i.e.
Authentication: Bearer [TOKEN]
but when the token is invalid/wrong or messed with, a jwt.exceptions.DecodeError is raised:
File "env/lib/python3.6/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "env/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 103, in wrapper
verify_jwt_in_request()
File "env/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 32, in verify_jwt_in_request
jwt_data = _decode_jwt_from_request(request_type='access')
File "env/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 267, in _decode_jwt_from_request
decoded_token = decode_token(encoded_token, csrf_token)
File "env/lib/python3.6/site-packages/flask_jwt_extended/utils.py", line 80, in decode_token
encoded_token, verify=False, algorithms=config.algorithm
File "env/lib/python3.6/site-packages/jwt/api_jwt.py", line 84, in decode
payload, _, _, _ = self._load(jwt)
File "env/lib/python3.6/site-packages/jwt/api_jws.py", line 183, in _load
raise DecodeError('Not enough segments')
jwt.exceptions.DecodeError: Not enough segments
I cannot rely on clients always using correct tokens all the time.
And I cannot catch the exception because it is raised in the decorator rather than my own function. So the result is a http 500 error. How should I handle the exception more gracefully?

Flask-jwt-extended should be handling those for you gracefully. If not, you are probably using another extension (like flask-restful for example) that is breaking native flask functionality. You can try setting this option to fix it app.config[‘PROPAGATE_EXCEPTIONS’] = True, or take a look at this thread for some advice if you are using a different flask extension that is causing problems https://github.com/vimalloc/flask-jwt-extended/issues/86

Python - RuntimeError: working outside of request context

Trying to get the GET parameters from the URL. I have it working in my __init__.py file, but in a different file its not working.
I tried to use with app.app_context(): but I am still getting the same issue.
def log_entry(entity, type, entity_id, data, error):
with app.app_context():
zip_id = request.args.get('id')
RuntimeError: working outside of request context
Any suggestions?
Additional Info:
This is using Flask web framework which is setup as a service (API).
Example URL the user would hit http://website.com/api/endpoint?id=1
As mentioned above using `zip_id = request.args.get('id') works fine in the main file but I am in runners.py (just another file with definitions in)
Full traceback:
Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/wsgi.py", line 703, in __next__
return self._next()
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/wrappers.py", line 81, in _iter_encoded
for item in iterable:
File "/Users/ereeve/Documents/TechSol/pi-automation-api/automation_api/runners.py", line 341, in create_agencies
log_entry("test", "created", 1, "{'data':'hey'}", "")
File "/Users/ereeve/Documents/TechSol/pi-automation-api/automation_api/runners.py", line 315, in log_entry
zip_id = request.args.get('id')
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/flask/globals.py", line 20, in _lookup_req_object
raise RuntimeError('working outside of request context')
RuntimeError: working outside of request context
Def in the same file calling the log_entry def
def create_agencies(country_code, DB, session):
document = DB.find_one({'rb_account_id': RB_COUNTRIES_new[country_code]['rb_account_id']})
t2 = new_t2(session)
log_entry("test", "created", 1, "{'data':'hey'}", "")

Weird error with Redis and Celery

I'm getting the following error in one of my Celery workers:
2015-07-21T15:02:04.010066+00:00 app[worker.1]: Traceback (most recent call last):
2015-07-21T15:02:04.010069+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 296, in trace_task
2015-07-21T15:02:04.010070+00:00 app[worker.1]: on_chord_part_return(task, state, R)
2015-07-21T15:02:04.010073+00:00 app[worker.1]: deps.delete()
2015-07-21T15:02:04.010074+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/result.py", line 773, in delete
2015-07-21T15:02:04.010071+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 587, in on_chord_part_return
2015-07-21T15:02:04.010078+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 329, in delete_group
2015-07-21T15:02:04.010076+00:00 app[worker.1]: (backend or self.app.backend).delete_group(self.id)
2015-07-21T15:02:04.010079+00:00 app[worker.1]: return self._delete_group(group_id)
2015-07-21T15:02:04.010081+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 499, in _delete_group
2015-07-21T15:02:04.010082+00:00 app[worker.1]: self.delete(self.get_key_for_group(group_id))
2015-07-21T15:02:04.010083+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 172, in delete
2015-07-21T15:02:04.010084+00:00 app[worker.1]: self.client.delete(key)
2015-07-21T15:02:04.010085+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 824, in delete
2015-07-21T15:02:04.010087+00:00 app[worker.1]: return self.execute_command('DEL', *names)
2015-07-21T15:02:04.010088+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
2015-07-21T15:02:04.010089+00:00 app[worker.1]: return self.parse_response(connection, command_name, **options)
2015-07-21T15:02:04.010090+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 579, in parse_response
2015-07-21T15:02:04.010091+00:00 app[worker.1]: return self.response_callbacks[command_name](response, **options)
2015-07-21T15:02:04.010093+00:00 app[worker.1]: ValueError: invalid literal for int() with base 10: 'QUEUED'
What I find weird is that I see no call to int in the last line of the stack trace. QUEUED probably came in as a worker's status. I'm using it as a custom worker status like this:
#before_task_publish.connect
def update_sent_state(sender=None, body=None, **kwargs):
# the task may not exist if sent using `send_task` which
# sends tasks by name, so fall back to the default result backend
# if that is the case.
task = current_app.tasks.get(sender)
backend = task.backend if task else current_app.backend
logging.debug("Setting status for %s" % body["id"])
backend.store_result(body['id'], None, "QUEUED")
What could be the issue here?
In case it's relevant, here's the code for my task. I only call fetch directly is fetch.
#app.task
def fetch(url_or_urls, subscribe=None):
"""This fetches a (list of) podcast(s) and stores it in the db. It assumes that it only gets called
by Podcast.get_by_url, or some other method that knows whether a given podcast has
already been fetched.
If *subscribe* is given, it should be a User instance to be subscribed to the given podcasts."""
if isinstance(url_or_urls, basestring):
url_or_urls = [url_or_urls]
body = _store_podcasts.s()
if subscribe:
body.link(_subscribe_user.s(user=subscribe))
return chord([_fetch_podcast_data.s(url) for url in url_or_urls])(body)
#app.task
def _fetch_podcast_data(url):
return do_fetch(url) # This function returns a dict of podcast data.
#app.task
def _store_podcasts(podcasts_data):
"""Given a list of dictionaries representing podcasts, store them all in the database."""
podcasts = [Podcast(**pdata) for pdata in podcasts_data]
return Podcast.objects.insert(podcasts)
#app.task
def _subscribe_user(podcasts, user):
"""Subscribe the given users to all the podcasts in the list."""
return user.subscribe_multi(podcasts)
Is there anything else that could be relevant here?
Library versions as shown by pip freeze:
redis==2.10.3
celery==3.1.18

It is hard to debug such a bug without working code. Here is what i think it could be.
Lets start here:
http://celery.readthedocs.org/en/latest/_modules/celery/backends/base.html#BaseBackend.store_result
def store_result(self, task_id, result, status,
traceback=None, request=None, **kwargs):
"""Update task state and result."""
result = self.encode_result(result, status)
self._store_result(task_id, result, status, traceback,
request=request, **kwargs)
return result
It calls ecnode_result. Lets check that out
def encode_result(self, result, status):
if status in self.EXCEPTION_STATES and isinstance(result, Exception):
return self.prepare_exception(result)
else:
return self.prepare_value(result)
It looks like "status" is expected to be something from predefined STATE constants.
Its code is here
http://celery.readthedocs.org/en/latest/_modules/celery/states.html#state
And docs here
http://celery.readthedocs.org/en/latest/reference/celery.states.html
That does not look like they expect to see something like "QUEUED" there. Try one of the predefined.

The redis python packages expects the response from the DEL action to always be an integer, which I assume is the count of deleted rows.
The call to int happens in the last line (return self.response_callbacks[command_name](response, **options)) where self.response_callbacks['DEL'] is equal to int.
As a workaround, you could subclass the redis.client.StrictRedis and set the DEL response callback to something other than int, just make sure you're familiar with the implications.

I got the same error these days. And founded my QUEUED response comes from redis MULTI commands. See https://redis.io/topics/transactions#usage.
It maybe that your are reading response from wrong connection. Maybe in multi-processing / multi-threading / eventlet, etc. Not sure.

Unit test fails saying 'WSGIRequest' object has no attribute 'city'

I have written a unit test to test an api....Its a GET call....
When I run it , i get this error ...Heres the traceback....
Traceback (most recent call last):
File "/home/arindam31/XXX-Name/mapi/tests.py", line 141, in test_get_cities
response = self.cl.get('/mapi/latest/cities/')
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 445, in get
response = super(Client, self).get(path, data=data, **extra)
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 229, in get
return self.request(**r)
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 387, in request
response = self.handler(environ)
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 84, in __call__
response = self.get_response(request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 169, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 218, in handle_uncaught_exception
return callback(request, **param_dict)
File "/home/arindam31/XXX-Name/saul/views.py", line 546, in do_handle_500
return render_to_response("500.html", context_instance=RequestContext(request))
File "/usr/local/lib/python2.7/dist-packages/django/template/context.py", line 177, in __init__
self.update(processor(request))
File "/home/arindam31/XXX-Name/saul/context_processors.py", line 46, in common_context
ret_dict['city'] = request.city
AttributeError: 'WSGIRequest' object has no attribute 'city'
Heres my unit test...
def test_get_cities(self):
request = HttpRequest()
request.city = self.c1
response = self.cl.get('/mapi/latest/cities/')
content = response.content
data = json.loads(content)
for city in City.objects.all():
assert city.name in data
assert data[city.name] == city.pk
Here , self.c1 is a city type object in the setUp part....HttpRequest is from django.http.
The view being tested is below:
def get_cities(request):
print "1"
if ENABLE_HTTPS_CHECK and not request.is_secure():
return HttpResponseForbidden()
if request.method != 'GET':
return HttpResponseNotAllowed('Not allowed')
city_map = _get_city_map()
response = HttpResponse(json.dumps(city_map)
content_type='application/json')
response['Cache-Control'] = 'no-cache'
return response

If you want to test your view with your own request object, you should call the view directly. You should use the RequestFactory to create your request.
As Daniel Roseman points out your request object having a city attribute remains broken, unless you have some middleware that would set it. You clearly have some middleware that require it to be set in saul/context_processors.py.
Using RequestFactory and calling the view directly circumvents the middleware entirely, so you can concentrate on testing your view (if the view supports the absent middleware).
If your view requires the middelware to be operational, you probably just need to be logged in using the test client. It has a session pr test method.

I have no idea what you are trying to do here. You instantiate a request object, assign a city attribute, then proceed to ignore that object and just use the standard test client. I don't know why you would think that that request would be used for the client get.
To be honest though I think that your entire design is broken. You don't show how you expect to get the parameter into the request in a normal non-test scenario, but usually you would pass it via the POST or GET parameters rather than annotating it into the request somehow. That would of course make it easier to test, since you could just pass the dictionary to the client call.