Develop Reference

How to handle requests exception in Python [duplicate] - python

try:
r = requests.get(url, params={'s': thing})
except requests.ConnectionError, e:
print(e)
Is this correct? Is there a better way to structure this? Will this cover all my bases?

Have a look at the Requests exception docs. In short:
In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a ConnectionError exception.
In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.
If a request times out, a Timeout exception is raised.
If a request exceeds the configured number of maximum redirections, a TooManyRedirects exception is raised.
All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException.
To answer your question, what you show will not cover all of your bases. You'll only catch connection-related errors, not ones that time out.
What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can't go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit).
You can either catch the base-class exception, which will handle all cases:
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.RequestException as e: # This is the correct syntax
raise SystemExit(e)
Or you can catch them separately and do different things.
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
raise SystemExit(e)
As Christian pointed out:
If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call Response.raise_for_status. That will raise an HTTPError, if the response was an http error.
An example:
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
Will print:
404 Client Error: Not Found for url: http://www.google.com/nothere

One additional suggestion to be explicit. It seems best to go from specific to general down the stack of errors to get the desired error to be caught, so the specific ones don't get masked by the general one.
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
Http Error: 404 Client Error: Not Found for url: http://www.google.com/blahblah
vs
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
OOps: Something Else 404 Client Error: Not Found for url: http://www.google.com/blahblah

Exception object also contains original response e.response, that could be useful if need to see error body in response from the server. For example:
try:
r = requests.post('somerestapi.com/post-here', data={'birthday': '9/9/3999'})
r.raise_for_status()
except requests.exceptions.HTTPError as e:
print (e.response.text)

Here's a generic way to do things which at least means that you don't have to surround each and every requests call with try ... except:
Basic version
# see the docs: if you set no timeout the call never times out! A tuple means "max
# connect time" and "max read time"
DEFAULT_REQUESTS_TIMEOUT = (5, 15) # for example
def log_exception(e, verb, url, kwargs):
# the reason for making this a separate function will become apparent
raw_tb = traceback.extract_stack()
if 'data' in kwargs and len(kwargs['data']) > 500: # anticipate giant data string
kwargs['data'] = f'{kwargs["data"][:500]}...'
msg = f'BaseException raised: {e.__class__.__module__}.{e.__class__.__qualname__}: {e}\n' \
+ f'verb {verb}, url {url}, kwargs {kwargs}\n\n' \
+ 'Stack trace:\n' + ''.join(traceback.format_list(raw_tb[:-2]))
logger.error(msg)
def requests_call(verb, url, **kwargs):
response = None
exception = None
try:
if 'timeout' not in kwargs:
kwargs['timeout'] = DEFAULT_REQUESTS_TIMEOUT
response = requests.request(verb, url, **kwargs)
except BaseException as e:
log_exception(e, verb, url, kwargs)
exception = e
return (response, exception)
NB
Be aware of ConnectionError which is a builtin, nothing to do with the class requests.ConnectionError*. I assume the latter is more common in this context but have no real idea...
When examining a non-None returned exception, requests.RequestException, the superclass of all the requests exceptions (including requests.ConnectionError), is not "requests.exceptions.RequestException" according to the docs. Maybe it has changed since the accepted answer.**
Obviously this assumes a logger has been configured. Calling logger.exception in the except block might seem a good idea but that would only give the stack within this method! Instead, get the trace leading up to the call to this method. Then log (with details of the exception, and of the call which caused the problem)
*I looked at the source code: requests.ConnectionError subclasses the single class requests.RequestException, which subclasses the single class IOError (builtin)
**However at the bottom of this page you find "requests.exceptions.RequestException" at the time of writing (2022-02)... but it links to the above page: confusing.
Usage is very simple:
search_response, exception = utilities.requests_call('get',
f'http://localhost:9200/my_index/_search?q={search_string}')
First you check the response: if it's None something funny has happened and you will have an exception which has to be acted on in some way depending on context (and on the exception). In Gui applications (PyQt5) I usually implement a "visual log" to give some output to the user (and also log simultaneously to the log file), but messages added there should be non-technical. So something like this might typically follow:
if search_response == None:
# you might check here for (e.g.) a requests.Timeout, tailoring the message
# accordingly, as the kind of error anyone might be expected to understand
msg = f'No response searching on |{search_string}|. See log'
MainWindow.the().visual_log(msg, log_level=logging.ERROR)
return
response_json = search_response.json()
if search_response.status_code != 200: # NB 201 ("created") may be acceptable sometimes...
msg = f'Bad response searching on |{search_string}|. See log'
MainWindow.the().visual_log(msg, log_level=logging.ERROR)
# usually response_json will give full details about the problem
log_msg = f'search on |{search_string}| bad response\n{json.dumps(response_json, indent=4)}'
logger.error(log_msg)
return
# now examine the keys and values in response_json: these may of course
# indicate an error of some kind even though the response returned OK (status 200)...
Given that the stack trace is logged automatically you often need no more than that...
Advanced version when json object returned
(... potentially sparing a great deal of boilerplate!)
To cross the Ts, when a json object is expected to be returned:
If, as above, an exception gives your non-technical user a message "No response", and a non-200 status "Bad response", I suggest that
a missing expected key in the response's JSON structure should give rise to a message "Anomalous response"
an out-of-range or strange value to a message "Unexpected response"
and the presence of a key such as "error" or "errors", with value True or whatever, to a message "Error response"
These may or may not prevent the code from continuing.
... and in fact to my mind it is worth making the process even more generic. These next functions, for me, typically cut down 20 lines of code using the above requests_call to about 3, and make most of your handling and your log messages standardised. More than a handful of requests calls in your project and the code gets a lot nicer and less bloated:
def log_response_error(response_type, call_name, deliverable, verb, url, **kwargs):
# NB this function can also be used independently
if response_type == 'No': # exception was raised (and logged)
if isinstance(deliverable, requests.Timeout):
MainWindow.the().visual_log(f'Time out of {call_name} before response received!', logging.ERROR)
return
else:
if isinstance(deliverable, BaseException):
# NB if response.json() raises an exception we end up here
log_exception(deliverable, verb, url, kwargs)
else:
# if we get here no exception has been raised, so no stack trace has yet been logged.
# a response has been returned, but is either "Bad" or "Anomalous"
response_json = deliverable.json()
raw_tb = traceback.extract_stack()
if 'data' in kwargs and len(kwargs['data']) > 500: # anticipate giant data string
kwargs['data'] = f'{kwargs["data"][:500]}...'
added_message = ''
if hasattr(deliverable, 'added_message'):
added_message = deliverable.added_message + '\n'
del deliverable.added_message
call_and_response_details = f'{response_type} response\n{added_message}' \
+ f'verb {verb}, url {url}, kwargs {kwargs}\nresponse:\n{json.dumps(response_json, indent=4)}'
logger.error(f'{call_and_response_details}\nStack trace: {"".join(traceback.format_list(raw_tb[:-1]))}')
MainWindow.the().visual_log(f'{response_type} response {call_name}. See log.', logging.ERROR)
def check_keys(req_dict_structure, response_dict_structure, response):
# so this function is about checking the keys in the returned json object...
# NB both structures MUST be dicts
if not isinstance(req_dict_structure, dict):
response.added_message = f'req_dict_structure not dict: {type(req_dict_structure)}\n'
return False
if not isinstance(response_dict_structure, dict):
response.added_message = f'response_dict_structure not dict: {type(response_dict_structure)}\n'
return False
for dict_key in req_dict_structure.keys():
if dict_key not in response_dict_structure:
response.added_message = f'key |{dict_key}| missing\n'
return False
req_value = req_dict_structure[dict_key]
response_value = response_dict_structure[dict_key]
if isinstance(req_value, dict):
# if the response at this point is a list apply the req_value dict to each element:
# failure in just one such element leads to "Anomalous response"...
if isinstance(response_value, list):
for resp_list_element in response_value:
if not check_keys(req_value, resp_list_element, response):
return False
elif not check_keys(req_value, response_value, response): # any other response value must be a dict (tested in next level of recursion)
return False
elif isinstance(req_value, list):
if not isinstance(response_value, list): # if the req_value is a list the reponse must be one
response.added_message = f'key |{dict_key}| not list: {type(response_value)}\n'
return False
# it is OK for the value to be a list, but these must be strings (keys) or dicts
for req_list_element, resp_list_element in zip(req_value, response_value):
if isinstance(req_list_element, dict):
if not check_keys(req_list_element, resp_list_element, response):
return False
if not isinstance(req_list_element, str):
response.added_message = f'req_list_element not string: {type(req_list_element)}\n'
return False
if req_list_element not in response_value:
response.added_message = f'key |{req_list_element}| missing from response list\n'
return False
# put None as a dummy value (otherwise something like {'my_key'} will be seen as a set, not a dict
elif req_value != None:
response.added_message = f'required value of key |{dict_key}| must be None (dummy), dict or list: {type(req_value)}\n'
return False
return True
def process_json_requests_call(verb, url, **kwargs):
# "call_name" is a mandatory kwarg
if 'call_name' not in kwargs:
raise Exception('kwarg "call_name" not supplied!')
call_name = kwargs['call_name']
del kwargs['call_name']
required_keys = {}
if 'required_keys' in kwargs:
required_keys = kwargs['required_keys']
del kwargs['required_keys']
acceptable_statuses = [200]
if 'acceptable_statuses' in kwargs:
acceptable_statuses = kwargs['acceptable_statuses']
del kwargs['acceptable_statuses']
exception_handler = log_response_error
if 'exception_handler' in kwargs:
exception_handler = kwargs['exception_handler']
del kwargs['exception_handler']
response, exception = requests_call(verb, url, **kwargs)
if response == None:
exception_handler('No', call_name, exception, verb, url, **kwargs)
return (False, exception)
try:
response_json = response.json()
except BaseException as e:
logger.error(f'response.status_code {response.status_code} but calling json() raised exception')
# an exception raised at this point can't truthfully lead to a "No response" message... so say "bad"
exception_handler('Bad', call_name, e, verb, url, **kwargs)
return (False, response)
status_ok = response.status_code in acceptable_statuses
if not status_ok:
response.added_message = f'status code was {response.status_code}'
log_response_error('Bad', call_name, response, verb, url, **kwargs)
return (False, response)
check_result = check_keys(required_keys, response_json, response)
if not check_result:
log_response_error('Anomalous', call_name, response, verb, url, **kwargs)
return (check_result, response)
Example call (NB with this version, the "deliverable" is either an exception or a response which delivers a json structure):
success, deliverable = utilities.process_json_requests_call('get',
f'{ES_URL}{INDEX_NAME}/_doc/1',
call_name=f'checking index {INDEX_NAME}',
required_keys={'_source':{'status_text': None}})
if not success: return False
# here, we know the deliverable is a response, not an exception
# we also don't need to check for the keys being present:
# the generic code has checked that all expected keys are present
index_status = deliverable.json()['_source']['status_text']
if index_status != 'successfully completed':
# ... i.e. an example of a 200 response, but an error nonetheless
msg = f'Error response: ES index {INDEX_NAME} does not seem to have been built OK: cannot search'
MainWindow.the().visual_log(msg)
logger.error(f'index |{INDEX_NAME}|: deliverable.json() {json.dumps(deliverable.json(), indent=4)}')
return False
So the "visual log" message seen by the user in the case of missing key "status_text", for example, would be "Anomalous response checking index XYZ. See log." (and the log would give a more detailed technical message, constructed automatically, including the stack trace but also details of the missing key in question).
NB
mandatory kwarg: call_name; optional kwargs: required_keys, acceptable_statuses, exception_handler.
the required_keys dict can be nested to any depth
finer-grained exception-handling can be accomplished by including a function exception_handler in kwargs (though don't forget that requests_call will have logged the call details, the exception type and __str__, and the stack trace).
in the above I also implement a check on key "data" in any kwargs which may be logged. This is because a bulk operation (e.g. to populate an index in the case of Elasticsearch) can consist of enormous strings. So curtail to the first 500 characters, for example.
PS Yes, I do know about the elasticsearch Python module (a "thin wrapper" around requests). All the above is for illustration purposes.

Related

I have a function for a scraper that connects to a webpage, and checks the response code (anything within 200 fine, anything else not ok). The function retries the connection, when it has a connection error or an SSLerror, and tries it again and again until the try limit has been reached. Within my try block, I try to validate the response with if else statement. If the response is ok, the response is returned, otherwise, the else statement should print the response code, but also execute the except block. Should this be done by manually raising an exception, and calling this in the except block as such?
#Try to get the url at least ten times, in case it times out
def retries(scraper, json, headers, record_url, tries=10):
for i in range(tries):
try:
response=scraper.post("https://webpageurl/etc", json=json, headers=headers)
if response.ok:
print ('OK!')
return response
else:
print (str(response))
raise Exception("Bad Response Code")
except (Exception, ConnectionError, SSLError):
if i < tries - 1:
sleep(randint(1, 2))
continue
else:
return 'Con error'

Since this is not really an exception, you could just check for a problem condition after the try .. except block:
#Try to get the url at least ten times, in case it times out
def retries(scraper, json, headers, record_url, tries=10):
problem = False
for i in range(tries):
try:
response=scraper.post("https://webpageurl/etc", json=json, headers=headers)
if response.ok:
print ('OK!')
return response
else:
print (str(response))
problem = True
except (Exception, ConnectionError, SSLError):
problem = True
if problem:
if i < tries - 1:
sleep(randint(1, 2))
continue
else:
return 'Con error'
Since your try..except is within the for loop, it would continue regardless, I assumed that was by design.
In the example above, I removed the "Bad Response Code" text, since you made no use of it anyway (you don't access the exception, and don't reraise it), but of course instead of a flag, you could also pass a specific problem code, or even a message.
The advantage of the if after the except is that no exception has to be raised to achieve the same, which is a much more expensive operation.
However, I'm not saying the above is always preferred - the advantage of an exception is that you could reraise it as needed and you can catch it outside the function instead of inside, if the exception would require it.

I am a newbie in python software development. I have created a script which includes many functions returning output of a request and handling the exceptions generated by them. One of the func. is as below:
def ApiResponse(emailHash):
r=None
errorMessage = ""
url = "https://dummy.com"
try:
r = requests.get(url, timeout = TIMEOUT_LIMIT)
except requests.exceptions.Timeout as e:
errorMessage = "Timeout Error"
except requests.ConnectionError as e:
errorMessage = "ConnectionError"
except requests.RequestException as e:
errorMessage = "Some other Request Exception"
return r
I have many such functions returning response for different requests. But, they all repeat the exception handling code. I want to handle exceptions separately in another function so that I don't have to repeat myself. I have tried stuff like passing on the variable "e" to other function and comparing it to the exception types and return different error messages accordingly but I cant seem to compare, lets say, e == requests.exceptions.Timeout.
What else can i do just to separate out that exception handling part? I aim to write a modular code!

You can use exception-decouple package for good readbility:
from exception_decouple import redirect_exceptions
def timeout_handler(emailHash, e):
return "Timeout Error"
def default_handler(emailHash, e):
return "Other Error"
#redirect_exceptions(timeout_handler, requests.exceptions.Timeout)
#redirect_exceptions(default_handler, requests.ConnectionError,
requests.RequestException)
def ApiResponse(emailHash):
r=None
url = "https://dummy.com"
r = requests.get(url, timeout = TIMEOUT_LIMIT)
return r

one of the ways I found was to compare the type of the error with the error. for example in the above case.
def exception_handler(e):
if type(e) is requests.exceptions.ConnectTimeout:
return "Timeout Error"
elif type(e) is requests.ConnectionError:
return "ConnectionError"
else:
return "Some other Request Exception"
this should work

I have a simple function that sends a get request to an API, and I want to be able to write a Unit test to assert that the scriprt prints an error and then exits if the request returns an exception.
def fetch_members(self):
try:
r = requests.get(api_url, auth=('user', api_key))
except requests.exceptions.HTTPError as error:
print(error)
sys.exit(1)
I've tried the following test but the HTTPError is never raised
import requests
from unittest.mock import patch
#patch('sys.exit')
#patch('sys.stdout', new_callable=StringIO)
#patch('requests.get')
def test_fetch_members_with_bad_request(self, mock_get, mock_stdout,
mock_exit):
response = requests.Response()
response.status_code = 400 # Bad reqeust
mock_get.return_value = response
mock_get.side_effect = requests.exceptions.HTTPError
with self.assertRaises(requests.exceptions.HTTPError):
fetch_members()
mock_exit.assert_called_once_with(1)
self.assertIsNotNone(mock_stdout.getvalue())
How would I go about correctly patching requests.get to always raise some sort of error?

Your function catches and handles the exception:
except requests.exceptions.HTTPError as error:
This means it'll never propagate further, so your assertRaises() fails. Just assert that sys.exit() has been called, that's enough.
There is also no point in setting a return value; the call raises an exception instead. The following should suffice:
#patch('sys.exit')
#patch('sys.stdout', new_callable=StringIO)
#patch('requests.get')
def test_fetch_members_with_bad_request(self, mock_get, mock_stdout,
mock_exit):
mock_get.side_effect = requests.exceptions.HTTPError
fetch_members()
mock_exit.assert_called_once_with(1)
self.assertIsNotNone(mock_stdout.getvalue())

I am trying to handle jira-python exception but my try, except does not seem to catch it. I also need to add more lines in order to be able to post this. So there they are, the lines.
try:
new_issue = jira.create_issue(fields=issue_dict)
stdout.write(str(new_issue.id))
except jira.exceptions.JIRAError:
stdout.write("JIRAError")
exit(1)
Here is the code that raises the exception:
import json
class JIRAError(Exception):
"""General error raised for all problems in operation of the client."""
def __init__(self, status_code=None, text=None, url=None):
self.status_code = status_code
self.text = text
self.url = url
def __str__(self):
if self.text:
return 'HTTP {0}: "{1}"\n{2}'.format(self.status_code, self.text, self.url)
else:
return 'HTTP {0}: {1}'.format(self.status_code, self.url)
def raise_on_error(r):
if r.status_code >= 400:
error = ''
if r.text:
try:
response = json.loads(r.text)
if 'message' in response:
# JIRA 5.1 errors
error = response['message']
elif 'errorMessages' in response and len(response['errorMessages']) > 0:
# JIRA 5.0.x error messages sometimes come wrapped in this array
# Sometimes this is present but empty
errorMessages = response['errorMessages']
if isinstance(errorMessages, (list, tuple)):
error = errorMessages[0]
else:
error = errorMessages
elif 'errors' in response and len(response['errors']) > 0:
# JIRA 6.x error messages are found in this array.
error = response['errors']
else:
error = r.text
except ValueError:
error = r.text
raise JIRAError(r.status_code, error, r.url)

I know I'm not answering the question, but I feel I need to warn people that could get confused by that code (as I did)...
Maybe you are trying to write your own version of jira-python or it's an old version?
In any case, here the link to the jira-python code for the JIRAError class
and here the list of codes
to catch the exception from that package I use the below code
from jira import JIRA, JIRAError
try:
...
except JIRAError as e:
print e.status_code, e.text

Maybe it is obvious and that's why you don't have it in your code paste, just in case, you do have
from jira.exceptions import JIRAError
somewhere in your code right?
Don't have enough reputation for comments so I'll add it answer for #arynhard:
I found the docs very light in particular in terms of example, you might find the scripts in this repo useful since they're all leveraging jira-python in someway or another.
https://github.com/eucalyptus/jira-scripts/

I might be wrong, but looks like you're catching jira.exceptions.JIRAError, while raising JIRAError - those are different types. You need to either remove "jira.exceptions." part from your except statement, or raise jira.exceptions.JIRAError instead.

Ok it's a very old one but I faced the same problem and this page is still showing up.
Here is how I trap the Exception, I used the Exception object.
try:
issue = jira.issue('jira-1')
except Exception as e:
if 'EXIST' in e.text:
print 'The issue does not exist'
exit(1)
regards

I might be approaching this the wrong way, but I've got a POST request going out:
response = requests.post(full_url, json.dumps(data))
Which could potentially fail for a number of reasons, some being related to the data, some being temporary failures, which due to a poorly designed endpoint may well return as the same error (server does unpredictable things with invalid data). To catch these temporary failures and let others pass I thought the best way to go about this would be to retry once and then continue if the error is raised again. I believe I could do it with a nested try/except, but it seems like bad practice to me (what if I want to try twice before giving up?)
That solution would be:
try:
response = requests.post(full_url, json.dumps(data))
except RequestException:
try:
response = requests.post(full_url, json.dumps(data))
except:
continue
Is there a better way to do this? Alternately is there a better way in general to deal with potentially faulty HTTP responses?

for _ in range(2):
try:
response = requests.post(full_url, json.dumps(data))
break
except RequestException:
pass
else:
raise # both tries failed
If you need a function for this:
def multiple_tries(func, times, exceptions):
for _ in range(times):
try:
return func()
except Exception as e:
if not isinstance(e, exceptions):
raise # reraises unexpected exceptions
raise # reraises if attempts are unsuccessful
Use like this:
func = lambda:requests.post(full_url, json.dumps(data))
response = multiple_tries(func, 2, RequestException)