Develop Reference

How to avoid many requests refused in using Python requests package?

I use requests package to fetch some earthquake catalog from website: ISC earthquake bulletin
When the content table is small, all good. But when it comes to massive search, or a loop search, I.e., I set different parameters to run the requests in a loop. It will return no available data:
Sorry, but your request cannot be processed at the present time. Please try again in a few minutes.
Can anyone told me how can I avoid too many refused requests?
Here’s my scripts:
# import package
import requests
Url = ‘http://www.isc.ac.uk/cgi-bin/web-db-v4?iscreview=on&out_format=CSV&ttime=on&ttres=on&tdef=on&amps=on&phaselist=&stnsearch=STN&sta_list=CLC&stn_ctr_lat=&stn_ctr_lon=&stn_radius=&max_stn_dist_units=deg&stn_top_lat=&stn_bot_lat=&stn_left_lon=&stn_right_lon=&stn_srn=&stn_grn=&bot_lat=&top_lat=&left_lon=&right_lon=&ctr_lat=&ctr_lon=&radius=&max_dist_units=deg&searchshape=GLOBAL&srn=&grn=&start_year=2009&start_month=7&start_day=01&start_time=00%3A00%3A00&end_year=2019&end_month=8&end_day=01&end_time=00%3A00%3A00&min_dep=&max_dep=&min_mag=6.0&max_mag=6.9&req_mag_type=Any&req_mag_agcy=Any&include_links=on&request=STNARRIVALS’
R.requests(URL)
print(R.text)

Use the Retry mechanism of HTTPAdapter to automatically re-send the request when a temporary failure happened. Some settings you maybe interested at:
total - Total number of retries to allow. If the limit is reached without a successful response, then the request is considered a failure.
backoff_factor - Since failed requests usually happens when the server is loaded, it would be beneficial to set a delay in between retries so that the server can breathe. Think of it as the 1st request would happen at the 1st second, the 2nd request at the 2nd second, the 3rd request at the 4th second, the 4th request at the 8th second, the 5th request at the 16th second, and so on up until the configured BACKOFF_MAX.
allowed_methods - The HTTP methods that you want to retry.
status_forcelist - The HTTP status codes that should be retried. Commonly, this is the 5xx series since those are the errors that originated from the server and might be successful if retried.
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
retry_strategy = Retry(
total=5,
backoff_factor=0.1,
allowed_methods=["GET"],
status_forcelist=[500, 502, 504],
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)
response = http.get("https://google.com")
print(response.status_code)
In this example, if ever the response fails, we are sure that a series of 5 retries was made separated through time by a factor of 0.1 but all still failed. But if the server failure isn't persistent, it is highly likely that this would be successful due to the number of retries made separated through time.

403 (Forbidden) error when using Fortnite Tracker api with urllib.request

I am attempting to get user statistics from the Fortnite tracker api.
I have an api key and am using the correct url as indicated in the documentation
Template url:
https://api.fortnitetracker.com/v1/profile/{platform}/{epic-nickname}
Desired url:
https://api.fortnitetracker.com/v1/profile/pc/xantium0
If I use this link in browser I get {"message":"No API key found in request"} (as I have not passed the API key) so the link should be correct. Also if I do not pass the api key with urllib then I still get a 403 error.
I have checked out how to pass a header in a request: How do I set headers using python's urllib?
and so far have this code:
import urllib.request as ur
request = ur.Request('https://api.fortnitetracker.com/v1/profile/pc/xantium0', headers={'TRN-Api-Key' : 'xxx'})
response = ur.urlopen(request)
print(response.read())
When run I get this error:
urllib.error.HTTPError: HTTP Error 403: Forbidden
403 checks out as:
HTTP 403 is a standard HTTP status code communicated to clients by an HTTP server to indicate that the server understood the request, but will not fulfill it. There are a number of sub-status error codes that provide a more specific reason for responding with the 403 status code.
https://en.wikipedia.org/wiki/HTTP_403
The response is the same if I don't pass the api key in the header.
I can only think of three reasons this code is not working:
I have passed the wrong header name (i.e. it's not TRN-Api-Key)
My code is incorrect and I am not actually passing a header to the server
I have been banned
My problem is that I think my code is correct:
From the documentation:
urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)
I have passed the url and I have passed the headers (wihout confusing with the data arguement). The api documentation also mentions it should be passed in the headers.
I am also quite sure I need to use the TRN-Api-Key as it is shown in the api documentation:
TRN-Api-Key: xxx
Also in this question (using Ruby):
header = {
key: "TRN-Api-Key: Somelong-api-key-here"
}
Or I have been banned (this is possible although I got the key 15 minutes ago) is there a way to check? Would this error be returned?
What is preventing me from getting the user statistics?

Try using requests, a pythonic, fast and widely used module.
import requests
url = 'https://api.fortnitetracker.com/v1/profile/pc/xantium0'
headers = {
'TRN-Api-Key' : 'xxx'
}
response = requests(url, headers=headers)
print('Requests was successful:', response.ok)
print(response.text)
If it doesn't work you can visit the url with your browser, then check the requests:
in Firefox press Cntrl+Shift+E, in Chrome Cntrl+E (or Inspect with Cntrl+Shift+I and then go to Network). Press on "https://api.fortnitetracker.com/v1/profile/pc/xantium0" and change the headers. On Firefox there's the button Modify and resend. Check the response and eventually, try to change the header api key name.
Hope this helps, let me know.

HTTP Error 307 - Temporary redirect in python script [duplicate]

I'm using Python 3.7 with urllib.
All work fine but it seems not to athomatically redirect when it gets an http redirect request (307).
This is the error i get:
ERROR 2020-06-15 10:25:06,968 HTTP Error 307: Temporary Redirect
I've to handle it with a try-except and manually send another request to the new Location: it works fine but i don't like it.
These is the piece of code i use to perform the request:
req = urllib.request.Request(url)
req.add_header('Authorization', auth)
req.add_header('Content-Type','application/json; charset=utf-8')
req.data=jdati
self.logger.debug(req.headers)
self.logger.info(req.data)
resp = urllib.request.urlopen(req)
url is an https resource and i set an header with some Authhorization info and content-type.
req.data is a JSON
From urllib documentation i've understood that the redirects are authomatically performed by the the library itself, but it doesn't work for me. It always raises an http 307 error and doesn't follow the redirect URL.
I've also tried to use an opener specifiyng the default redirect handler, but with the same result
opener = urllib.request.build_opener(urllib.request.HTTPRedirectHandler)
req = urllib.request.Request(url)
req.add_header('Authorization', auth)
req.add_header('Content-Type','application/json; charset=utf-8')
req.data=jdati
resp = opener.open(req)
What could be the problem?

The reason why the redirect isn't done automatically has been correctly identified by yours truly in the discussion in the comments section. Specifically, RFC 2616, Section 10.3.8 states that:
If the 307 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
Back to the question - given that data has been assigned, this automatically results in get_method returning POST (as per how this method was implemented), and since that the request method is POST, and the response code is 307, an HTTPError is raised instead as per the above specification. In the context of Python's urllib, this specific section of the urllib.request module raises the exception.
For an experiment, try the following code:
import urllib.request
import urllib.parse
url = 'http://httpbin.org/status/307'
req = urllib.request.Request(url)
req.data = b'hello' # comment out to not trigger manual redirect handling
try:
resp = urllib.request.urlopen(req)
except urllib.error.HTTPError as e:
if e.status != 307:
raise # not a status code that can be handled here
redirected_url = urllib.parse.urljoin(url, e.headers['Location'])
resp = urllib.request.urlopen(redirected_url)
print('Redirected -> %s' % redirected_url) # the original redirected url
print('Response URL -> %s ' % resp.url) # the final url
Running the code as is may produce the following
Redirected -> http://httpbin.org/redirect/1
Response URL -> http://httpbin.org/get
Note the subsequent redirect to get was done automatically, as the subsequent request was a GET request. Commenting out req.data assignment line will result in the lack of the "Redirected" output line.
Other notable things to note in the exception handling block, e.read() may be done to retrieve the response body produced by the server as part of the HTTP 307 response (since data was posted, there might be a short entity in the response that may be processed?), and that urljoin is needed as the Location header may be a relative URL (or simply has the host missing) to the subsequent resource.
Also, as a matter of interest (and for linkage purposes), this specific question has been asked multiple times before and I am rather surprised that they never got any answers, which follows:
How to handle 307 redirection using urllib2 from http to https
HTTP Error 307: Temporary Redirect in Python3 - INTRANET
HTTP Error 307 - Temporary redirect in python script

Mock a HTTP request that times out with HTTPretty

Using the HTTPretty library for Python, I can create mock HTTP responses of choice and then pick them up i.e. with the requests library like so:
import httpretty
import requests
# set up a mock
httpretty.enable()
httpretty.register_uri(
method=httpretty.GET,
uri='http://www.fakeurl.com',
status=200,
body='My Response Body'
)
response = requests.get('http://www.fakeurl.com')
# clean up
httpretty.disable()
httpretty.reset()
print(response)
Out: <Response [200]>
Is there also the possibility to register an uri which cannot be reached (e.g. connection timed out, connection refused, ...) such that no response is received at all (which is not the same as an established connection which gives an HTTP error code like 404)?
I want to use this behaviour in unit testing to ensure that my error handling works as expected (which does different things in case of 'no connection established' and 'connection established, bad bad HTTP status code'). As a workaround, I could try to connect to an invalid server like http://192.0.2.0 which would time out in any case. However, I would prefer to do all my unit testing without using any real network connections.

Meanwhile I got it, using a HTTPretty callback body seems to produce the desired behaviour. See inline comments below.
This is actually not exactly the same as I was looking for (it is not a server that cannot be reached and hence the request times out but a server that throws a timeout exception once it is reached, however, the effect is the same for my usecase.
Still, if anybody knows a different solution, I'm looking forward to it.
import httpretty
import requests
# enable HTTPretty
httpretty.enable()
# create a callback body that raises an exception when opened
def exceptionCallback(request, uri, headers):
# raise your favourite exception here, e.g. requests.ConnectionError or requests.Timeout
raise requests.Timeout('Connection timed out.')
# set up a mock and use the callback function as response's body
httpretty.register_uri(
method=httpretty.GET,
uri='http://www.fakeurl.com',
status=200,
body=exceptionCallback
)
# try to get a response from the mock server and catch the exception
try:
response = requests.get('http://www.fakeurl.com')
except requests.Timeout as e:
print('requests.Timeout exception got caught...')
print(e)
# do whatever...
# clean up
httpretty.disable()
httpretty.reset()

Can I set max_retries for requests.request?

The Python requests module is simple and elegant but one thing bugs me.
It is possible to get a requests.exception.ConnectionError with a message like:
Max retries exceeded with url: ...
This implies that requests can attempt to access the data several times. But there is not a single mention of this possibility anywhere in the docs. Looking at the source code I didn't find any place where I could alter the default (presumably 0) value.
So is it possible to somehow set the maximum number of retries for requests?

This will not only change the max_retries but also enable a backoff strategy which makes requests to all http:// addresses sleep for a period of time before retrying (to a total of 5 times):
import requests
from requests.adapters import HTTPAdapter, Retry
s = requests.Session()
retries = Retry(total=5,
backoff_factor=0.1,
status_forcelist=[ 500, 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))
s.get('http://httpstat.us/500')
As per documentation for Retry: if the backoff_factor is 0.1, then sleep() will sleep for [0.05s, 0.1s, 0.2s, 0.4s, ...] between retries. It will also force a retry if the status code returned is 500, 502, 503 or 504.
Various other options to Retry allow for more granular control:
total – Total number of retries to allow.
connect – How many connection-related errors to retry on.
read – How many times to retry on read errors.
redirect – How many redirects to perform.
method_whitelist – Set of uppercased HTTP method verbs that we should retry on.
status_forcelist – A set of HTTP status codes that we should force a retry on.
backoff_factor – A backoff factor to apply between attempts.
raise_on_redirect – Whether, if the number of redirects is exhausted, to raise a MaxRetryError, or to return a response with a response code in the 3xx range.
raise_on_status – Similar meaning to raise_on_redirect: whether we should raise an exception, or return a response, if status falls in status_forcelist range and retries have been exhausted.
NB: raise_on_status is relatively new, and has not made it into a release of urllib3 or requests yet. The raise_on_status keyword argument appears to have made it into the standard library at most in python version 3.6.
To make requests retry on specific HTTP status codes, use status_forcelist. For example, status_forcelist=[503] will retry on status code 503 (service unavailable).
By default, the retry only fires for these conditions:
Could not get a connection from the pool.
TimeoutError
HTTPException raised (from http.client in Python 3 else httplib).
This seems to be low-level HTTP exceptions, like URL or protocol not
formed correctly.
SocketError
ProtocolError
Notice that these are all exceptions that prevent a regular HTTP response from being received. If any regular response is generated, no retry is done. Without using the status_forcelist, even a response with status 500 will not be retried.
To make it behave in a manner which is more intuitive for working with a remote API or web server, I would use the above code snippet, which forces retries on statuses 500, 502, 503 and 504, all of which are not uncommon on the web and (possibly) recoverable given a big enough backoff period.

It is the underlying urllib3 library that does the retrying. To set a different maximum retry count, use alternative transport adapters:
from requests.adapters import HTTPAdapter
s = requests.Session()
s.mount('http://stackoverflow.com', HTTPAdapter(max_retries=5))
The max_retries argument takes an integer or a Retry() object; the latter gives you fine-grained control over what kinds of failures are retried (an integer value is turned into a Retry() instance which only handles connection failures; errors after a connection is made are by default not handled as these could lead to side-effects).
Old answer, predating the release of requests 1.2.1:
The requests library doesn't really make this configurable, nor does it intend to (see this pull request). Currently (requests 1.1), the retries count is set to 0. If you really want to set it to a higher value, you'll have to set this globally:
import requests
requests.adapters.DEFAULT_RETRIES = 5
This constant is not documented; use it at your own peril as future releases could change how this is handled.
Update: and this did change; in version 1.2.1 the option to set the max_retries parameter on the HTTPAdapter() class was added, so that now you have to use alternative transport adapters, see above. The monkey-patch approach no longer works, unless you also patch the HTTPAdapter.__init__() defaults (very much not recommended).

Be careful, Martijn Pieters's answer isn't suitable for version 1.2.1+. You can't set it globally without patching the library.
You can do this instead:
import requests
from requests.adapters import HTTPAdapter
s = requests.Session()
s.mount('http://www.github.com', HTTPAdapter(max_retries=5))
s.mount('https://www.github.com', HTTPAdapter(max_retries=5))

After struggling a bit with some of the answers here, I found a library called backoff that worked better for my situation. A basic example:
import backoff
#backoff.on_exception(
backoff.expo,
requests.exceptions.RequestException,
max_tries=5,
giveup=lambda e: e.response is not None and e.response.status_code < 500
)
def publish(self, data):
r = requests.post(url, timeout=10, json=data)
r.raise_for_status()
I'd still recommend giving the library's native functionality a shot, but if you run into any problems or need broader control, backoff is an option.

A cleaner way to gain higher control might be to package the retry stuff into a function and make that function retriable using a decorator and whitelist the exceptions.
I have created the same here:
http://www.praddy.in/retry-decorator-whitelisted-exceptions/
Reproducing the code in that link :
def retry(exceptions, delay=0, times=2):
"""
A decorator for retrying a function call with a specified delay in case of a set of exceptions
Parameter List
-------------
:param exceptions: A tuple of all exceptions that need to be caught for retry
e.g. retry(exception_list = (Timeout, Readtimeout))
:param delay: Amount of delay (seconds) needed between successive retries.
:param times: no of times the function should be retried
"""
def outer_wrapper(function):
#functools.wraps(function)
def inner_wrapper(*args, **kwargs):
final_excep = None
for counter in xrange(times):
if counter > 0:
time.sleep(delay)
final_excep = None
try:
value = function(*args, **kwargs)
return value
except (exceptions) as e:
final_excep = e
pass #or log it
if final_excep is not None:
raise final_excep
return inner_wrapper
return outer_wrapper
#retry(exceptions=(TimeoutError, ConnectTimeoutError), delay=0, times=3)
def call_api():

You can use the requests library to accomplish all in one go.
The following code will retry 3 times if you receive 429,500,502,503 or 504 status code, each time with a longer delay set through "backoff_factor". See https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/ for a nice tutorial.
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
retry_strategy = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
method_whitelist=["HEAD", "GET", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)
response = http.get("https://en.wikipedia.org/w/api.php")