Develop Reference

Python Requests timeout parameter is being ignored

I'm using Python 2.7, I want every request to timeout after some seconds, but the requests timeout almost immediately. Following is my code.
requestsTimeout = 5
link = 'http://' + IP + '/api/v1.0/system/info'
while (1):
try:
return requests.get(link, timeout = requestsTimeout)
except requests.exceptions.RequestException as e:
log._print0(_printID, 'getting DAQ Info' ,str(e)) # just printing
time.sleep(0.1)
Now if I disconnect my wifi, I should get a printout of timeout exception after every 5 seconds, but I'm getting prints at a very fast rate (multiple times in one second).

When host is unreachable ConnectionError is raised without waiting time set by timeout. You could overcome this by handling this exception separately:
requestsTimeout = 5
link = 'http://' + IP + '/api/v1.0/system/info'
while True:
try:
return requests.get(link, timeout=requestsTimeout)
except requests.exceptions.ConnectionError as e:
time.sleep(requestsTimeout)
except requests.exceptions.RequestException as e:
log._print0(_printID, 'getting DAQ Info' ,str(e)) # just printing
time.sleep(0.1)

Trouble retrying on interrupted system call

I have recently run into an issue at work where we are having intermittent problems with an internal website not loading due to an interrupted system call. We are using urllib2 to access the website. I can't share the exact code, but here is basically how we do it:
payload = {'userName': user_name,
'emailAddress': email_address,
'password': password}
headers = {'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': token}
values = json.dumps(payload)
req = urllib2.Request(url, values, headers)
try:
response = urllib2.urlopen(req, timeout=30)
break
except IOError, e:
if e.errno != errno.EINTR:
print e.errno
raise
We log the errono and the raised exception. The exception is:
IOError: <urlopen error [Errno 4] Interrupted system call>
And the errno is None. I expected it to be 4.
Is there a better way to catch this error in Python 2.7? I am aware of PEP475, but we cannot upgrade to Python 3 right now.

The <urlopen error [Errno 4] Interrupted system call> indicates it is actually a URLError from urllib2, which subclasses IOError, but handles arguments completely differently. That is why the attributes errno and strerror are not initialized. It both passes strings as reason:
raise URLError("qop '%s' is not supported." % qop)
and wraps exceptions from other sources:
try:
h.request(req.get_method(), req.get_selector(), req.data, headers)
except socket.error, err: # XXX what error?
h.close()
raise URLError(err)
This is why you will not find errno in the usual place:
>>> try:
urlopen('http://asdf')
except URLError, e:
pass
...
>>> e
URLError(gaierror(-2, 'Name or service not known'),)
>>> e.errno
>>> e.reason
gaierror(-2, 'Name or service not known')
>>> e.reason.errno
-2
This worked in this case, but the reason attribute could be a string or a socket.error, which has (had) its own problems with errno.
The definition of URLError in urllib2.py:
class URLError(IOError):
# URLError is a sub-type of IOError, but it doesn't share any of
# the implementation. need to override __init__ and __str__.
# It sets self.args for compatibility with other EnvironmentError
# subclasses, but args doesn't have the typical format with errno in
# slot 0 and strerror in slot 1. This may be better than nothing.
def __init__(self, reason):
self.args = reason,
self.reason = reason
def __str__(self):
return '<urlopen error %s>' % self.reason
So long story short, it's a horrible mess. You have to check e.reason for
Is it just a string? If so, there'll be no errno anywhere.
Is it a socket.error? Handle quirks of that. Again the errno attribute can be unset, or None, since it could also be raised with a single string argument.
Is it a subclass of IOError or OSError (which subclass EnvironmentError)? Read errno attribute of that – and hope for the best.
This can be and probably is overly cautious for your case, but it is good to understand the edges. Tornado had similar issues and is using a utility function to get errno from exception, but unfortunately that function does not work with URLErrors.
What could cover at least some cases:
while True: # or some amount of retries
try:
response = urllib2.urlopen(req, timeout=30)
break
except URLError, e:
if getattr(e.reason, 'errno', None) == errno.EINTR:
# Retry
continue

Python - How to capture gevent socket timeout exception

import gevent.monkey
gevent.monkey.patch_socket()
import requests
from gevent.pool import Pool
import socket
urls = ["http://www.iraniansingles.com"]
def check_urls(urls):
pool = Pool(1)
for url in urls:
pool.spawn(fetch, url)
pool.join()
def fetch(url):
print url
try:
resp = requests.get(url, verify=False, timeout=5.0)
print resp.status_code
except socket.timeout:
print "SocketTimeout"
check_urls(urls)
If I remove the first 2 lines, my program printing SocketTimeout. But with monkeypatch, my program waits forever.
Can someone tell me how to capture that socket timeout exception with monkeypatch?

Problem was gevent default timeout set to None. So we have to set default socket timeout manually.
from gevent import socket
socket.setdefaulttimeout(5)

Python: checking internet connection (more than once)

I have implemented a quick solution to check for internet connection in one python program, using what I found on SO :
def check_internet(self):
try:
response=urllib2.urlopen('http://www.google.com',timeout=2)
print "you are connected"
return True
except urllib2.URLError as err:
print err
print "you are disconnected"
It works well ONCE, and show that I am not connected if I try it once. But if I re-establish the connection and try again, then it still says I am not connected.
Is the urllib2 connection not closed somehow ? Should I do something to reset it ?

This could be because of server-side caching.
Try this:
def check_internet(self):
try:
header = {"pragma" : "no-cache"} # Tells the server to send fresh copy
req = urllib2.Request("http://www.google.com", headers=header)
response=urllib2.urlopen(req,timeout=2)
print "you are connected"
return True
except urllib2.URLError as err:
print err
I haven't tested it. But according to the 'pragma' definition, it should work.
There is a good discussion here if you want to know about pragma: Difference between Pragma and Cache-control headers?

This is how I used to check my connectivity for one of my applications.
import httplib
import socket
test_con_url = "www.google.com" # For connection testing
test_con_resouce = "/intl/en/policies/privacy/" # may change in future
test_con = httplib.HTTPConnection(test_con_url) # create a connection
try:
test_con.request("GET", test_con_resouce) # do a GET request
response = test_con.getresponse()
except httplib.ResponseNotReady as e:
print "Improper connection state"
except socket.gaierror as e:
print "Not connected"
else:
print "Connected"
test_con.close()
I tested the code enabling/disabling my LAN connection repeatedly and it works.

It will be faster to just make a HEAD request so no HTML will be fetched.
Also I am sure google would like it better this way :)
# uncomment for python2
# import httplib
import http.client as httplib
def have_internet():
conn = httplib.HTTPConnection("www.google.com")
try:
conn.request("HEAD", "/")
return True
except:
return False

How to handle timeouts with httplib (python 2.6)?

I'm using httplib to access an api over https and need to build in exception handling in the event that the api is down.
Here's an example connection:
connection = httplib.HTTPSConnection('non-existent-api.com', timeout=1)
connection.request('POST', '/request.api', xml, headers={'Content-Type': 'text/xml'})
response = connection.getresponse()
This should timeout, so I was expecting an exception to be raised, and response.read() just returns an empty string.
How can I know if there was a timeout? Even better, what's the best way to gracefully handle the problem of a 3rd-party api being down?

Even better, what's the best way to gracefully handle the problem of a 3rd-party api being down?
what's mean API is down , API return http 404 , 500 ...
or you mean when the API can't be reachable ?
first of all i don't think you can know if a web service in general is down before trying to access it so i will recommend for first one you can do like this:
import httplib
conn = httplib.HTTPConnection('www.google.com') # I used here HTTP not HTTPS for simplify
conn.request('HEAD', '/') # Just send a HTTP HEAD request
res = conn.getresponse()
if res.status == 200:
print "ok"
else:
print "problem : the query returned %s because %s" % (res.status, res.reason)
and for checking if the API is not reachable i think you will be better doing a try catch:
import httplib
import socket
try:
# I don't think you need the timeout unless you want to also calculate the response time ...
conn = httplib.HTTPSConnection('www.google.com')
conn.connect()
except (httplib.HTTPException, socket.error) as ex:
print "Error: %s" % ex
You can mix the two ways if you want something more general ,Hope this will help

urllib and httplib don't expose timeout. You have to include socket and set the timeout there:
import socket
socket.settimeout(10) # or whatever timeout you want

This is what I found to be working correctly with httplib2. Posting it as it might still help someone :
import httplib2, socket
def check_url(url):
h = httplib2.Http(timeout=0.1) #100 ms timeout
try:
resp = h.request(url, 'HEAD')
except (httplib2.HttpLib2Error, socket.error) as ex:
print "Request timed out for ", url
return False
return int(resp[0]['status']) < 400