In case of a connection error, I want Python to wait and re-try. Here's the relevant code, where "link" is some link:
import requests
import urllib.request
import urllib.parse
from random import randint
try:
r=requests.get(link)
except ConnectionError or TimeoutError:
print("Will retry again in a little bit")
time.sleep(randint(2500,3000))
r=requests.get(link)
Except I still periodically get a connection error. And I never see the text "Will retry again in a little bit" so I know the code is not re-trying. What am I doing wrong? I'm pasting parts of the error code below in case I'm misreading the error. TIA!
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))
During handling of the above exception, another exception occurred:
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))
For me, using a custom user agent in the request fixes this issue. With this method you spoof your browser.
Works:
url = "https://www.nasdaq.com/market-activity/stocks/amd"
headers = {'User-Agent': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4'}
response = requests.get(url, headers=headers)
Doesn't work:
url = "https://www.nasdaq.com/market-activity/stocks/amd"
response = requests.get(url)
The second request is not inside a try block so exceptions are not caught. Also in the try-except block you're not catching other exceptions that may occur.
You could use a loop to attempt a connection two times, and break if the request is successful.
for _ in range(2):
try:
r = requests.get(link)
break
except (ConnectionError, TimeoutError):
print("Will retry again in a little bit")
except Exception as e:
print(e)
time.sleep(randint(2500,3000))
I think you should use
except (ConnectionError, TimeoutError) as e:
print("Will retry again in a little bit")
time.sleep(randint(2500,3000))
r=requests.get(link)
See this similar question, or check the docs.
I had the same problem. It turns out that urlib3 relies on socket.py, which raises an OSError. So, you need to catch that:
try:
r = requests.get(link)
except OSError as e:
print("There as an error: {}".format(e))
Related
import urllib.request
request = urllib.request.Request('http://1.0.0.8/')
try:
response = urllib.request.urlopen(request)
print("Server Online")
#do stuff here
except urllib.error.HTTPError as e: # 404, 500, etc..
print("Server Offline")
#do stuff here
I'm trying to write a simple program that will check a list of LAN webserver is up. Currently just using one IP for now.
When I run it with an IP of a web server I get back Server Online.
When I run it with a IP that doesn't have web server I get
"urllib.error.URLError: <urlopen error [WinError 10061] No connection could be made because the target machine actively refused it>"
but would rather a simple "Server Offline" output. Not sure how to get the reply to output Server Offline.
In your code above you’re just looking for HTTPError exceptions.
Just add another except clause to the end that would reference the exception that you are looking for, in this case the URLError:
import urllib.request
request = urllib.request.Request('http://1.0.0.8/')
try:
response = urllib.request.urlopen(request)
print("Server Online")
#do stuff here
except urllib.error.HTTPError as e:
print("Server Offline")
#do stuff here
except urllib.error.URLError as e:
print("Server Offline")
#do stuff here
well, we could combine the errors as well.
except (urllib.error.URLError, urllib.error.HTTPError):
print("Poor server is offline.")
My script in Python 2.7 scrapes a website every minute, but sometimes it gives the error:
urlopen error [Errno 54] Connection reset by peer>
How I can handle the exception? I am trying something like this:
from socket import error as SocketError
import errno
try:
response = urllib2.urlopen(request).read()
except SocketError as e:
if e.errno != errno.ECONNRESET:
print "there is a time-out"
pass
print "There is no time-out, continue"
You can handle the exception like this:
from socket import error as SocketError
import errno
try:
urllib2.urlopen(request).read()
except SocketError, e:
errorcode = e[0]
if errorcode!=errno.ECONNREFUSED:
print "There is a time-out"
# Not the error we are looking for, re-raise
raise e
You can read Error Code
I have recently run into an issue at work where we are having intermittent problems with an internal website not loading due to an interrupted system call. We are using urllib2 to access the website. I can't share the exact code, but here is basically how we do it:
payload = {'userName': user_name,
'emailAddress': email_address,
'password': password}
headers = {'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': token}
values = json.dumps(payload)
req = urllib2.Request(url, values, headers)
try:
response = urllib2.urlopen(req, timeout=30)
break
except IOError, e:
if e.errno != errno.EINTR:
print e.errno
raise
We log the errono and the raised exception. The exception is:
IOError: <urlopen error [Errno 4] Interrupted system call>
And the errno is None. I expected it to be 4.
Is there a better way to catch this error in Python 2.7? I am aware of PEP475, but we cannot upgrade to Python 3 right now.
The <urlopen error [Errno 4] Interrupted system call> indicates it is actually a URLError from urllib2, which subclasses IOError, but handles arguments completely differently. That is why the attributes errno and strerror are not initialized. It both passes strings as reason:
raise URLError("qop '%s' is not supported." % qop)
and wraps exceptions from other sources:
try:
h.request(req.get_method(), req.get_selector(), req.data, headers)
except socket.error, err: # XXX what error?
h.close()
raise URLError(err)
This is why you will not find errno in the usual place:
>>> try:
urlopen('http://asdf')
except URLError, e:
pass
...
>>> e
URLError(gaierror(-2, 'Name or service not known'),)
>>> e.errno
>>> e.reason
gaierror(-2, 'Name or service not known')
>>> e.reason.errno
-2
This worked in this case, but the reason attribute could be a string or a socket.error, which has (had) its own problems with errno.
The definition of URLError in urllib2.py:
class URLError(IOError):
# URLError is a sub-type of IOError, but it doesn't share any of
# the implementation. need to override __init__ and __str__.
# It sets self.args for compatibility with other EnvironmentError
# subclasses, but args doesn't have the typical format with errno in
# slot 0 and strerror in slot 1. This may be better than nothing.
def __init__(self, reason):
self.args = reason,
self.reason = reason
def __str__(self):
return '<urlopen error %s>' % self.reason
So long story short, it's a horrible mess. You have to check e.reason for
Is it just a string? If so, there'll be no errno anywhere.
Is it a socket.error? Handle quirks of that. Again the errno attribute can be unset, or None, since it could also be raised with a single string argument.
Is it a subclass of IOError or OSError (which subclass EnvironmentError)? Read errno attribute of that – and hope for the best.
This can be and probably is overly cautious for your case, but it is good to understand the edges. Tornado had similar issues and is using a utility function to get errno from exception, but unfortunately that function does not work with URLErrors.
What could cover at least some cases:
while True: # or some amount of retries
try:
response = urllib2.urlopen(req, timeout=30)
break
except URLError, e:
if getattr(e.reason, 'errno', None) == errno.EINTR:
# Retry
continue
I am trying to use requests_futures (https://github.com/ross/requests-futures) for asynchronous requests which seems to work fine. The only problem is, it doesn't throw any exceptions for me (i.e. TimeOut Exception). The code I used is:
from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession
session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))
def callback(sess, resp):
# Print the ip address in callback
print 'IP', resp.text
proxy = {'http': 'http://176.194.189.57:8080'}
try:
future = session.get('http://api.ipify.org', background_callback=callback, timeout=5, proxies=proxy)
except Exception as e:
print "Error %s" % e
# future2 = session.get('http://api.ipify.org', background_callback=callback, timeout=5)
The first session.get() should throw an Exception as it isn't a valid proxy.
For the exception to be raised, you have to check the result() method of the future object you just created.
Hey I'm wondering how to handle specific error codes. For example, [Errno 111] Connection refused
I want to catch this specific error in the socket module and print something.
If you want to get the error code, this seems to do the trick;
import errno
try:
socket_connection()
except socket.error as error:
if error.errno == errno.ECONNREFUSED:
print(os.strerror(error.errno))
else:
raise
You can look up errno error codes.
On Unix platforms, at least, you can do the following.
import socket, errno
try:
# Do something...
except socket.error as e:
if e.errno == errno.ECONNREFUSED:
# Handle the exception...
else:
raise
Before Python 2.6, use e.args[ 0 ] instead of e.errno.
This seems hard to do reliably/portably but perhaps something like:
import socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 4167))
except socket.error, e:
if 'Connection refused' in e:
print '*** Connection refused ***'
which yields:
$ python socketexception.py
*** Connection refused ***
Pretty yucky though.
I'm developing on Windows and found myself in the same predicament. But the error message always contains the error number. Using that information I just convert the exception to a string str(Exception), convert the error code I wanna check for to a string str(socket.errno.ERRORX) and check if the error code is in the exception.
Example for a connection reset exception:
except Exception as errorMessage:
if str(socket.errno.ECONNRESET) in str(errorMessage):
print("Connection reset")
#etc...
This avoids locale specific solutions but is still not platform independent unfortunately.