What I'm tryng to do is to download images through Google Custom Search API using Python. It's my first time and maybe I'm making some banal error catching exceptions with Python.
It all works fine even when I get into some error which is not 10054 Connection Reset By Peer. The code is something like this, I've just taken out the useless part:
try:
urllib.request.urlretrieve(myUrl['link'],str(count)+'.jpg')
except URLError as e:
print(e.reason)
Sometimes it happens that connection is reset by peer and the console shows this error.
urllib.request.urlretrieve(myUrl['link'],str(count)+'.jpg')
File "C:\Python33\lib\urllib\request.py", line 210, in urlretrieve
block = fp.read(bs)
File "C:\Python33\lib\http\client.py", line 503, in read
return super(HTTPResponse, self).read(amt)
File "C:\Python33\lib\http\client.py", line 542, in readinto
n = self.fp.readinto(b)
File "C:\Python33\lib\socket.py", line 297, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
Press any key to continue . . .
Now I'm not really interested in make that URL work, but I just want my loop to keep going and not to stop. How can I catch that exception?
Edit:
I also noticed that sometimes the code correctly catches the error and print(e.reason) correctly outputs [WinError 10054] without stopping the loop. That's very strange.
If you don't know the exact problem, you can catch all exceptions as so:
try:
urllib.request.urlretrieve(myUrl['link'],str(count)+'.jpg')
except URLError as e:
print(e.reason)
except KeyboardInterrupt as ki:
raise ki
except:
print("Unknown Error")
Related
I am using urllib3.PoolManager to make http requests. And in some part of my code I use this code to make a request
resp = h.request(self.method, self.url, body=body, headers=headers, timeout=TIMEOUT, retries=retries)
and I get the error SSL: CERTIFICATE_VERIFY_FAILED. Below is the full stack trace.
File "/lib/python2.7/site-packages/urllib3/request.py", line 69, in request
**urlopen_kw)
File "/lib/python2.7/site-packages/urllib3/request.py", line 90, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/lib/python2.7/site-packages/urllib3/poolmanager.py", line 248, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/lib/python2.7/site-packages/urllib3/connectionpool.py", line 621, in urlopen
raise SSLError(e)
[SSL: CERTIFICATE_VERIFY_FAILED]
The error is expected. But the problem is I cannot catch the error in try except block.
I tried to use
except ssl.SSLError:
but that does not catch this error.
I also tried ssl.CertificateError but no results.
I can catch it by using the Exception class but I need to catch the specific errors and handle them differently. Can someone please find me a solution to this?
I found the solution. The exception class that was being raised is urllib3.exceptions.SSLError.
Late answer, but you can catch SSL Errors using requests.exceptions.SSLError
import requests, traceback
try:
r = requests.get('https://domain.tld')
except (requests.exceptions.SSLError):
print(traceback.format_exc())
To find the specific type of any exception, you can use type()
import requests
try:
r = requests.get('https://domain.tld')
except Exception as e:
print(type(e))
Output:
<class 'requests.exceptions.ConnectionError'>
Which leads us to:
import requests
try:
r = requests.get('https://domain.tld')
except requests.exceptions.ConnectionError as e:
print("Caught correctly")
Output:
Caught correctly
I'm making calls to an API and am receiving the following timeout error:
socket.timeout The read operation timed out
which traces back from
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/http/client.py", line 1198, in getresponse
response.begin()
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/socket.py", line 576, in readinto
return self._sock.recv_into(b)
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/ssl.py", line 937, in recv_into
return self.read(nbytes, buffer)
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/ssl.py", line 799, in read
return self._sslobj.read(len, buffer)
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/ssl.py", line 583, in read
v = self._sslobj.read(len, buffer)
socket.timeout The read operation timed out
How can I catch this error from the top of the traceback?
This is a duplicate question, however the correct approach is to catch socket.timeout by using
import socket
try:
...
except socket.timeout:
...
The error states the exception that was thrown explicitly and you can rely on that. Even though the calling method is the ssl lib, the error seems to be related to the actual socket connection.
Google search led to me https://community.home-assistant.io/t/difficulty-catching-an-exception/12955/6
It seems like SSL throws ssl.SSLError when it times out. You could try something like:
import ssl
def getresponse():
...
try:
response.begin
except ssl.SSLError as err:
handle_error(err)
I'm unable to solve the problem I get with this project I’m doing at my university.
I’m using »token-stream.py« from https://github.com/kappaloris/markrov-text-generator
I constantly get the following error:
File "/Users/daniel/Desktop/markrov-text-generator-master/token-stream.py", line 102, in <module>
main()
File "/Users/daniel/Desktop/markrov-text-generator-master/token-stream.py", line 97, in main
streamer.sample()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 398, in sample
self._start(async)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 340, in _start
self._run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 280, in _run
raise exception
requests.exceptions.ConnectionError: ('Connection aborted.', error(22, 'Invalid argument'))
Unfortunately I'm far from being even a beginner in programming, so it would be great if you could solve this problem »for dummies«.
The exception happens in tweepy and the linked code contains this part:
## API KEY SETTINGS
auth1 = tweepy.auth.OAuthHandler('YOUR KEY','YOUR KEY')
auth1.set_access_token('YOUR KEY','YOUR KEY')
############
This authentication information is used in the to set up the failing connection.
It seems like you have to replace these 'YOUR KEY' strings with your authentication information. You should have a key and a secret, and those need to be specified in both calls.
Probably you didn't do that and therefore you can't connect.
I am trying to write a python script to establish a telnet connection (using telnetlib) to a range of hosts:
for i in range(len(HOST)):
print "scanning " + HOST[i] + " ...\n"
tn = telnetlib.Telnet(HOST[i],23,3)
The problem is when one of the connections times out, the script executions interrupts and returns the following error:
Traceback (most recent call last):
File "C:\Python27\telnet.py", line 24, in <module>
tn = telnetlib.Telnet(HOST[i],23,3)
File "C:\Python27\lib\telnetlib.py", line 209, in __init__
self.open(host, port, timeout)
File "C:\Python27\lib\telnetlib.py", line 225, in open
self.sock = socket.create_connection((host, port), timeout)
File "C:\Python27\lib\socket.py", line 571, in create_connection
raise err
socket.timeout: timed out
Anyone knows how to skip this error and continue the script?
You need to use a try...except block to catch the exception and tell the interpreter to ignore it. For example:
import socket
for i in range(len(HOST)):
print "scanning " + HOST[i] + " ...\n"
try:
tn = telnetlib.Telnet(HOST[i],23,3)
except socket.timeout:
pass
In this case it's a good idea to explicitly state which exception you want to catch (socket.timeout). Sockets can throw many different types of exceptions so using a generic except: statement might mask a problem with opening, reading or writing to the socket.
I have a Django management command, launched via supervisord, that uses tweepy to consume the twitter streaming API.
The agent works quite well however I notice in the logs there's an SSLError every 10-15 minutes and supervisord is re-launching the agent.
The tweepy package is latest, version 1.11. The server is ubuntu 12.04 LTS. I've tried installing the cacert into the key chain as mentioned in the link below, but no luck.
Twitter API SSL Root CA Certificate
Any suggestions?
[2012-08-26 19:28:15,656: ERROR] Error establishing the connection
Traceback (most recent call last):.../.../datasinks.py", line 102, in start
stream.filter(locations=self.locations)
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 228, in filter
self._start(async)
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 172, in _start
self._run()
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 117, in _run
self._read_loop(resp)
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 150, in _read_loop
c = resp.read(1)
File "/usr/lib/python2.7/httplib.py", line 541, in read
return self._read_chunked(amt)
File "/usr/lib/python2.7/httplib.py", line 574, in _read_chunked
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
SSLError: The read operation timed out
Following is an outline of the code.
from tweepy import API, OAuthHandler
from tweepy.streaming import StreamListener, Stream
# snip other imports
class TwitterSink(StreamListener, TweetSink):
def __init__(self):
self.auth = OAuthHandler(settings.TWITTER_OAUTH_CONSUMER_KEY, settings.TWITTER_OAUTH_CONSUMER_SECRET)
self.auth.set_access_token(settings.TWITTER_OAUTH_ACCESS_TOKEN_KEY, settings.TWITTER_OAUTH_ACCESS_TOKEN_SECRET)
self.locations = '' # Snip for brevity
def start(self):
try:
stream = Stream(self.auth, self,timeout=60, secure=True)
stream.filter(locations=self.locations)
except SSLError as e:
logger.exception("Error establishing the connection")
except IncompleteRead as r:
logger.exception("Error with HTTP connection")
# snip on_data()
# snip on_timeout()
# snip on_error()
The certificate doesn't seem to be the problem. The error is just a timeout. Seems like an issue with tweepy's SSL handling to me. The code is equipped to handle socket.timeout and reopen the connection, but not a timeout arriving through SSLError.
Looking at the ssl module code (or docs), though, I don't see a pretty way to catch that. The SSLError object is raised without any arguments, just a string description. For lack of a better solution, I'd suggest adding the following right before line 118 of tweepy/streaming.py:
except SSLError, e:
if 'timeout' not in exception.message.lower(): # support all timeouts
exception = e
break
if self.listener.on_timeout() == False:
break
if self.running is False:
break
conn.close()
sleep(self.snooze_time)
Why it's timing out in the first place is a good question. I have nothing better than repeating Travis Mehlinger's suggestion of setting a higher timeout.
Here is how I have it (modified solution from here https://groups.google.com/forum/?fromgroups=#!topic/tweepy/80Ayu1joGJ4):
l = MyListener()
auth = OAuthHandler(settings.CONSUMER_KEY, settings.CONSUMER_SECRET)
auth.set_access_token(settings.ACCESS_TOKEN, settings.ACCESS_TOKEN_SECRET)
# connect to stream
stream = Stream(auth, l, timeout=30.0)
while True:
# Call tweepy's userstream method with async=False to prevent
# creation of another thread.
try:
stream.filter(follow=reporters, async=False)
# Normal exit: end the thread
break
except Exception, e:
# Abnormal exit: Reconnect
logger.error(e)
nsecs = random.randint(60, 63)
logger.error('{0}: reconnect in {1} seconds.'.format(
datetime.datetime.utcnow(), nsecs))
time.sleep(nsecs)
There is another alternative solution provided on Github:
https://github.com/tweepy/tweepy/pull/132