I'm trying to find a design pattern - as I'm sure one exists, this problem is common.
In my application if the user loses their Internet connection, I want to be able to pause the application, allowing the user to check their connection and retry. When a connection is successful the application leaves where it left off.
I've attempted this like so:
while True:
try:
for url in urls:
downloadPage(url)
except ConnectionException:
raw_input('Connection lost. Press enter to try again')
continue
But this doesn't work, because if the exception is raised within the for loop, it will catch it, but when it continues it will restart from the beginning of the urls list.
I do need to check for connection errors both before the application starts running, and during each request. That way I can pause it. But I don't want to litter all my code with try/catch blocks.
Is there a pattern for this?
Why not this?
while True:
for url in urls:
success = False
while (not success):
try:
downloadPage(url)
success = True
except ConnectionException:
raw_input('Connection lost. Press enter to try again')
You can move the try within the for loop
for url in urls:
while True:
try:
downloadPage(url)
except ConnectionException:
raw_input('Connection lost. Press enter to try again')
This will attempt to connect maximum of 3 times before dropping the current url and moving on to the next one. So you are not stuck if connection can't be established, but still gave a fair chance to every url.
for url in urls:
retries = 3
while True:
try:
downloadPage(url)
except ConnectionException:
retries -= 1
if retries == 0:
print "Connection can't be established for url: {0}".format(url)
break
raw_input('Connection lost. Press enter to try again')
You can "abstract away" the cruft in just one place (to not have to "litter all my code with try/catch blocks" as you put it) -- that's what context managers are for! A simplistic example...:
import contextlib
#contextlib.contextmanager
def retry_forever(exception=ConnectionException, message='Connection lost. Press enter to try again'):
while True:
try: yield
except exception:
raw_input(message)
else: break
Now, you can just use
for url in urls:
with retry_forever():
downloadPage(url)
The better variants (with max # of retries, &c) can just as elegantly be refactored into this very useful form.
You can use retrying package
Just write a block of code that keeps repeating on failure, until max retries is hit
Related
The server that runs my Python code which get some data off of a website has to jump through so many hoops just to get to the DNS server that the latency gets to the point where it times out the DNS resolution (this is out of my control). Sometimes it works, sometimes it doesn't.
So I am trying to do some exception handling and try to ensure that it works.
Goal:
Increase the DNS timeout. I am unsure of a good time but let's go with 30 seconds.
Try to resolve the website 5 times, if it resolves, proceed to scrape the website. If it doesn't, keep trying until the 5 attempts are up.
Here is the code using google.com as an example.
import socket
import http.client
#Confirm that DNS resolution is successful.
def dns_lookup(host):
try:
socket.getaddrinfo(host, 80)
except socket.gaierror:
return "DNS resolution to the host failed."
return True
#Scrape the targeted website.
def request_website_data():
conn = http.client.HTTPConnection("google.com")
conn.request("GET", "/")
res = conn.getresponse()
if (res.status == 200):
print("Connection to the website worked! Do some more stuff...")
else:
print("Connection to the website did not work. Terminating.")
#Attempt DNS resolution 5 times, if it succeeds then immediately request the website and break the loop.
for x in range(5):
dns_resolution = dns_lookup('google.com')
if dns_resolution == True:
request_website_data()
break
else:
print(dns_resolution)
I am looking through the socket library socket.settimeout(value) and I am unsure if that's what I'm looking for. What would I insert into my code to have a more forgiving and longer DNS resolution time?
Thank you.
Currently I have a program that has proxies and makes a request to get my ip address with that proxy and returns it back in json.
An example request back is this:
Got Back: {'ip': '91.67.240.45', 'country': 'Germany', 'cc': 'DE'}
I want my program to try and make a request to the url and if it does not get the request because the proxy is down I want to try again 5 times before moving onto the next ip address.
I thought this except block would work but it is not breaking out of the loop when the 5 iterations are over and I am not sure why.
My program does however work when the proxy is up for the first try as it breaks after the first attempt and then moves onto the next ip address.
Here is what I currently have:
import requests
import time
proxies = [
"95.87.220.19:15600",
"91.67.240.45:3128",
"85.175.216.32:53281",
"91.236.251.131:8118",
"91.236.251.131:8118",
"88.99.10.249:1080",
]
def sol(ip):
max_tries = 5
for i in range(1, max_tries+1):
try:
print(f"Using Proxy: {ip}")
r = requests.get('https://api.myip.com', proxies={"https": ip})
print(f"Got Back: {r.json()}")
break
except OSError:
time.sleep(5)
print(f"Retrying...: {i}")
break
for i in proxies:
sol(i)
How can I make it s my loop has 5 tries before moving onto the next ip address.
My program does however work when the proxy is up for the first try as it breaks after the first attempt and then moves onto the next ip address.
It does this unconditionally, because you have an unconditional break after the except block. Code keeps going past a try/except when the except is entered, assuming it doesn't have an abnormal exit of its own (another exception, return etc.).
So,
I thought this except block would work but it is not breaking out of the loop when the 5 iterations are over and I am not sure why.
It doesn't break out "after the 5 iterations are over" because it breaks out after the first iteration, whether or not that attempt was successful.
If I understand correctly, you can just remove break from the last line. If you have an unconditional break in a loop, it will always iterate once.
def sol(ip):
max_tries = 5
for i in range(1, max_tries+1):
try:
print(f"Using Proxy: {ip}")
r = requests.get('https://api.myip.com', proxies={"https": ip})
print(f"Got Back: {r.json()}")
break
except OSError:
time.sleep(5)
print(f"Retrying...: {i}")
# break <---- Remove this line
Using retrying would look something like the following:
from retrying import retry
import requests
#retry(stop_max_attempt_number=5)
def bad_host():
print('trying get from bad host')
return requests.get('https://bad.host.asdqwerqweo79ooo/')
try:
bad_host()
except IOError as ex:
print(f"Couldn't connect because: {str(ex)}")
...which give the following output:
trying get from bad host
trying get from bad host
trying get from bad host
trying get from bad host
trying get from bad host
Couldn't connect to bad host because: HTTPSConnectionPool(host='bad.host.asdqwerqweo79ooo', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10b6bd910>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
Getting fancy
If you want to get fancy, you could also add things like exponential backoff and selectively retrying certain exceptions.
Here's an example:
import random
import time
def retry_ioerror(exception):
return isinstance(exception, IOError)
#retry(
wait_exponential_multiplier=100,
wait_exponential_max=1000,
retry_on_exception=retry_ioerror,
stop_max_attempt_number=10)
def do_something():
t = time.time()
print(f'trying {t}')
r = random.random()
if r > 0.9:
return 'yay!'
if r > 0.8:
raise RuntimeError('Boom!')
else:
raise IOError('Bang!')
try:
result = do_something()
print(f'Success! {result}')
except RuntimeError as ex:
print(f"Failed: {str(ex)}")
With Kubernetes Container running a Python script:
import time
while True:
try:
for i in range(10):
if i==0:
raise Exception('Exception occurred!')
except:
pass
time.sleep(1)
I would like to pass the Exception's message 'Exception occurred!' down to the Container so this error message could be seen with:
kubectl describe pod pod_id
Would it be possible?
Anything you print() will be visible in kubectl logs. (You may need to set an environment variable PYTHONUNBUFFERED=1 in your pod spec.)
Your code as you've written it will never print anything. The construct
try:
...
except:
pass
silently ignores any and all exceptions out of the try block. The bare except: even captures some system-level exceptions like SystemExit or KeyboardInterrupt; this is almost always wrong. Often you want your except blocks to be as tightly scoped as you can, and the Python tutorial on user-defined exceptions is a helpful pattern.
(The exception to this, particularly in a Kubernetes context, is that you will often want a very broad exception handler to do something like return an HTTP 500 error to a network request, rather than crashing the application.)
A better example might look like:
import time
class OneException(Exception):
pass
def iteration():
for i in range(10):
try:
if i == 1:
raise OneException("it is one")
print(i, math.sqrt(i), math.sqrt(-i))
# will work when i==0 but fail when i==2
except OneException as e:
print(i, repr(e))
# and proceed to the next iteration
if __name__ == '__main__':
while True:
# The top-level loop. We want a very broad catch here.
try:
iteration()
except Exception as e:
print('iteration failed', repr(e))
time.sleep(1)
I'm creating a game in the Blender Game Engine. And I have coded an IRC script which works fine on OS X and Linux distros. The output is similar to this:
Logging in...
LOGIN_ERROR
LOGIN_ERROR
LOGIN_ERROR
LOGIN_ERROR
LOGIN_ERROR
LOGIN_ERROR
<name> has joined.
Logged in!
And then I can call my sendmsg() function to send messages to the IRC channel.
This is the error I get when I try to run on Windows 7:
My python IRC code:
http://pastebin.com/aG6TwTir
Ignore the "bge" references. Those variables and such are filled from the game engine.
In the game engine, I call login() once, and it spits out "LOGIN_ERROR" so I know it's trying to connect, and then it will connect, therefore not throwing an exception and ending the function.
In OS X and Linux, it runs perfectly and seemlessly in the background while the player can continue to play as it connects.
In windows 7, it throws that error.
So I guess what needs to happen is a way to wait for the script to connect to the server. Then once connected, I can send the login information and join the channel.
So how do I wait for the connection?
FYI: I have the sockets non-blocking, since the script needs to run on the same thread as the game engine, on every frame.
Main() is run every frame, not the whole script. At the menu, it executes the script and calls login(). Then once in the game, it will call Main() every frame.
Oh and I'm using Python 3.3.
Any help is greatly apreciated! ^_^
EDIT:
How do I handle this exception?
This code:
def login():
...
try:
...
except:
...
login() # <===
recursively calls itself; given a high enough number of login failures, depending on the stack size limit (which depends on platform I guess), you'll get a stack overflow.
See also: Setting stacksize in a python script
Although I would always just avoid recursion and use looping instead, unless I know in advance that the recursion depth will never be more than ~100:
while True:
try:
do_login()
except: # NOTE: USE A SPECIFIC EXCEPTION CLASS HERE, BTW
continue
else:
break
You have recursion happening in your error handling
def login():
#print('login')
# Bind the socket
try:
s.connect((HOST, PORT))
# Send login info
s.send(bytes('NICK %s\r\n' % NICK, 'UTF-8'))
s.send(bytes('USER %s %s bla :%s\r\n' % (IDENT, HOST, REALNAME), 'UTF-8'))
s.send(bytes('JOIN %s\r\n' % CHAN, 'UTF-8'));
print('Logging in...')
chatlog('Logging in...')
except:
print('LOGIN_ERROR')
login()
So in your function login() you have a try, then in the except you call login() again. This will just loop over and over again if the login fails.
I'd better use the following sample codes to explain my problem:
while True:
NewThread = threading.Thread(target = CheckSite, args = ("http://example.com", "http://demo.com"))
NewThread.start()
time.sleep(300)
def CheckSite(Url1, Url2):
try:
Response1 = urllib2.urlopen(Url1)
Response2 = urllib2.urlopen(Url2)
del Response1
del Response2
except Exception, reason:
print "How should I delete Response1 and Response2 when exception occurs?"
del Response1
del Response2 #### You can't simply write this as Reponse2 might not even exist if exception shows up running Response1
I've wrote a really looong script, and it's used to check different sites running status(response time or similar stuff), just like what I did in the previous codes, I use couple of threads to check different site separately. As you can see in each thread there would be several server requests and of course you will get 403 or similar every now and then. I always think those wasted connections(ones with exceptions) would be collected by some kind of garbage collector in python, so I just leave them alone.
But when I check my network monitor, I found those wasted connections still there wasting resources. The longer the script running, the more wasted connections appears. I really don't want to do try-except clause each time sending server request so that del responsecan be used in each except part to destroy the wasted connection. There gotta be a better way to do this, anybody can help me out?
What exactly do you expect "delete" to mean in this context, anyway, and what are you hoping to accomplish?
Python has automatic garbage collection. These objects are defined, further, in such a way that the connection will be closed whenever the garbage collector gets around to collecting the corresponding objects.
If you want to ensure that connections are closed as soon as you no longer need the object, you can use the with construct. For example:
def CheckSite(Url1, Url2):
with urllib2.urlopen(Url1) as Response1:
with urllib2.urlopen(Url2) as Response2:
# do stuff
I'd also suggest to use the with statement in conjunction with the contextlib.closing function.
It should close the connection when it finishes the job or when it gets an exception.
Something like:
with contextlib.closing(urllib2.open(url)) as reponse:
pass
#del response #to assure the connection does not have references...
You shoud use Response1.close(). with doesn't work with urllib2.urlopen directly, but see the contextlib.closing example in the Python documentation.
Connections can stay open for hours if not properly closed, even if the process creating them exits, due the reliable packet delivery features of TCP.
You should not check for Exception rather you should catch URLError as noted in the Documentation.
If an exception isn't thrown, does the connection persist? Maybe what you're looking for is
try:
Response1 = urllib2.urlopen(Url1)
Response2 = urllib2.urlopen(Url2)
Response1.close()
Response2.close()
except URLError, reason:
print "How should I delete Response1 and Response2 when exception occurs?"
if Response2 is not None:
Response2.close()
elif Response1 is not None:
Response1.close()
But I don't understand why you're encapsulating both in a single try. I would do the following personally.
def CheckSites(Url1, Url2):
try:
Response1 = urllib2.urlopen(Url1)
except URLError, reason:
print "Response 1 failed"
return
try:
Response2 = urllib2.urlopen(Url2)
except URLError, reason:
print "Response 2 failed"
## close Response1
Response1.close()
## do something or don't based on 1 passing and 2 failing
return
print "Both responded"
## party time. rm -rf /
Note that this accomplishes the same thing because in your code, if Url1 fails, you'll never even try to open the Url2 connection.
** Side Note **
Threading is really not helping you here at all. You might as well just try them sequentially because only one thread is going to be running at a time.
http://dabeaz.blogspot.com/2009/08/inside-inside-python-gil-presentation.html
http://wiki.python.org/moin/GlobalInterpreterLock