Currently I have a program that has proxies and makes a request to get my ip address with that proxy and returns it back in json.
An example request back is this:
Got Back: {'ip': '91.67.240.45', 'country': 'Germany', 'cc': 'DE'}
I want my program to try and make a request to the url and if it does not get the request because the proxy is down I want to try again 5 times before moving onto the next ip address.
I thought this except block would work but it is not breaking out of the loop when the 5 iterations are over and I am not sure why.
My program does however work when the proxy is up for the first try as it breaks after the first attempt and then moves onto the next ip address.
Here is what I currently have:
import requests
import time
proxies = [
"95.87.220.19:15600",
"91.67.240.45:3128",
"85.175.216.32:53281",
"91.236.251.131:8118",
"91.236.251.131:8118",
"88.99.10.249:1080",
]
def sol(ip):
max_tries = 5
for i in range(1, max_tries+1):
try:
print(f"Using Proxy: {ip}")
r = requests.get('https://api.myip.com', proxies={"https": ip})
print(f"Got Back: {r.json()}")
break
except OSError:
time.sleep(5)
print(f"Retrying...: {i}")
break
for i in proxies:
sol(i)
How can I make it s my loop has 5 tries before moving onto the next ip address.
My program does however work when the proxy is up for the first try as it breaks after the first attempt and then moves onto the next ip address.
It does this unconditionally, because you have an unconditional break after the except block. Code keeps going past a try/except when the except is entered, assuming it doesn't have an abnormal exit of its own (another exception, return etc.).
So,
I thought this except block would work but it is not breaking out of the loop when the 5 iterations are over and I am not sure why.
It doesn't break out "after the 5 iterations are over" because it breaks out after the first iteration, whether or not that attempt was successful.
If I understand correctly, you can just remove break from the last line. If you have an unconditional break in a loop, it will always iterate once.
def sol(ip):
max_tries = 5
for i in range(1, max_tries+1):
try:
print(f"Using Proxy: {ip}")
r = requests.get('https://api.myip.com', proxies={"https": ip})
print(f"Got Back: {r.json()}")
break
except OSError:
time.sleep(5)
print(f"Retrying...: {i}")
# break <---- Remove this line
Using retrying would look something like the following:
from retrying import retry
import requests
#retry(stop_max_attempt_number=5)
def bad_host():
print('trying get from bad host')
return requests.get('https://bad.host.asdqwerqweo79ooo/')
try:
bad_host()
except IOError as ex:
print(f"Couldn't connect because: {str(ex)}")
...which give the following output:
trying get from bad host
trying get from bad host
trying get from bad host
trying get from bad host
trying get from bad host
Couldn't connect to bad host because: HTTPSConnectionPool(host='bad.host.asdqwerqweo79ooo', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10b6bd910>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
Getting fancy
If you want to get fancy, you could also add things like exponential backoff and selectively retrying certain exceptions.
Here's an example:
import random
import time
def retry_ioerror(exception):
return isinstance(exception, IOError)
#retry(
wait_exponential_multiplier=100,
wait_exponential_max=1000,
retry_on_exception=retry_ioerror,
stop_max_attempt_number=10)
def do_something():
t = time.time()
print(f'trying {t}')
r = random.random()
if r > 0.9:
return 'yay!'
if r > 0.8:
raise RuntimeError('Boom!')
else:
raise IOError('Bang!')
try:
result = do_something()
print(f'Success! {result}')
except RuntimeError as ex:
print(f"Failed: {str(ex)}")
Related
The server that runs my Python code which get some data off of a website has to jump through so many hoops just to get to the DNS server that the latency gets to the point where it times out the DNS resolution (this is out of my control). Sometimes it works, sometimes it doesn't.
So I am trying to do some exception handling and try to ensure that it works.
Goal:
Increase the DNS timeout. I am unsure of a good time but let's go with 30 seconds.
Try to resolve the website 5 times, if it resolves, proceed to scrape the website. If it doesn't, keep trying until the 5 attempts are up.
Here is the code using google.com as an example.
import socket
import http.client
#Confirm that DNS resolution is successful.
def dns_lookup(host):
try:
socket.getaddrinfo(host, 80)
except socket.gaierror:
return "DNS resolution to the host failed."
return True
#Scrape the targeted website.
def request_website_data():
conn = http.client.HTTPConnection("google.com")
conn.request("GET", "/")
res = conn.getresponse()
if (res.status == 200):
print("Connection to the website worked! Do some more stuff...")
else:
print("Connection to the website did not work. Terminating.")
#Attempt DNS resolution 5 times, if it succeeds then immediately request the website and break the loop.
for x in range(5):
dns_resolution = dns_lookup('google.com')
if dns_resolution == True:
request_website_data()
break
else:
print(dns_resolution)
I am looking through the socket library socket.settimeout(value) and I am unsure if that's what I'm looking for. What would I insert into my code to have a more forgiving and longer DNS resolution time?
Thank you.
I want to build a Python program to check the NSlookup result.
In my program, I want to output like:-
If I input google then it will show “not unique” as output, but
when I provide input like Rajat then the output will be unique because rajat.com is not a valid site (invalid URL no IP is linked with this URL)
Below is my code.in this code will show not unique when I input google but throw an error when I input Rajat.
So I just want to know how to handle this error or how we can get the output as unique when the program throws an error.
import socket
dns=input("Enter DNS: ")
result= socket.gethostbyname(dns+".com")
if not result:
print("Unique")
else:
print("Not Unique")
The gethostbyname() function raises the socket.gaierror exception if the given hostname is not found, so you have an opportunity to catch it and perform what you want:
import socket
dns=input("Enter DNS: ")
try:
result= socket.gethostbyname(dns + ".com")
except socket.gaierror:
print("Unique")
else:
print("Not Unique"))
Note:
In my humble opinion the better messages would be something as “Not used” / “Used” or
“Not found” / “Found”. Yours are a little confusing.
You can use try-except to catch an error. In this code snippet we try to connect to the given domain and if the connection is successful (the website exists) it prints Not unique, if not, it prints Unique.
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
dns = input("Enter DNS: ")
try:
s.connect((dns + ".com", 80))
except Exception as e:
print("Unique")
else:
print("Not unique")
I don't have enough reputation yet to comment on another post so I'll ask it here. I am getting the "ConnectionError 10054 An existing connection was forcibly closed by the remote host" while doing a 'requests.get'. Now, I read this code on the other post but not sure if it is complete; can you please edit it so that it keeps retrying my 'requests.get' when the error comes up then exit of course when my 'requests.get' is successful. Thank you
import socket
retry_count = 5 # this is configured somewhere
for retries in range(retry_count):
try:
data1 = requests.get(url)
return True
except (error_reply, error_perm, error_temp):
return False
except socket.gaierror, e:
if e.errno != 10054:
return False
reconnect()
return False
Getting error;
return True
^
SyntaxError: 'return' outside function
The reason why you are having an error is simple. You haven't made a function, so you can't call return. The simplest way to fix this is using a break which allows you to jump out of the loop. Here's an example of using break in your code.
import socket
import requests
retry_count = 5
for retries in range(retry_count):
try:
data1 = requests.get(url)
#Jumps Out Of Loop
break
except (socket.gaierror, requests.ConnectionError) as e:
if e.errno != 10054:
continue
reconnect()
#Does Something If Loop Never Breaks
else:
print("Couldn't Connect")
I'm trying to make some kind of a scanner with python (just for fun)
it will send a get request to an random ip and see if there is any answer
the problem is that every time that the connection fails the program will stop running .
this is the code
import time
import requests
ips = open("ip.txt", "r")
for ip in ips:
r = requests.get(url="http://"+ip+"/",verify=False)
print(r.status_code)
time.sleep(0.5)
this is what i get by trying just a random ip :
requests.exceptions.ConnectionError: HTTPConnectionPool(host='177.0.0.0', port=80): Max retries exceeded with url:
This is throwing an error. To protect against this, use a try/except statement:
for ip in ips:
try:
r = requests.get(url="http://"+ip+"/",verify=False)
print(r.status_code)
except requests.exceptions.RequestException as e:
print('Connecting to ip ' + ip + ' failed.', e)
time.sleep(0.5)
I'm trying to find a design pattern - as I'm sure one exists, this problem is common.
In my application if the user loses their Internet connection, I want to be able to pause the application, allowing the user to check their connection and retry. When a connection is successful the application leaves where it left off.
I've attempted this like so:
while True:
try:
for url in urls:
downloadPage(url)
except ConnectionException:
raw_input('Connection lost. Press enter to try again')
continue
But this doesn't work, because if the exception is raised within the for loop, it will catch it, but when it continues it will restart from the beginning of the urls list.
I do need to check for connection errors both before the application starts running, and during each request. That way I can pause it. But I don't want to litter all my code with try/catch blocks.
Is there a pattern for this?
Why not this?
while True:
for url in urls:
success = False
while (not success):
try:
downloadPage(url)
success = True
except ConnectionException:
raw_input('Connection lost. Press enter to try again')
You can move the try within the for loop
for url in urls:
while True:
try:
downloadPage(url)
except ConnectionException:
raw_input('Connection lost. Press enter to try again')
This will attempt to connect maximum of 3 times before dropping the current url and moving on to the next one. So you are not stuck if connection can't be established, but still gave a fair chance to every url.
for url in urls:
retries = 3
while True:
try:
downloadPage(url)
except ConnectionException:
retries -= 1
if retries == 0:
print "Connection can't be established for url: {0}".format(url)
break
raw_input('Connection lost. Press enter to try again')
You can "abstract away" the cruft in just one place (to not have to "litter all my code with try/catch blocks" as you put it) -- that's what context managers are for! A simplistic example...:
import contextlib
#contextlib.contextmanager
def retry_forever(exception=ConnectionException, message='Connection lost. Press enter to try again'):
while True:
try: yield
except exception:
raw_input(message)
else: break
Now, you can just use
for url in urls:
with retry_forever():
downloadPage(url)
The better variants (with max # of retries, &c) can just as elegantly be refactored into this very useful form.
You can use retrying package
Just write a block of code that keeps repeating on failure, until max retries is hit