I'm running a small python script to check every 10 seconds to see if my internet access is available (been having problems with my isp sucking). I've been running it for probably 2 months and it's worked perfectly, but now it randomly exits. Sometimes it exits within 20 seconds of me starting it, and sometimes it waits 5 minutes. the code is:
import time
import datetime
import urllib2
waitTime = 300
outfile = "C:\Users\simmons\Desktop\internetConnectivity.txt"
def internetOffline():
with open (outfile, "a") as file:
file.write("Internet Offline: %s\n" % time.ctime())
print("Internet Went Down!")
def internetCheck():
try:
urllib2.urlopen('https://www.google.com', timeout = 2)
except urllib2.URLError:
internetOffline()
while (1):
internetCheck()
time.sleep( 10 )
My question is not only, how would I print out what is happening when it exits, but also, does anyone know of a more efficient way of doing this, so it possibly causes less network traffic. It's not a problem now, but I was just wondering about more efficient methods.
this could be from going to google to many times im not to sure
run the program in you're IDE and then read the error it throws on exit this should tell you what or where the program is exiting
Here is a good way to do this:
import urllib2
def internet_on():
try:
response=urllib2.urlopen('http://74.125.228.100',timeout=1)
return True
except urllib2.URLError as err: pass
return False
74.125.228.100 is one of the IP-addresses for google.com. Change http://74.125.228.100 to whatever site can be expected to respond quickly
I got this solution from this question, take a look at it, it should help
Related
I am trying to find a way in which I can re run my code if there is a timeout or an error.
This is something that happens if the internet connection drops - So I'd need to delay for a few seconds whilst it comes back online and try again..
Would there be a way in which I could run my code in another python script and tell it to re run if it times out or disconnects?
Thanks in advance
If you are talking about http request connection error and if you are using the requests library you can use this urllib retry
FYI https://docs.python-requests.org/en/master/api/#requests.adapters.HTTPAdapter
Of course, other libraries will have their own retry function.
If you just want simple retry code, use the below code
retries_count = 3 # hard value
delay = 3 # hard value
while True:
try:
... run some code
return or break
except {Your Custom Error}:
if retries_count <= 0:
raise
retries_count -= 1
time.sleep(delay)
A Google search for "python throttling" will give you a lot of reference.
You can use a try-catch in an infinite loop like this:
while True:
try:
# your code
# call another python script
except:
time.sleep(10)
also, you can check if the error and run whatever you need to run, depend on the error type. for example:
while True:
try:
# your code
# call another python script
# break
except Exception as e:
if e == 'error type':
time.sleep(10)
else:
pass
You actually can do it with try except block.
You can find all about it here: https://docs.python.org/3/tutorial/errors.html
You just make one script, that has something like that:
while True:
try:
another_script_name.main()
except:
time.sleep(20)
Of course, you need to import both time and the other script you made.
What these lines are doing is just an infinite loop, that always tries to run the main function on the other script you made, and if some sort of error occurs, the system will sleep for 20 seconds and then will try again, because it is in the infinite loop.
I want to know if anyone knows how to open 2 different URLs at the same time using urllib.request.urlopen? I can only open one URL at a time so far using try and except:
try:
preflash = urllib.request.urlopen("http://192.168.100.6", timeout=5).getcode()
print("Web page status code:", preflash, "FAIL")
sys.exit(0)
except urllib.error.URLError:
correct = urllib.request.urlopen("http://192.168.100.5", timeout=10).getcode()
print("Web page status code:", correct)
print("IP address: 192.168.100.5 is reachable")
How can i open both URLs at the same time and when 192.168.100.6 is reachable then the system just exits as you can clearly see in my code. But if 192.168.100.5 is reachable then the programm continues to function.
How can I open both 192.168.100.6 & 192.168.100.5 at the same time and then using urllib or some other method and print "Fail" if 192.168.100.6 is reachable and print "Pass" if 192.168.100.5 is reachable.
I have read into threading but i really do not know how to implement it so please help me guys, show me how this could be done I really have no clue and i am so confused after reading about threading multi threading aiohttp aiosync I am just so confused.
If someone could show me how it is done I could use that as a base to understand what is going on.
I am just a beginner so please take it easy on me all.
I have a scraping code within a for loop, but it would take several hours to complete, and the program stops when my Internet connection breaks. What I (think I) need is a condition at the beginning of the scraper that tells Python to keep trying at that point.
I tried to use the answer from here:
for w in wordlist:
#some text processing, works fine, returns 'textresult'
if textresult == '___': #if there's nothing in the offline resources
bufferlist = list()
str1=str()
mlist=list() # I use these in scraping
br = mechanize.Browser()
tried=0
while True:
try:
br.open("http://the_site_to_scrape/")
# scraping, with several ifs. Each 'for w' iteration results with scrape_result string.
except (mechanize.HTTPError, mechanize.URLError) as e:
tried += 1
if isinstance(e,mechanize.HTTPError):
print e.code
else:
print e.reason.args
if tried > 4:
exit()
time.sleep(120)
continue
break
Works while I'm online. When the connection breaks, Python writes the 403 code and skips that word from wordlist, moves on to the next and does the same. How can I tell Python to wait for connection within the iteration?
EDIT: I would appreciate it if you could write at least some of the necessary commands and tell me where they should be placed in my code, because I've never dealt with exception loops.
EDIT - SOLUTION I applied Abhishek Jebaraj's modified solution. I just added a very simple exception handling command:
except:
print "connection interrupted"
time.sleep(30)
Also, Jebaraj's getcode command will raise an error. Before r.getcode, I used this:
import urllib
r = urllib.urlopen("http: the site ")
The top answer to this question helped me as well.
Write another while loop inside which will keep trying to connect to the internet.
It will break only when it receives status code of 200 and then you can continue with your program.
Kind of like
retry = True
while retry:
try:
r = br.open(//your site)
if r.getcode()/10==20:
retry = False
except:
// code to handle any exception
// rest of your code
Newbie question here,
I am trying to hack an existing python program that searches for bluethooth signal
all works fine if there is a bluetooh transmitter around,
but the program simply sits there if there is no Bluetooth signal around.
I found out that it is getting hung on this line
pkt = sock.recv(255)
I am naively guessing that is simply sitting there waiting for data, I want it to give me an error or timeout after lets say 10 seconds.
How do I do this? Is my thinking correct?
Thanks
Call settimeout before recving. Then it will raise an error if it takes too long.
sock.settimeout(10)
try:
pkt = sock.recv(255)
except socket.error:
print "connection timed out!"
return
I wrote a scrapy crawler that uses an Internet connection that is pretty unreliable. This is something that is a given. I cannot easily or cheaply change it - once in a while the Internet connection will be lost and after a few seconds or so it will be restored.
I observe behaviour where a Scrapy 18.4 crawler would freeze indefinitely without printing any error messages. It stops reacting to Ctrl+C, which makes me think this happens somewhere pretty deep in the reactor stack, though I cannot be sure.
There are absolutely no error messages which makes things rather hopeless to debug.
Question: Would anyone have any clues as to how to debug this problem? I don't really have any meaningful logs to attach for the reasons laid out above.
You can set up a class for timeout and then run your code in a try except. Something like this:
import signal
class timeout:
def __init__(self, seconds=1, error_message='Timeout'):
self.seconds = seconds
self.error_message = error_message
def handle_timeout(self, signum, frame):
raise TimeoutError(self.error_message)
def __enter__(self):
signal.signal(signal.SIGALRM, self.handle_timeout)
signal.alarm(self.seconds)
def __exit__(self, type, value, traceback):
signal.alarm(0)
def run_crawl():
while True:
print("This runs")
try:
with timeout(seconds=3):
run_crawl()
except Exception as e:
print(e)
Note: Since this uses signal it will only work on UNIX-based systems.
If you want it to go back to running the crawler (auto-restart), then you can just put it all in an infinite loop.
while True:
print("Restarting spider")
try:
with timeout(seconds=3):
run_crawl()
except Exception as e:
print(e)
This is all assuming that you can just keep restarting the bot after x seconds without major negative results. IE if you are constantly scraping the same page(s) over and over again, then this would work pretty seamlessly.
However, if you are scraping a very long list [once] and just want it to finish without an error, then it would work less well, but could still be used by setting x to a number that represents an amount of time greater than the duration of the entire process when it executes successfully (this is true no matter the length of the execution of the program - don't set x to 3 seconds if you are scraping one site that takes 7 seconds to complete, and the same is true if you are doing 5 that take 30 seconds or 500 that take 5 minutes, you will need to set x to an amount greater than that duration.
The reason that I specifically separated your bot having a quick completion time vs. not is that if it fails during a loop with a timeout of 30 seconds then you lose, on average, 15 seconds if it fails, but if you have a 30 minute execution time then you would lose, on average, 15 minutes when it fails, and if your internet is going out every 15 minutes, on average, then you would have it failing a vast majority of the time and you'll need to look more into debugging the problem and actually solving it instead of working around it, as this "solution" should definitely be considered a workaround.