Prevent exit on exception in Python - python

I am trying to catch an exception thrown by Selenium. When the exception is thrown I would like to restart the scraping process.
try:
startScraping(rootPage)
except (InvalidSessionIdException):
startScraping(rootPage)
finally:
driver.close()
Above is the code I am using.
The problem I am facing is that when an InvalidSessionIdException occurs the script still stops execution and shows a stacktrace.

the second startScraping(rootPage) (in the except block) is not protected by any try/except block...
If an exception occurs, retrying immediately is probably bound to fail ... again. I would
catch the exception
print a meaningful warning
wait a while
repeat until it works, or a given number of times with a for loop
like this
import time
nb_retries = 10
for nb_attempts in range(nb_retries):
try:
startScraping(rootPage)
break # success, exit the loop
except InvalidSessionIdException as e:
print("Warning: {}, attempt {}/{}".format(e,nb_attempts+1,nb_retries))
time.sleep(1)
else:
# loop went through after 10 attempts and no success
print("unable to scrape after {} retries".format(nb_retries))
driver.close()

Try this if you want to restart the process and ignoring the exception:
while True:
try:
startScraping(rootPage)
break # after finishing the scraping process
except (InvalidSessionIdException):
pass # or print the excepion
driver.close()
as mentioned in the code, you can print the exception or do any other exception handling you may want.

Related

How to continue with the execution even if i catch an error [duplicate]

This question already has answers here:
Catch exception and continue try block in Python
(12 answers)
Closed 11 months ago.
I want to control the error just printing it but continuing with the rest, por example:
try:
list =('a',2,3,5,6,'b',8)
print(list[8])
print("continue execution")
except:
print("An exception occurred")
It just will print the error but not the continue execution, is possible to continue even after exception?
What is missing and you're looking for is the finally instruction.
Syntax:
try:
#search for errors
except:
#if errors found, run this code
finally:
#run this code no matter if errors found or not, after the except block
The finally block contains the code that will be run after the except block, no matter if errors were raised or not.
See if this rearrangement of your own code helps:
list =('a',2,3,5,6,'b',8)
try:
print(list[8])
except:
print("An exception occurred")
print("continue execution")
If it matters for you to "print continue" right after exception occurrence , then you can put whatever you want to do before exception in a "bool function" and then inside the "try" check if function succeed or not.it means "if error occurred continue ,else show "An exception occurred".
try:
assert doStuff() == False
print("continue execution")
except:
print("An exception occurred")

How to continue execution after raising assertion error in selenium python

I have written the code as below.
If the element is displayed then it has to log the error message at the end of the complete execution
If the element not displayed then report the message as
"Not displayed"
try:
if self.driver.find_element(By.XPATH,"").is_displayed():
raise AssertionError("Element should not be displayed")
except NoSuchElementException:
self.log.info("Element not displayed")
If element displayed then raising the assertion error where execution will stop if the element does not exist then continuing the execution.
How can I write the script for both the conditions?
If I understood correctly, this is just a case of using the proper syntax. You don't need a try / except here. Using what you put in the question:
if self.driver.find_element(By.XPATH,"").is_displayed():
raise AssertionError("Element should not be displayed")
else:
self.log.info("Element not displayed")
use else
try:
print("Hello")
except:
print("Something went wrong")
else:
print("Nothing went wrong")
finally:
print("This will be printed irrespective of the outcome")

How to print an error message which appeared while multiprocessing (not raise just print ) on using Pool().map()

I am downloading images using multiprocessing and a unique thing I noticed that you can not get Error Message except there is BaseException. For simple cases, you can run a loop and do the following:
for i in range(start, start+end):
url = df.iloc[i,ind]
try:
load_save_image_from_url(url,DIR,str(i),resize=resize,resize_shape=resize_shape)
count+=1
except (KeyboardInterrupt, SystemExit):
sys.exit("Forced exit prompted by User: Quitting....")
except Exception as e:
print(f"Error at index {i}: {e}\n")
pass
It works completely fine But when you use Multiprocessing you can not either use logging or print because just a log or print is described which is last one.
try:
pool = Pool(workers)
pool.map(partial(load_save_image_from_url,OUT_DIR=DIR,resize=resize,resize_shape=resize_shape),
lis_tups)
except (KeyboardInterrupt, SystemExit):
sys.exit("Forced exit prompted by User: Quitting....")
except ConnectionError:
logging.error(f"Connection Error for some URL")
pass
except Exception as e:
logging.error(f'Some Other Error most probably Image related')
pass
pool.close()
pool.join()
Using pool.get() can work but it has to be in a loop and that too at the end of program.
How can I print an error or log an error when there is an exception while being in multiprocessing?
You can catch the exception inside the load_image function itself and return it to the main process.
result = pool.map(load_image, ...)
if result instanceof Exception:
# handle it
else:
# image loading succeded
def load_image():
try:
# make a get request
return image
except Exception as e:
return e

How can I raise Exception using eventlet in Python?

I have a simply code:
import eventlet
def execute():
print("Start")
timeout = Timeout(3)
try:
print("First")
sleep(4)
print("Second")
except:
raise TimeoutException("Error")
finally:
timeout.cancel()
print("Third")
This code should throw TimeoutException, because code in 'try' block executing more than 3 seconds.
But this exception shallows in the loop. I can't see it in the output
This is output:
Start
First
Process finished with exit code 0
How can I raise this exception to the output?
Change sleep(4) to
eventlet.sleep(4)
This code will not output Start... because nobody calls execute(), also sleep is not defined. Show real code, I will edit answer.
For now, several speculations:
maybe you have from time import sleep, then it's a duplicate of Eventlet timeout not exiting and the problem is that you don't give Eventlet a chance to run and realize there was a timeout, solutions: eventlet.sleep() everywhere or eventlet.monkey_patch() once.
maybe you don't import sleep at all, then it's a NameError: sleep and all exceptions from execute are hidden by caller.
maybe you run this code with stderr redirected to file or /dev/null.
Let's also fix other issues.
try:
# ...
sleeep() # with 3 'e', invalid name
open('file', 'rb')
raise Http404
except:
# here you catch *all* exceptions
# in Python 2.x even SystemExit, KeyboardInterrupt, GeneratorExit
# - things you normally don't want to catch
# but in any Python version, also NameError, IOError, OSError,
# your application errors, etc, all irrelevant to timeout
raise TimeoutException("Error")
In Python 2.x you never write except: only except Exception:.
So let's catch only proper exceptions.
try:
# ...
execute_other() # also has Timeout, but shorter, it will fire first
except eventlet.Timeout:
# Now there may be other timeout inside `try` block and
# you will accidentally handle someone else's timeout.
raise TimeoutException("Error")
So let's verify that it was yours.
timeout = eventlet.Timeout(3)
try:
# ...
except eventlet.Timeout as t:
if t is timeout:
raise TimeoutException("Error")
# else, reraise and give owner of another timeout a chance to handle it
raise
Here's same code with shorter syntax:
with eventlet.Timeout(3, TimeoutException("Error")):
print("First")
eventlet.sleep(4)
print("Second")
print("Third")
I hope you really need to substitute one timeout exception for another.

Bypassing IOErrors and HTTP Exceptions in Selenium

I have a large scraping job that I am trying to run, using Selenium and PhantomJS in Python. It throws a couple of different errors after having run correctly for about 24 hours. Tested this a couple of times. Obviously, any new code added is a bit hard to test, as I have to wait for 24 hours to see it anything was solved. So, I was wondering if anyone with more experience could take a look at this piece of code and see if it seems ok. What I am trying to do is keeping the while loop going in spite of errors from the browser.
while something:
try:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
except httplib.HTTPException:
print 'HTTPException'
time.sleep(20)
pass
except IOError:
print 'IOError'
time.sleep(20)
pass
Your code looks fine. You can use except (httplib.HTTPException, IOError) as e:, print type(e).__name__ to combine the handlers, and you can drop the pass:
while something:
try:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
except (httplib.HTTPException, IOError) as e:
print type(e).__name__
time.sleep(20)
I'd use the logging module to provide logging information here; the logger.exception() function would include the exception and traceback in the output:
logger = logging.getLogger(__name__)
# ...
except (httplib.HTTPException, IOError) as e:
logger.exception('Ignoring exception, sleeping for 20 seconds')
time.sleep(20)

Categories

Resources