Thinking of best practices, is it good to use try except when trying to get respond from requests.get(url) or using selenium webdriver.get(url) ?
Maybe more general question, when its meant to be use try except - except of file handling?
Thank you.
for example:
import requests
try:
respond = requests.get('https://www.google.com')
print(respond.status_code)
except Exception as e:
print(f'error while open url - {e}')
I would say it is good practice, even tough it might never come up especially when dealing with stable and widely used sites, such as Google-sites.
But in the case that the site you are trying to request is down or not responding to you, in my experience try except comes in handy and speeds up the process of finding the cause of an error.
This is good practice to have, from experience if you leave the GET request outside of a try, if you wanted to loop through a list of URLs, this would end up falling over.
Also, if it fails, you could then handle that in the exception, for example your ip is blocked from accessing x website, you can output the url and either retry with proxies or write out the url for future handling.
Related
I'm using a Python script to get Stock price from an API,
Everythinks works great but sometimes I receive html errors instead of prices,
These errors prevent the script from continuing and the terminal stopped working,
how do I test the response from the API before passing the information to the next script?
I don't want the terminal to stop when it receives server errors.
Only one line to get price :
get_price = api_client.quote('TWTR')
The question, "How do I test the response from the API before passing the information" is well suited for a try/except block. For example the following will attempt to call quote() and if an exception is raised it will print a message to the console:
try:
get_price = api_client.quote('TWTR')
except <Exception Type Goes Here>:
# handle the exception case.
print('Getting quote did not work as expected')
The example is useful for illustrating the point but you should improve the exception handling. I recommend you take a look at resources to help you understand how to appropriately handle exceptions in your code. A few that I have found useful would be:
The Python Docs: Learn what an exception is and how they can be used
Python Exception Handling: A blog post that details how to combine some of the concepts in the Python documentation practically.
I am using proxies rotating and sometimes it could happen one of these randomly doesn't work and I get "ERR_TIMED_OUT" enable to reach server and script just crashed without continuing, is it possible to refresh the webpage automatically in silenium when this happens (so the proxy will rotate). I thought about catch exception putting then driver.refresh() but how can I catch into the entire code without try- except for every instruction? is there another solution? thanks!
You can use event_firing_webdriver:
https://www.selenium.dev/selenium/docs/api/py/webdriver_support/selenium.webdriver.support.event_firing_webdriver.html
you can decorate method get() and execute try except there (with refresh on timeout exception).
If I request data from an API using a raspberry pi in a while/for loop in python and append data to csv and one iteration fails due to something like faulty wifi connection that comes and goes, what is a foolproof method of having an indication that an error occurred and have it keep trying again either immediately or after some rest period?
Use try/except to catch the exception, e.g.:
while True:
try:
my_function_that_sometimes_fails()
except Exception e:
print e
I guess retry package (and decorator) will suit your needs. You can specify what kind of exception it should catch and how many times it should retry before stopping completely. You can also specify the amount of time between each try.
I'm using PhantomJS as a webdriver to load some urls. Usually, the program runs fine. However, it hangs on driver.get(url) a lot, and i'm wondering if there is anything I can do about it?
driver = webdriver.PhantomJS(executable_path= path_to_phantomjs_exe, service_log_path= path_to_ghostdriver_log)
driver.get(url)
It will just hang trying to load a certain url forever. But if i try it again, it might work. Are webdrivers/phantomJS really just that unstable? I guess last resort would be to constantly call driver.get(url) until it finally loads, but is that really going to be necessary? Thanks!
EDIT: It seems to only hang when loading the first link out of a list of them. It eventually does load, however, but after a few minutes. The rest of the links load within seconds. Any help at all would be great.
I've answered this exact problem on this post here: Geb/Selenium tests hang loading new page but copied it here because I see that this question is older.
I hope you can find a way to implement this into your code, but this is what worked for me when I was having a similar situation with PhantomJS hanging.
I traced it to be hanging on a driver.get() call, which for me was saying something wasn't going through or the webdriver just wasn't - for some reason - giving the load successful command back to the driver, allowing the script to continue.
So, I added the following:
driver = webdriver.PhantomJS()
# set timeout information
driver.set_page_load_timeout(15)
I've tested this at a time of 5 (seconds) and it just didn't wait long enough and nothing would happen. 15 seconds worked great for me, but that's maybe something you should test.
On top of this, I also created a loop whenever there was an option for the webdriver to timeout, so that the driver.get() could attempt to re-send the .get() command. Implementing a try / except stacked scenario, I was able to approach this:
while finished == 0:
try:
driver.get(url3)
finished = 1
except:
sleep(5)
I have seen an except handle as:
except TimeoutException as e:
#Handle your exception here
print(e)
but I had no use for this. It might be nice to know how to catch specific exceptions, though.
See this solution for more options for a timeout: Setting timeout on selenium webdriver.PhantomJS
So I was having the same problem:
driver = webdriver.PhantomJS(executable_path= path_to_phantomjs_exe, service_log_path= path_to_ghostdriver_log)
driver.get(url)
So I changed the service_log_path to:
service_log_path=os.path.devnull
This seemed to work for me!!!
I'm downloading a huge set of files with following code in a loop:
try:
urllib.urlretrieve(url2download, destination_on_local_filesystem)
except KeyboardInterrupt:
break
except:
print "Timed-out or got some other exception: "+url2download
If the server times-out on URL url2download when connection is just initiating, the last exception is handled properly. But sometimes server responded, and downloading is started, but the server is so slow, that it'll takes hours for even one file, and eventually it returns something like:
Enter username for Clients Only at albrightandomalley.com:
Enter password for in Clients Only at albrightandomalley.com:
and just hangs there (although no username/passworde is aksed if the same link is downloaded through the browser).
My intention in this situation would be -- skip this file and go to the next one. The question is -- how to do that? Is there a way in python to specify how long is OK to work on downloading one file, and if more time is already spent, interrupt, and go forward?
Try:
import socket
socket.setdefaulttimeout(30)
If you're not limited to what's shipped with python out of the box, then the urlgrabber module might come in handy:
import urlgrabber
urlgrabber.urlgrab(url2download, destination_on_local_filesystem,
timeout=30.0)
There's a discussion of this here. Caveats (in addition to the ones they mention): I haven't tried it, and they're using urllib2, not urllib (would that be a problem for you?) (Actually, now that I think about it, this technique would probably work for urllib, too).
This question is more general about timing out a function:
How to limit execution time of a function call in Python
I've used the method described in my answer there to write a wait for text function that times out to attempt an auto-login. If you'd like similar functionality you can reference the code here:
http://code.google.com/p/psftplib/source/browse/trunk/psftplib.py