How do I go about measuring page download+rendering time of a website in selenium (Python)? Currently I have something like
... # set up web driver and loop through a list of domain names
print currDomainName
start = datetime.now()
browser.get(currDomainName)
end = datetime.now()
... # do something with the time diff
But it doesn't quite work because get() isn't guaranteed to block until page render is complete. Its official doc also says
In some circumstances, WebDriver may return control before the page has finished, or even started, loading.
In fact, in my short test code the print statement could literally be printing two or three urls further down before the one in webdriver finishes loading.
I know using explicit wait on certain webpage elements can enforce blocking, but is there a generic way of doing this? I'm measuring this on about a few thousand websites, so it would be great to have something independent from the web content.
Only way how to make sure that everything is loaded is to use ExplicitWait. Downside is that you have to know which element slows the rendering of each particular page but this is the only way how to keep your measure precise.
start = datetime.now()
browser.get(currDomainName)
WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID,
"reallyHugeImageId")))
end = datetime.now()
Note that time.sleep() would break your measures because you would always wait exact time. So only way is to investigate which elements are not yet rendered even if the WebDriver returns and wait from them through ExpectedCondition.
Related
I have a selenium script that runs a loop on a process that prints text into my python IDLE to get a list of data, which I then copy and paste to excel - which is great. But not optimal.
I am currently using sleep at the start of the loop to give me time to change parameters on my trading view strategy tester, which gives me a different result to be printed as intended. However, I would like more or less time on most occasions. Waiting for the path to change does not work as the path never changes, it just gets looped again with a different text output based on the changes I would have made manually on the strategy settings.
Also worth noting that I am using firefox geckobrowser as opposed to chrome and that the purpose of the script is to automate a tedious manual task of data collection.
Is it at all possible to create a pop-up button or something of the like via selenium that says "Next" for example, that I can click and let the script know its okay for the loop to continue? Can I simply replace sleep with click "Next" or something of the like for example?
for i in range(20):
time.sleep(10)
netProfit = driver.find_element("xpath", '/html/body/div[2]/div[7]/div[2]/div[4]/div/div[2]/div/div[1]/div[1]/div[2]/div[2]')
numberTrades = driver.find_element("xpath", '/html/body/div[2]/div[7]/div[2]/div[4]/div/div[2]/div/div[1]/div[2]/div[2]')
winRate = driver.find_element("xpath", '/html/body/div[2]/div[7]/div[2]/div[4]/div/div[2]/div/div[1]/div[3]/div[2]')
print(netProfit.text, numberTrades.text, winRate.text)
This works for now, but it's a pain waiting for timer on occasions I don't need it and even more of a pain on occasions when it's not enough time.
I am expecting something like this to replace time.sleep(10)
WebDriverWait(driver, 180).until("Next" button is clicked)
Ideally I would like a solution that can be done in selenium as my experience is limited, but any workarounds would be appreciated all the same.
i'm trying to find a way, to sleep for as long as it takes for an selenium element to be loaded. i tried it with actionchains and with the inbuilt sleep where you can define a max sleep timeout. However after doing so, it still wasn't what i tried to do (card number got cut off on the first part thus the enterd card number wasn't valid. i then added a 10 seconds time.sleep interval which solved the issue however, sometimes it dosn't work just because it takes more time to load (maybe connection is bad) or the sleep interval is set too high what makes the code unncececary slow.
I expected it to work. I create this code for someone else and this person had the issues with it being too fast while on my side everything went flawlessly.
I don't get any error message it's just a question, how i can implement the sleep within time.sleep for as long as it takes for the element to be fully loaded.
I've tried to use Webdriverwait but this didn't did the thing:
WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
There is no such Expected Condition to wait until the element to be fully loaded.
I don't know exactly what are you trying to get but most probably the element is loaded in DOM so the expected condition is met even if the content of that element is not there yet.
If you are trying to access a button you can use the EC.element_to_be_clickable.
If it is something visible on your screen you can use EC.visibility_of_element_located.
The solution here lies in the ExpectedConditions and not in the sleep time which is really random if you will get the content and it will be really slow.
Hope this helps!
I use python/selenium for web-scraping and automation through Chrome. Recently I've come across a problem where some pages or websites would load endlessly as a means to prevent bots from scraping them (It often starts to happen after first time). I had no idea how to counter this until I came across the function in question:
wait = WebDriverWait(self.driver, 3)
element = wait.until(
EC.presence_of_element_located((By.XPATH, '//button[.="Sign up"]'))
)
The way I understood this function is that it sat a timeout for the page (3 seconds in this example) and at the same time checking whether an element has been loaded into the page or not. If the element has been loaded or if the wait times out, the program stops waiting for the page to load and continues on with the next line of code, which is, in my case, a print("timeout") function.
Thing is, as I use the function as demonstrated above the program still waits for the page to load, as "timeout" only appears in the console after I manually stop the page loading from the browser.
I am sure the element is present in the page (and quite visible) as it continues to load endlessly, because once I press the X button and stop the page loading, "timeout" is printed and the element is interacted with successfully.
I also tried EC.visibility_of_element_locatedto the same effect.
What is going wrong and how do I achieve the functionality I need?
Since you didn't share a link to the page you are working on I can only guess.
So my guess is: you are using a wrong locator.
Try using this
wait = WebDriverWait(self.driver, 3)
element = wait.until(EC.presence_of_element_located((By.XPATH, '//button[contains(.,"Sign up")]')))
Or this
wait = WebDriverWait(self.driver, 3)
element = wait.until(EC.presence_of_element_located((By.XPATH, '//button[contains(.,"sign up")]')))
This is a code that I use to register my courses for semester. I want this code to run fastest as possible so that I can select my preferred courses faster than other students who select them manually, the seats and section of desired classes fill-up fast. My concern is that if my code runs faster and the browser cant keep up with it then it will cause issue. For example I will be running on google chrome browser. And what should I change to make this code faster without risking Thanks in advance.
import pyautogui
import time
import webbrowser
pyautogui.FAILSAFE = False
time.sleep(1)
pyautogui.hotkey('alt', 'tab')
##Subject choose
time.sleep(0.5)
pyautogui.hotkey('ctrl', 'f')
pyautogui.write('PHY182.1')
pyautogui.press('enter')
pyautogui.press('esc')
pyautogui.hotkey('shift', 'tab')
pyautogui.press('space')
I did something similar I while ago, not exactly your case but I believe it would help. If you want it to be as fast as possible while making sure that the browser if keeping up with you, you should make your script recognize that the page you are going to be on has fully loaded, for this I would use selenium, so when the page is fully loaded it does the thing really fast, but not before. I think that if you did it like that it would work.
You have to setup selenium first, it's a bit long so I won't explain here, I'll just give you an example of it being used.
Selenium example:
driver.maximize_window()
driver.get(url) # goes to url and waits till it's loaded #
einput = driver.find_element_by_id("identifierId") # find the element you want through it's name #
einput.send_keys(email) # what you want to write #
einput.send_keys(Keys.RETURN) # enter #
I used this code for a script I made that logged people into google really fast in incognito. But I believe you could adapt it to your case. It's not perfect though because some pages 'never finish loading' but give it a try.
Im having this code, if there's the time.sleep(2) it clicks and if it's not there it doesnt click on the elements. It doesn't even wait a second
time.sleep(2)
WebDriverWait(self.broswer, 30).until(EC.element_to_be_clickable((By.CLASS_NAME, 'something')))
self.browser.find_element_by_class_name('something').click()
I tried to use the following instead
self.browser.implicitly_wait(2)
but it doesnt wait and I cant use time.sleep()
Time.Sleep() is counting time in milliseconds, therefore Time.Sleep(2) is 2 milliseconds, maybe if you try,
Time.Sleep(2000);
it may work for you.
time.sleep() and WebDriverWait() are different methods of waiting. It's best practice to avoid use of time.sleep() and prefer WebDriverWait.
In another comment you said that you don't even want to wait, you just want to click the button. If that's the case, you should be able to just use this
self.browser.find_element_by_class_name('something').click()
but you said that you get an error. What is the error that you get if you only run that one line?
Side note...
WebDriverWait.until() should return the WebElement specified so if you decide you want to keep the wait, you can do this
WebDriverWait(self.broswer, 30).until(EC.element_to_be_clickable((By.CLASS_NAME, 'something'))).click()
I come from a Java background and don't know python so I may have some typos here and there... but this should get you headed in the right
what is the problem with time.sleep()
If you don't want to user time.sleep() and still if you want to force web driver to wait for specific time then you can give condition also.
you can go through with below url for java
http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/support/ui/ExpectedConditions.html#invisibilityOfElementLocated-org.openqa.selenium.By-
1)
Use the time.sleep() (which is in seconds "Suspend execution of the current thread for the given number of seconds")only when you really need it.
2)
Learn, deeply, the differences between IMPLICIT WAIT and EXPLICIT WAIT
Sometimes,I had the need to force the "physical" hard-ware wait, the I used the time.sleep(), but I really suggest you to comprehend the difference between the two waiting method which selenium provides you.
#Dor Alt: Remember that the (from documentation) "An explicit waits is code you define to wait for a certain condition to occur before proceeding further in the code." If you get any kind of error, please write it here and tell us which browser you're using for your test, because different browsers have (very, very and very) different behaviours! :)
I mean, if you are using Chrome and IE (as far as I know) if the webdriver doesn't have the visualization of the element you want to click, it does not click! In this case, for example, you should move/scroll the page with a script like this: driver.execute_script('window.scrollTo(0, {0})'.format(element.location['y']))