I am trying to write a script that will automatically open to a specific bible verse with it's commentaries open, however, click does not activate the commentary window to open. What is wrong?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Firefox()
driver.get("https://catenabible.com/mt/1")
assert "Catena" in driver.title
elem = driver.find_element(By.ID, "mt001001")
elem.click()
You are missing a wait.
You should wait for the page to complete loading before clicking that element.
The best way to do that is to use the Expected Conditions explicit waits.
The following code should work better:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Firefox()
wait = WebDriverWait(driver, 20)
driver.get("https://catenabible.com/mt/1")
assert "Catena" in driver.title
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#mt001001 .bibleVerseText"))).click()
Others answers have stated that you have to click the A tag or the SPAN is not clickable but that isn't true. You just need to wait for the element to be clickable. This opens the right-side Commentaries panel for me.
import time
driver.get("https://catenabible.com/mt/1")
assert "Catena" in driver.title
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "mt001001"))).click()
count = 0
while len(driver.find_elements((By.CSS_SELECTOR, "div.commentaryPanel.expanded"))) == 0 and count < 5:
driver.find_element((By.ID, "mt001001")).click()
count += 1
time.sleep(0.25) # brief pause between loops
assert count == 5, "Commentary was not visible after 5 tries"
...
EDIT:
I got it to work by using an explicit wait of 2 seconds to allow for the element to be clickable:
import time
driver.get("https://catenabible.com/mt/1")
time.sleep(2)
driver.find_element(By.XPATH, '//*[#id="mt001001"]/span').click()
It is not ideal to use this wait because it will slow the script down, and you never really know how long will be enough time to wait. But at least it works.
Related
When I load the my site as shown below, a splash screen appears to confirm I am 21 years of age. I am trying to load the element to click yes, but I am unable to bypass the age verification screen.
My approach was to load the page. Add a sleep time, find the element and click Yes. However it wont work.
driver.get('https://seelbachs.com/products/sagamore-spirit-cask-strength-rye-whiskey')
time.sleep(5)
element= driver.find_element_by_xpath('//*[#id="enter"]')
element.click()
Error received.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="enter"]"}
So either my find.element is off or my time.sleep is not helping with the splash page.
It seems the problem is that the pop-up is in an iframe.
This code appears to do the trick
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver import ActionChains
import time
driver = webdriver.Chrome('/usr/local/bin/chromedriver')
driver.get('https://seelbachs.com/products/sagamore-spirit-cask-strength-rye-whiskey')
# xpath of iframe
frame_xpath = '/html/body/div[5]/div/div/div/div/iframe'
wait = WebDriverWait(driver, 10)
# wait until iframe appears and select iframe
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, frame_xpath)))
# select button
xpath = '//*[#id="enter"]'
time.sleep(2)
element= driver.find_element_by_xpath(xpath)
ActionChains(driver).move_to_element(element).click(element).perform()
referencing This answer on Stack Overflow
You need to switch to the frame, check the below code
driver.get('https://seelbachs.com/products/sagamore-spirit-cask-strength-rye-whiskey')
sleep(5)
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[contains(#id,'fancyboxAge-frame')]"))
OR
#Using the explicitWait
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#id,'fancyboxAge-frame')]")))
element= driver.find_element_by_xpath('//*[#id="enter"]')
element.click()
import
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
I am using selenium to try to scrape data from a website (https://www.mergentarchives.com/), and I am attempting to get the innerText from this element:
<div class="x-paging-info" id="ext-gen200">Displaying reports 1 - 15 of 15</div>
This is my code so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Firefox()
driver.maximize_window()
search_url = 'https://www.mergentarchives.com/search.php'
driver.get(search_url)
assert 'Mergent' in driver.title
company_name_input = '//*[#id="ext-comp-1009"]'
search_button = '//*[#id="ext-gen287"]'
driver.implicitly_wait(10)
driver.find_element_by_xpath(company_name_input).send_keys('3com corp')
driver.find_element_by_xpath(search_button).click()
driver.implicitly_wait(20)
print(driver.find_element_by_css_selector('#ext-gen200').text)
basically I am just filling out a search form, which works, and its taking me to a search results page, where the number of results is listed in a div element. When I attempt to print the text of this element, I simply get a blank space, there is nothing written and no error.
[Finished in 21.1s]
What am I doing wrong?
I think you may need explicit Wait :
wait = WebDriverWait(driver, 10)
info = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[#class = 'x-paging-info' and #id='ext-gen200']"))).get_attribute('innerHTML')
print(info)
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You may need to put a condition by verifying if search results loaded or not and once its loaded you can use below code
print(driver.find_element_by_id('ext-gen200').text)
I've written a script in Python in association with selenium to keep clicking on MORE button to load more items until there are no new items left to load from a webpage. However, my below script can click once on that MORE button available on the bottom of that page.
Link to that site
This is my try so far:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://angel.co/companies?company_types[]=Startup&company_types[]=Private+Company&company_types[]=Mobile+App&locations[]=1688-United+States"
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)
while True:
for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))):
print(elems.get_attribute("href"))
try:
loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class='more']")))
driver.execute_script("arguments[0].scrollIntoView();", loadmore)
loadmore.click()
except Exception:break
driver.quit()
How can I keep clicking on that MORE button until there are no such button left to click and parse the links as I've already tried using for loop.
I've managed to solve the problem pursuing sir Andersson's logic within my exising script. This is what the modified script look like.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://angel.co/companies?company_types[]=Startup&company_types[]=Private+Company&company_types[]=Mobile+App&locations[]=1688-United+States"
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)
while True:
try:
loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class='more']")))
driver.execute_script("arguments[0].click();", loadmore)
wait.until(EC.staleness_of(loadmore))
except Exception:break
for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))):
print(elems.get_attribute("href"))
driver.quit()
why not just?
while (driver.FindElements(By.ClassName("more")).Count > 0)
{
driver.FindElement(By.ClassName("more")).Click();
//Some delay to wait lazyload to complete
}
c# example. pretty sure that it can be done with python as well
So, what's I want to do is to run a function on a specific webpage (which is a match with my regex).
Right now I'm checking it every second and it works, but I'm sure that there is a better way (as it's flooding that website with getting requests).
while flag:
time.sleep(1)
print(driver.current_url)
if driver.current_url == "mydesiredURL_by_Regex":
time.sleep(1)
myfunction()
I was thinking to do that somehow with WebDriverWait but not really sure how.
This is how I implemented it eventually. Works well for me:
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5)
desired_url = "https://yourpageaddress"
def wait_for_correct_current_url(desired_url):
wait.until(
lambda driver: driver.current_url == desired_url)
I was thinking to do that somehow with WebDriverWait
Exactly. First of all, see if the built-in Expected Conditions may solve that:
title_is
title_contains
Sample usage:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
wait.until(EC.title_is("title"))
wait.until(EC.title_contains("part of title"))
If not, you can always create a custom Expected Condition to wait for url to match a desired regular expression.
To really know that the URL has changed, you need to know the old one. Using WebDriverWait the implementation in Java would be something like:
wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.not(ExpectedConditions.urlToBe(oldUrl)));
I know the question is for Python, but it's probably easy to translate.
Here is an example using WebdriverWait with expected_conditions:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://example.com/before'
changed_url = 'https://example.com/after'
driver = webdriver.Chrome()
driver.get(url)
# wait up to 10 secs for the url to change or else `TimeOutException` is raised.
WebDriverWait(driver, 10).until(EC.url_changes(changed_url))
Use url_matches Link to match a regex pattern with a url. It does re.search(pattern, url)
from selenium import webdriver
import re
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
pattern='https://www.example.com/'
driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)
wait.until(EC.url_matches(pattern))
I am new to Selenium and I would like to write a Python script that searches some keywords on google and automatically opens a page when the keywords are found.
I have been trying to work around with the script from this website:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
if __name__ == "__main__":
driver = webdriver.Firefox()
wait = WebDriverWait(driver, 100)
driver.get("http://google.com")
inputElement = driver.find_element_by_name("q")
inputElement.send_keys("Irwin Kwan")
wait.until(EC.element_to_be_clickable((By.XPATH,"//a[#href='http://irwinhkwan.wordpress.com/']")))
blog = driver.find_element_by_xpath("//a[#href='http://irwinhkwan.wordpress.com/']")
blog.click()
driver.quit()
It works perfectly if the xpath is displayed in the page. However, if it is not, we wait forever.
How can I check if an element is there, and if it is there I click on it?
I tried using 'try' as it is given in the documentation, or NoSuchElementException, but I admit being confused with the "double negation ;-):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()
Ultimately, I would like a while loop that goes to the page number X of the google results and click each time it finds a specific address.
If someone can give me a hand that would be great! I am pretty sure it is trivial, but despite all the documentation available online I haven't been able to work this out for the last couple of days...
here is a little wrapper for WebElement I’ve written, hope it helps
class Element(object):
def __init__(self, xpath=None):
self.xpath = xpath
def __get_element(self):
return Driver().find_element_by_xpath(self.xpath)
def is_exist(self):
try:
return self.__get_element().is_displayed()
except NoSuchElementException, e:
logger.exception(e.message)
return False
you cant click an element that is not visible, first you need to expose it, and then only click.
hope it helps