Background
I am writing a script to automate web form submissions for a set of websites using Python and Selenium. However, the script needs to work on similar websites where the underlying HTML and general structure of a form is not known in advance.
I have managed to get this working well for standard input fields (e.g. text fields, radio buttons, etc), however I am having difficulty handling date picker fields, as these fields are very site-specific.
Despite these site-specific differences, there appears to be a common usage pattern: (i) click the date picker field to reveal the calendar pop-up; then (ii) click on the desired date.
Revealing the pop-up calendar by clicking on the date picker field is straightforward and results in a new snippet of HTML (defining the calendar) being inserted somewhere within the page source. If I could extract just this new HTML snippet then I could use this information to reason about the input field being a calendar and how to interact with it.
Question
How can I extract new HTML code that was generated by a user interaction, such as clicking on an input field?
Edited response according to user's feedbacks
Get html content of element
How to get & print WebElement instance content with python / selenium:
my_web_element.get_attribute('innerHTML')
# Or, depending what you want
my_web_element.get_attribute('outerHTML')
How to wait for an element
In this example I click on a link with id 'target' and wait for an element with id 'calendar' is visible.
driver.find_element_by_id('target').click()
calendar = WebDriverWait(driver, 30).until(
ec.visibility_of_element_located((By.ID, 'calendar'))
)
Conclusion
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import selenium.webdriver.support.expected_conditions as ec
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('http://spam.egg.bacon')
driver.find_element_by_id('target').click()
calendar = WebDriverWait(driver, 30).until(
ec.visibility_of_element_located((By.ID, 'calendar'))
)
print calendar.get_attribute('innerHTML')
Prints: <div id="calendar">I'm a calendar</div> with this complet example: https://gist.github.com/arount/285e5add758bb14074efc4708f1d6b07
Related
I am new to Selenium and I am trying to mimic user actions on a site to fetch data from a built in html page on button click. I am able to populate all the field details, but button click is not working, it looks like js code not running.
I tried many options like adding wait time, Action chain etc but it didnt work, i am providing site and code i have written.
driver = webdriver.Chrome()
driver.get("https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm")
driver.implicitly_wait(10)
assigned values to all the other fields
driver.find_element_by_id('rdDateToDate').click()
Dfrom = driver.find_element_by_id('fromDate')
Dfrom.send_keys("02-Oct-2020")
Dto = driver.find_element_by_id('toDate')
Dto.send_keys("08-Oct-2020")
innerHTML = driver.execute_script("document.ready")
sleep(5)
getdata_btn = driver.find_element_by_id('getButton')
ActionChains(driver).move_to_element(getdata_btn).click().click().perform()
I recommend using a full xpath.
chrome.get("https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm")
time.sleep(2)
print("click")
fullxpath = "/html/body/div[2]/div[3]/div[2]/div[1]/div[3]/div/div[1]/div/form/div[19]/p[2]/input"
chrome.find_element_by_xpath(fullxpath).click()
I have tried the button clicking and it worked with XPath ... I though its because of someone used the ID twice on a website, but I can not find it ... so i have no idea whats going wrong there ...
Good luck :)
I'm making a project which goes to my orders page on amazon and collects data like product name, price, delivery date using selenium (cuz there is no api for that, and cant do with bs4). I get login and get to orders page without any problem.But I'm stuck where i have to find the delivery date using find element by class( I chose class because all other delivery date text have same class), but selenium says it cannot find it.
No, its not in an iframe as i cant see the option for This Frame when i right click on that element.
here is the code -
import requests
from selenium import webdriver
import time
userid = #userid
passwd = #passwd
browser = webdriver.Chrome()
browser.get('https://www.amazon.in/gp/your-account/order-history?ref_=ya_d_c_yo')
email_input = browser.find_element_by_id('ap_email')
email_input.send_keys(userid)
email_input.submit()
passwd_input = browser.find_element_by_id('ap_password')
passwd_input.send_keys(passwd)
passwd_input.submit()
time.sleep(5)
date = browser.find_element_by_class_name('a-color-secondary value')
print(date.text)
Finding element by xpath seems to work, but fails to find the date for all orders as xpath is different for every element.
Any help is appreciated.
Thanks
Refers to this line:
date = browser.find_element_by_class_name('a-color-secondary value')
It seem like your element target having multiple class name, a-color-secondary and value. Sadly .find_element_by_class_name just for single class name.
Instead you can use .find_element_by_css_selector:
date = browser.find_element_by_css_selector('.a-color-secondary.value')
I am trying to automate my certain activities using selenium. I was launching a webpage and a security windows popup appears for the login credentials.
I have automated that using Autoit. Now after login I need to click on the option and I have tried it based on the find_element_by_text and find_element_by_id.
I was getting an error as Unable to find the element with CSS selector and I have seen some other post in the StackOverflow with the same issue but I could not able to fix it by myself.
Here is how my HTML looks like. Could you please guide me on this and also please share any document for further checking. Thanks.
driver = webdriver.Ie(r"C:\\IEDriverServer\\IEDriverServer.exe")
driver.get('URL of the page')
#driver.implicitly_wait(10) # seconds
autoit.win_wait("Windows Security")
# now make that crap active so we can do stuff to it
autoit.win_activate("Windows Security")
# type in the username
autoit.send('username')
# tab to the password field
autoit.send("{TAB}")
# type in the password
autoit.send('password')
# kill!
autoit.send("{ENTER}")
driver.maximize_window()
driver.implicitly_wait(120) # seconds
#submit_button_incidents = driver.find_element_by_link_text("3-Normal Incidents")
submit_button_incidents= driver.find_element_by_id("nodeImgs13pm")
submit_button_incidents.click()
driver.implicitly_wait(10)
++ updating the info
I have tried to copy the whole HTML but the page was restricted so I cant able to view the full HTML page other than the basic templates. Adding some more screenshots of the developer tools.
Here how my webpage looks like.
try with this code :
submit_button_incidents = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'My Group'])")))
submit_button_incidents.click()
and do not use implicit wait too many times in code.
Implicit wait is set for life time of web driver instance.
Reference :
Selenium official document for wiats
Xpath tutorial
cssSelector tutorial
UPDATE :
As OP has shared the HTML code. As per the requirement you can go ahead with this code.
As there are two elements with text as My Groups's queue.
For first one you can write XPATH as :
//a[#class='scbdtext']/span[contains(text(),'My Group')]
Code:
submit_button_incidents = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='scbdtext']/span[contains(text(),'My Group')]")))
submit_button_incidents.click()
For second one you can write XPATH as :
//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]
Code :
submit_button_incidents = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]")))
submit_button_incidents.click()
Hope this will help.
Use ActionChains with double click for this to work in Python.
from selenium.webdriver import ActionChains
# Get the element however you want
element = driver.find_element_by_xpath("//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]")
ActionChains(driver).double_click(settings_icon).perform()
I am programming a python scraper with help of Selenium. The first few steps are:
goes on booking.com, insert a city name, selects the first date and then tries to open the check-out calendar.
Here is where my problem occurs. I am not able to click the check-out calendar button (The important are of the website).
I tried to click every element regarding to the to check-out calendar (The elements of check-out calendar) with element.click(). I also tried the method
element = self.browser.find_element_by_xpath('(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]') self.browser.execute_script("arguments[0].click();", element)
It either does nothing (in case of execute.script() and click() on div elements) or it throws following exception when directly clicking the button:
Element <button class="sb-date-field__icon sb-date-field__icon-btn bk-svg-wrapper"
type="button"> is not clickable at point (367.5,316.29998779296875)
because another element <div class="sb-date-field__display"> obscures it
Here is a short code to test it:
browser = webdriver.Firefox()
browser.get("https://www.booking.com/")
wait = WebDriverWait(browser, 5)
element = wait.until(EC.presence_of_element_located((
By.XPATH, '(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')))
element = wait.until(EC.element_to_be_clickable((
By.XPATH, '(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')))
element.click()
I have a temporarily solution for my problem but I am not satisfied with it.
element = browser.find_element_by_xpath('(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')
hov = ActionChains(browser).move_to_element(element)
hov.click().perform()
This will open the calendar by hovering over the object and clicking it. This strangely opens the calendar.
The methods mentioned above still don't work.
Define clicka as an xpath. Now use executescript to click the element.
driver.execute_script("arguments[0].click();", clicka)
I'm not 100% sure that I got everything you posted, because the layout is a bit messy.
However, I tried to test the issue with both Selenium Java and Firefox Scratchpad (a Web Developer tool that allows to run JavaScript scripts) and it worked perfectly - the button was clickable on both of them.
If you're interested in further testing using this tool, this is the code I've used:
In JavaScript:
function getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
var myElement = getElementByXpath('(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')
myElement.click()
and in Java:
FirefoxDriver driver = new FirefoxDriver();
WebDriverWait wait = new WebDriverWait(driver, 10);
driver.navigate().to("https://www.booking.com");
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("(//div[contains(#class,'checkout-field')]//button[#aria-label='Open calendar'])[1]")));
driver.findElement(By.xpath("(//div[contains(#class,'checkout-field')]//button[#aria-label='Open calendar'])[1]")).click();
System.out.println("success");
if your are having the control check out button on all the web site managing with explicit wait required lots of codding you can use implicit wait below is in the java.
System.setProperty("webdriver.chrome.driver",
"G:\\TopsAssignment\\SampleJavaExample\\lib\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
I'd like to use python selenium to search at https://www.homeaway.com/
The following works:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://www.homeaway.com/")
driver.find_element_by_xpath("//input[#id='searchKeywords']").send_keys("Philadelphia, PA, USA")
But I'm running into an issue using the the calendar dropdown date picker, as it's not taking any of the values.
I've tried the following
Attempt 1 to enter a date into the start and end date field:
driver.find_element_by_xpath("//input[#id='stab-searchbox-start-date']").send_keys("02/01/2017")
driver.find_element_by_xpath("//input[#id='stab-searchbox-end-date']").send_keys("03/01/2017")
Note: It looks like homeaway website is completely ignoring the above commands unless you mannually click on the website with your mouse and then use the above selenium commands. In other words, the above commands are not working without a manual mouse click on the website first.
Attempt 2 to enter a date into the start and end date field:
driver.find_element_by_xpath("//div[#id='2017-02-01_2017-02']").click()
Attempt 3 to enter a date into the start and end date field
driver.execute_script("document.querySelectorAll('#stab-searchbox-start-date')[0].value = '02/01/2017'")
driver.execute_script("document.querySelectorAll('#stab-searchbox-end-date')[0].value = '03/01/2017'")
driver.find_element_by_xpath("//button[#class='btn btn-primary btn-lg searchbox-submit js-searchSubmit']").click()
Note: this looks like it works, but the dates are actually not registered when you click on search despite the dates being entered into the star and end date text boxes. It seems that homeaway will only register the dates if you use the calendar dropdown.
Note that you can not send_keys in these types when input text is not supported.
try this code, it will work:
from selenium import webdriver
url='http://www.homeaway.com/'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)
driver.find_element_by_id("search-location").send_keys("Philadelphia, PA, USA")
driver.find_element_by_id('search-checkin').click()
driver.find_element_by_xpath('//*[#id="ui-datepicker-div"]//*[text()="30"]').click()
driver.find_element_by_id('search-checkout').click()
driver.find_element_by_xpath('//*[#id="ui-datepicker-div"]//*[text()="31"]').click()