I'm trying to gather price information for each product variation from this web page: https://www.safetysign.com/products/7337/ez-pipe-marker
I'm using Selenium and FireFox with Python 3 and Windows 10.
Here is my current code:
driver = webdriver.Firefox()
driver.get('https://www.safetysign.com/products/7337/ez-pipe-marker')
#frame = driver.find_element_by_class_name('product-dual-holder')
# driver.switch_to.frame('skuer5c866ddb91611')
# driver.implicitly_wait(5)
driver.find_element_by_id('skuer5c866ddb91611-size-label-324').click()
price = driver.find_element_by_class_name("product-pricingnodecontent product-price-content").text.replace('$', '')
products.at[counter, 'safetysign.com Price'] = price
print(price)
print(products['safetysign.com URL'].count()-counter)
So, I'm trying to start by just selecting the first product variation by id (I've also tried class name). But, I get an Unable to locate element error. As suggested in numerous SO posts, I tried to change frames (even though I can't find a frame tag in the html that contains this element). I tried switching to different frames using index, class name, and id of different div elements that I thought might be a frame, but none of this worked. I also tried using waits, but that return the same error.
Any idea what I am missing or doing wrong?
To locate the elements you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can create a List and iterate over it to click() each item and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver=webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("https://www.safetysign.com/products/7337/ez-pipe-marker")
for product in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//form[#class='product-page-form']//div[#class='sku-contents']//following::ul[1]/li//label[starts-with(#for, 'skuer') and contains(., 'Pipe')]"))):
WebDriverWait(driver, 20).until(EC.visibility_of(product)).click()
driver.quit()
They may well be dynamic. Select by label type selector instead and index to click on required item e.g. 0 for the item you mention (first in the list). Also, add a wait condition for labels to be present.
If you want to limit to just those 5 size choices then use the following css selector instead of label :
.sku-contents ul:nth-child(3) label
i.e.
sizes = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".sku-contents ul:nth-child(3) label")))
sizes[0].click()
After selecting size you can grab the price from the price node depending on whether you want the price for a given sample size e.g. 0-99.
To get final price use:
.product-under-sku-total-label
Code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://www.safetysign.com/products/7337/ez-pipe-marker'
driver = webdriver.Chrome()
driver.get(url)
labels = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "label")))
labels[0].click()
price0to99 = driver.find_element_by_css_selector('.product-pricingnodecontent').text
priceTotal = driver.find_element_by_css_selector('.product-under-sku-total-label').text
print(priceTotal, price0To99)
# driver.quit()
Related
Using selenium automation webdriver i'm unable to locate the text box element on a travel website using python. Using locator present in the webdriver such as Id/name/css_selector/class_name or xpath/full xpath.
Below is the screenshot of the python code:
[Code_1][1]
While the first one is located the second one isn't. The corresponding HTML code is
[text_box2][2]
How can i fill(automate) both fileds corresponding flight destinations i.e leaving and going
[1]: https://i.stack.imgur.com/gpoIr.jpg
[2]: https://i.stack.imgur.com/Q7mGs.jpg
Here is a slightly different approach for locating things. I am using waits borrowed from CruisePandey in this thread. I used firefox, but that is adaptable. Some notes of what was hard:
I had to make sure to be on the Flights tab.
I had to click in the from and to fields, which were buttons after all, and then wait to be able to type into the revealed input fields.
Finally, I had to choose the first element from the dropdown list that resulted.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox(executable_path='/usr/bin/geckodriver')
driver.maximize_window()
driver.get('https://www.expedia.co.in')
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Flights")))
driver.find_element_by_link_text('Flights').click()
wait.until(EC.presence_of_element_located((By.ID, "location-field-leg1-origin")))
driver.find_element(By.CLASS_NAME,'uitk-faux-input').click()
fieldInput = driver.find_element_by_id('location-field-leg1-origin')
wait.until(EC.visibility_of(fieldInput))
fieldInput.send_keys("SFO")
wait.until(EC.presence_of_element_located((By.TAG_NAME, "strong")))
driver.find_element_by_tag_name('strong').click()
driver.find_elements(By.CLASS_NAME,'uitk-faux-input')[1].click()
fieldInput = driver.find_element_by_id('location-field-leg1-destination')
wait.until(EC.visibility_of(fieldInput))
fieldInput.send_keys("BOS")
wait.until(EC.presence_of_element_located((By.XPATH,"//strong[contains(text(), 'Boston')]")))
driver.find_element_by_xpath("//strong[contains(text(), 'Boston')]").click()
You may want to use the below xpath for going To input field :
//label[text()='Going to']//following-sibling::input[#name]
Code :
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[text()='Going to']//following-sibling::input[#name]"))).send_keys('something')
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
if you want to automate from 1 place holder to another, i have use this set of code & it works for me
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[#id='location-field-leg1-origin-menu']/div[2]/ul/li[1]/button"))).click()
b = wait.until(EC.element_to_be_clickable((By.XPATH, ".//*[#id='location-field-leg1-destination-menu']/div[1]/button"))).send_keys("NYC")
b = wait.until(EC.element_to_be_clickable((By.XPATH,".//*[#id='location-field-leg1-destination-menu']/div[2]/ul/li[1]/button"))).click()
c = wait.until(EC.element_to_be_clickable((By.XPATH, ".//*[#id='d1-btn']"))).click()
My code looks like this:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time as t
PATH = "D:\CDriver\chromedriver.exe"
driver = webdriver.Chrome(PATH)
website = "https://jobs.siemens.com/jobs?page=1"
driver.get(website)
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "_ngcontent-wfx-c163="""))
)
print(element.text)
except:
driver.quit()
driver.quit()
Im trying to get the 6 numbers inside span _ngcontent-wfx-c163="">215022</span but cant seem to get it working, many others have had problems using span, but they have had a class inside it, mine doesnt.
How can I print the insides of the span tag that I have bolded?
If you are looking for req.ID to extract you can use the below CSS_SELECTOR :
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "p.req-id.ng-star-inserted>span"))
Note that there are 10 spans for req ID. you may use find_elements instead of find_element or probably EC.presence_of_all_elements_located which will give you list object. you can manipulate list as per your requirement.
read more about their difference here
Hoping you can help. I'm relatively new to Python and Selenium. I'm trying to pull together a simple script that will automate news searching on various websites. The primary focus was football and to go and get me the latest Manchester United news from a couple of places and save the list of link titles and URLs for me. I could then look through the links myself and choose anything I wanted to review.
In trying the the independent newspaper (https://www.independent.co.uk/) I seem to have come up against a problem with element not interactable when using the following approaches:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('chromedriver')
driver.get('https://www.independent.co.uk')
time.sleep(3)
#accept the cookies/privacy bit
OK = driver.find_element_by_id('qcCmpButtons')
OK.click()
#wait a few seconds, just in case
time.sleep(5)
search_toggle = driver.find_element_by_class_name('icon-search.dropdown-toggle')
search_toggle.click()
This throws the selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable error
I've also tried with XPATH
search_toggle = driver.find_element_by_xpath('//*[#id="quick-search-toggle"]')
and I also tried ID.
I did a lot of reading on here and then also tried using WebDriverWait and execute_script methods:
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="quick-search-toggle"]')))
driver.execute_script("arguments[0].click();", element)
This didn't seem to error but the search box never appeared, i.e. the appropriate click didn't happen.
Any help you could give would be fantastic.
Thanks,
Pete
Your locator is //*[#id="quick-search-toggle"], there are 2 on the page. The first is invisible and the second is visible. By default selenium refers to the first element, sadly the element you mean is the second one, so you need another unique locator. Try this:
search_toggle = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="row secondary"]//a[#id="quick-search-toggle"]')))
search_toggle.click()
First you need to open search box, then send search keys:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import os
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
browser = webdriver.Chrome(executable_path=os.path.abspath(os.getcwd()) + "/chromedriver", options=chrome_options)
link = 'https://www.independent.co.uk'
browser.get(link)
# accept privacy
button = browser.find_element_by_xpath('//*[#id="qcCmpButtons"]/button').click()
# open search box
li = browser.find_element_by_xpath('//*[#id="masthead"]/div[3]/nav[2]/ul/li[1]')
search_tab = li.find_element_by_tag_name('a').click()
# send keys to search box
search = browser.find_element_by_xpath('//*[#id="gsc-i-id1"]')
search.send_keys("python")
search.send_keys(Keys.RETURN)
Can you try with below steps
search_toggle = driver.find_element_by_xpath('//*[#class="row secondary"]/nav[2]/ul/li[1]/a')
search_toggle.click()
I'm trying to scrape a list from a website. There are two different lists, and one will load only after the first option is chosen. Issue is, I'm unable to select the first option. I scraped the list of all available options. But after writing it, I have to select it from the given option, and I'm unable to do so.
I've tried using browser.find_element_by_css_selector(....).click(), but it's showing the elementnotfound exception even after putting the proper wait condition. I think that's because it's unable to find that element.
browser.find_element_by_css_selector("#Brand_name").send_keys(company[i])
element= browser.find_element_by_css_selector("#Brand_name_selectWrap")
browser.implicitly_wait(5) # seconds
browser.find_element_by_css_selector("""#Brand_name_selectWrap > ul > li.selected""").click()
PS: Following is the link which I'm trying to scrape. I need all the mobiles listed company wide.
https://bangalore.quikr.com/Escrow/post-classifieds-ads/?postadcategoryid=227
Can someone kindly suggest some better way?
You can gather all options and their rel attributes into a dictionary and then loop that with appropriate wait conditions for the sublist to appear:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
d = webdriver.Chrome()
d.get('https://bangalore.quikr.com/Escrow/post-classifieds-ads/?postadcategoryid=227')
options = {i.get_attribute('textContent'):i.get_attribute('rel') for i in d.find_elements_by_css_selector('#Brand_name_selectWrap .optionLists li:not(.optionHeading) a')}
input_element = d.find_element_by_id('Brand_name')
for k,v in options.items():
input_element.click()
input_element.send_keys(k)
selector = '[rel="' + v + '"]'
WebDriverWait(d, 3).until(EC.element_to_be_clickable((By.CSS_SELECTOR, selector))).click()
WebDriverWait(d, 2).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#Model_selectWrap.showCustomSelect")))
I created a dictionary that had all the options for mobiles in it. The keys were the actual text that would go in the input box and the values are the rel attribute values for those elements. Each option has a rel attribute. It means that I can input the phone name via the key so as to generate the dropdowns to select your mobile from possible values, then use the rel attribute in a css attribute = value selector to ensure I click on the right one
The rel attribute inside anchor tags (<a>) describes the relation to the document where the link points to.
The selector variable just hold the current css attribute = value selector for getting an mobile drop down option by its rel attribute value.
The element is covered by an other element so you'll need to use ActionChains to preform the click.
You'll needto import it:
from selenium.webdriver.common.action_chains import ActionChains
Then to click on the input only after that send_keys:
input_el = browser.find_element_by_css_selector('#Brand_name')
ActionChains(browser).move_to_element(input_el).click().perform()
It's a good practice to use WebDriverWait to validate your conditions:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
wait = WebDriverWait(browser, 15)
input_el = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#Brand_name")))
ActionChains(browser).move_to_element(input_el).click().perform()
I am trying to click on a specific element that dynamically changes locations and therefore it changes xpaths and css selectors as well.
Tried xpath.
//*[#id="hld"]/div/div[X]/div[1]/h2/select
Note: The X will range from 2 to 10 depending on various factors.
There are no class names or IDs to use either. All I have to work with are the tag names.
My current code is as follows.
h2 = driver.find_element_by_tag_name("h2")
select = h2.find_element_by_tag_name("select")
select.click()
Unfortunately the select tag will load some time after the h2 tag, and I am trying to do a webdriverwait to wait until the element is clickable/visible before running the above code.
Sadly the proper syntax to single out the select element isn't clear to me. Below is the code to find the h2 tag, but I am trying to expand it out to focus in on the select tag.
WebDriverWait(driver, 30).until(
EC.visibility_of_element_located((By.TAG_NAME, "h2")))
Any help is greatly appreciated.
Try removing the dynamic div locator - the driver will iterate through the elements on the page and only click on it if it exists.
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
xpath = ".//[#id="hld"]/div/div/div/h2/select"
timeout = 30
WebDrierWait(driver, timeout).until(ec.presence_of_element_located(By.XPATH))
driver.find_element_by_xpath(xpath).click()
Otherwise, if the driver is finding multiple xpaths that match your xpath, you could try something like:
elements = driver.find_elements_by_xpath(xpath)
for element in elements:
try:
element.click()
except ElementNotVisibleException:
pass