I'm trying to build a web scraper with selenium that pulls pricing and nutrition data from https://www.hannaford.com/home.jsp for a school project. The difficulty is that pricing data is store specific. To solve this I have used selenium to enter the users zip-code on the website. Afterwards, a pop up window opens with five of the nearest stores. The program needs to select a store from these locations.
I've copied the x-path of the button that selects the first store and set it to be clicked, but selenium still does not continue past the popup.
Here is my code:
hannaford =
"https://www.hannaford.com/home.jsp"
#opens website
driver.get(hannaford)
driver.find_element_by_xpath("""//*[#id="node01"]/a""").click()
driver.find_element_by_xpath("""//*[#id="pageContentWrapperInner"]/div[2]/div/div[1]/p[2]/a""").click()
inputElement = driver.find_element_by_xpath("""//*[#id="cityStateZipLayer"]""")
inputElement.send_keys(zip)
inputElement.send_keys(Keys.ENTER)
#attempt to select store in popup
driver.find_element_by_xpath("""//*[#id="glo-store-results-layer-store-list"]/div[1]/span/form/a""").click()
and here is the html associated with the link I'm trying to click:
<span class="fr rightSide">
<form class="fr" action="/custserv/save_user_store.cmd" method="post" name="selectThisStoreForm"><input type="hidden" name="form_state" value="selectThisStoreForm"><input name="storeId" type="hidden" value="21185" autofillparam="ON"><a href="" onclick="javascript:saveUserStoreNoClose('21185');return false;" class="altLink">
<input class="shopNow" type="image" autofillparam="ON" src="/assets/hf/assets/images/buttons/btn_shopNow.gif" border="0" alt="Shop Now"></a>
</form></span>
How could I resolve this issue. Is it even possible to select dynamic elements like this in selenium?
Related
I am trying to web scraping a popular movie database on the Internet. After a few thousands of requests, my spider is detected and I have to manually click on a simple button to continue my data collection.
I have tried to automatize this process by clicking automatically in that button. For that purpose, I am using Selenium library. I have located the xpath of the button, but, for some reason, .click() method doesn't work.
Here is my code:
def click_button():
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")
DRIVER_PATH = 'geckodriver'
driver = webdriver.Firefox(options=options, executable_path=DRIVER_PATH)
driver.get('https://www.filmaffinity.com/es/main.html')
element = driver.find_element(By.XPATH,"//input[#value='Send']")
element.click()
driver.quit()
Also, I have tried other common alternatives, such as waiting until the element gets clickable or visible, but this didn't work. I have also tried to execute the click as a javascript script.
This is the window I see when my spider is detected:
Too many request window (picture)
As you see, there is a reCaptcha-protection icon on the lower right corner, but I don't have to solve any captcha puzzle to confirm I am not a robot, I have just to click on the send button you can see on the picture.
The html which contains the button is the following one:
<div class="content">
<h1>Too many requests</h1>
<div class="image">
<img height="400" src="/images/too-many-request.jpg">
</div>
<div class="text">
Are you sure you do not dream of electric sheep?
</div>
<form name="toomanyrequest">
<div class="alert">
please enter the Captcha to continue.
</div>
<div>
<input type="submit" value="Send">
</div>
</form>
</div>
Which do you think is the trouble with my code or approach? I have to find a way to click on that button to continue my scraping, but I don't know what I am doing wrong (I have little experience at this field).
Maybe the xpath is not correct? Maybe the captcha protection is blocking the click action? I don't get any exception when I execute my code; but nothing happens and the "Too many request" window does not disappear.
Thank you very much.
Try this:
driver.execute_script("arguments[0].click()", element)
or:
driver.execute_script("arguments[0].click();", element)
<div id="ccodeinput">
<input class="dropdownheader2" type="TEXT" name="CCODE" value="435435" size="14" maxlength="22">
</div>
This is the code for the Search field(see screenshot - upper right side). I am unable to pass values from a list to this search bar. But I am able to press the search button beside it.
How can I enter values in the search field?
This is what I have tried:
inputElement = chrome_driver.find_element_by_id('ccodeinput')
inputElement.send_keys(435435)
As per the HTML you have shared the <input> tag is a child of <div> tag, so to put text within the intended element you can use :
chrome_driver.find_element_by_xpath("//input[#class='dropdownheader2' and #name='CCODE']").send_keys("435435")
I was doing some web scraping with python (Linkedin site) and got stuck with the following 2 issues: 1) How do I input text on a search bar? 2) How to click a button? First, this is the search bar code:
<input aria-autocomplete="list" autocomplete="off" spellcheck="false"
placeholder="Búsqueda" autocorrect="off" autocapitalize="off" id="a11y-
ember6214" role="combobox" class="ember-text-field ember-view" aria-
expanded="false">
To input the text I was using the xpath (and it works) but it changes every time I login into the site:
search = driver.find_element_by_xpath('//*[#id="a11y-ember997"]')
search.send_keys('MedMake')
So could I use instead part of the input bar code above so that I can rerun my script multiple times?
My second point is 2) how to click a button. Again I was using the xpath but it changes after every login. My code was:
button = driver.find_element_by_xpath('//*[#id="nav-search-controls-wormhole"]/button')
button.click()
I inspected the button code and I would instead like to use data-vertical="PEOPLE" or any other of this unique fields (the tag button is not enough since there are many buttons on Linkedin site). By the way,how are all these inner fields called? I believe part of my problem arises from the lack of html code understanding.
<button data-vertical="PEOPLE" data-control-
name="vertical_nav_people_toggle" data-ember-action="" data-ember-
action-8620="8620" data-is-animating-click="true">
Gente
</button>
If id attribute values are dynamic you can use other attributes with static values:
search = driver.find_element_by_xpath('//input[#placeholder="Búsqueda"]')
search.send_keys('MedMake')
button = driver.find_element_by_xpath('//button[normalize-space()="Gente"]')
button.click()
First one use xpath
//input[contains(#class,'ember-text-field')]
Second one use the xpath
//button[#class='vertical_nav_people_toggle']
EDIT
There was a bug in Firefor driver. It was solved by using chrome driver.
I am new to python and I am developing a script that will send message on whatsapp using webwhatsapp.
But I am not able to write anything in text inputfeild using driver.send_key('message') method.
<input type="text" class="input input-search" data-tab="2" dir="auto" spellcheck="false" title="Search or start new chat" value="">
CODE
contactSearch=driver.find_element_by_xpath('//*[#title="Search or start new chat"]').click()
contactSearch.send_keys("name")
I have tried contactSearch.clear() and click() method also but the input remains blank.the code is from https://web.whatsapp.com/ go there login and there will be a feild called Search or start new chat.I want to input text in that field.
You are not logged in to whatsapp in your selenium session and hence there is no "New chat" button. By default, selenium starts a brand new browser session with default preferences, no browser history or cookies.
I'm a selenium newbie and just trying to learn the basics. I have a simple CherryPy webapp that takes a first name and last name as input:
My Webapp:
<p>
<label></label>
<input name="first_name"></input>
<br></br>
</p>
<p>
<label></label>
<input name="last_name"></input>
<br></br>
</p>
In my python console I have:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://localhost:8080')
The page loads fine in FF but I'm a little lost on how to get text into the 'first_name' and 'last_name' text boxes. I see examples where you do something like inputElement = driver.find_element_by_id("n") and then inputElement.send_keys('my_first_name') but I don't have an id...just a name. Do I need to add stuff to my web page? Thanks!
You can use find_element_by_name:
driver.find_element_by_name('first_name').send_keys("my_first_name")
driver.find_element_by_name('last_name').send_keys("my_last_name")