I am trying to web scraping a popular movie database on the Internet. After a few thousands of requests, my spider is detected and I have to manually click on a simple button to continue my data collection.
I have tried to automatize this process by clicking automatically in that button. For that purpose, I am using Selenium library. I have located the xpath of the button, but, for some reason, .click() method doesn't work.
Here is my code:
def click_button():
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")
DRIVER_PATH = 'geckodriver'
driver = webdriver.Firefox(options=options, executable_path=DRIVER_PATH)
driver.get('https://www.filmaffinity.com/es/main.html')
element = driver.find_element(By.XPATH,"//input[#value='Send']")
element.click()
driver.quit()
Also, I have tried other common alternatives, such as waiting until the element gets clickable or visible, but this didn't work. I have also tried to execute the click as a javascript script.
This is the window I see when my spider is detected:
Too many request window (picture)
As you see, there is a reCaptcha-protection icon on the lower right corner, but I don't have to solve any captcha puzzle to confirm I am not a robot, I have just to click on the send button you can see on the picture.
The html which contains the button is the following one:
<div class="content">
<h1>Too many requests</h1>
<div class="image">
<img height="400" src="/images/too-many-request.jpg">
</div>
<div class="text">
Are you sure you do not dream of electric sheep?
</div>
<form name="toomanyrequest">
<div class="alert">
please enter the Captcha to continue.
</div>
<div>
<input type="submit" value="Send">
</div>
</form>
</div>
Which do you think is the trouble with my code or approach? I have to find a way to click on that button to continue my scraping, but I don't know what I am doing wrong (I have little experience at this field).
Maybe the xpath is not correct? Maybe the captcha protection is blocking the click action? I don't get any exception when I execute my code; but nothing happens and the "Too many request" window does not disappear.
Thank you very much.
Try this:
driver.execute_script("arguments[0].click()", element)
or:
driver.execute_script("arguments[0].click();", element)
Related
I'm a real noob and I started to learn Python a few weeks ago.
I need to create an automation for eCommerce, I need to log in, select a page, **select a checkbox ** and click the button "submit". It seems very simple I know but on this website, the normal procedure doesn't work:
This is the HTML code:
<div class="loader-container"\>
<table class="table-grey table__orders table"\>
<thead\>
<th class="check-all table__orders-checker"\>
*\*\<div class="check-all__box"\>
<input type="checkbox" value="checkAll"\>\*\*
</div\>
This is what I coded to click the "check all" checkbox:
driver.find_element(
By.xpath, "//div[contains(#class, 'loader-container')]/table/thead/tr/th/div/input[#type='checkbox']").click()
I've tried also:
WebDriverWait(driver, 30).until(EC.element_to_be_clickable(
By.XPATH, "//div[contains(#class, 'loader-container')]/table/thead/tr/th/div/input[#type='checkbox']")).click()
Both without success, nothing happens when it should click this checkbox, I've also other try looking on this forum or YouTube but I'm missing something because I can't fix it.
Thank you in advance for your patience!
When I enter my account and make request to the invite link and turns out that invite link expired, selenium correctly determines the button "Continue to Discord":
button = driver.find_element_by_css_selector("[type='button']")
But when invite is still valid, selenium can't determine button "Accept Invite". Codes of these both buttons:
<button type="button" class="marginTop40-i-78cZ button-3k0cO7 button-38aScr lookFilled-1Gx00P colorBrand-3pXr91 sizeLarge-1vSeWK fullWidth-1orjjo grow-q77ONN"><div class="contents-18-Yxp">Accept Invite</div></button>
<button type="button" class="marginTop40-i-78cZ button-3k0cO7 button-38aScr lookFilled-1Gx00P colorBrand-3pXr91 sizeLarge-1vSeWK fullWidth-1orjjo grow-q77ONN"><div class="contents-18-Yxp">Continue to Discord</div></button>
Why can't selenium find first button if their codes are actually the same? I tried almost all methods. By xpath, by css selector and by class. Nothing helped. Seems like changing browser doesn't help too. Any ideas?
<a class="btn btn-xs btn-success" href="/user-location-auth/location?id=4013"
title="View User Location Access" data-toggle="tooltip" data-method="post">
<span class="glyphicon glyphicon-map-marker"></span></a> <span
class="glyphicon glyphicon-map-marker"></span> <<pseudo:before>>
</<pseudo:before>>
Hi all, I have been tasked with a project that requires clicking on a button in a table (with 14 cells). The first 13 cells are all unhidden, but the 14th one (where the buttons are located), are hidden unfortunately. I have tried action chains, I have tried hovering over the element and then clicking on it. I seem to be at a loss here. This piece of code did work, however I want to avoid using javascript as I want to replicate a user experience.
z4 = driver.find_element_by_css_selector('a.btn-success > span.glyphicon-map-marker')
driver.execute_script('arguments[0].click();', z4)
Hi everyone and cruisepandey, I found a solution that works. First, I waited for the presence of the element to be located. Once that occurred, I grabbed the element and moved to it with ActionChains (I think I was misusing ActionChains earlier). Then I was able to click on the Bootstrap button.
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, """
//tr[#data-key='2312312321321']/td/a[contains(#class, 'btn-xs btn-
success')]/span[contains(#class, 'glyphicon')]""")))
y = driver.find_element(By.XPATH, """//tr[#data-
key='2312312321321']/td/a[contains(#class, 'btn-xs btn-
success')]/span[contains(#class, 'glyphicon')]""")
#Use action chains to click on the button
ActionChains(driver).move_to_element(y).click().perform()
I was implementing a bot to click on upvote button in reddit
Here is the html code of the upvote button.
<div class="arrow up login-required access-required" data-event-action="upvote" role="button" aria-label="upvote" tabindex="0"></div>
<div class="score likes">•</div>
<div class="score unvoted">•</div>
<div class="score dislikes">•</div>
<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0"></div>
I am able to click on the upvote button using
target = driver.find_element_by_xpath("//div[#class='arrow up login-required access-required']")
target.click()
OR
target = driver.find_element_by_css_selector("div.arrow.up")
target.click()
The clicking works fine in FIREFOX web driver , but when I try to implement the same in PhantomJS , the browser is not clicking.
Here is the screenshots of 2 browsers
Try performing click() operation via jQuery or JavaScript using execute_script method of selenium. Below is the sample code that will perform the click operation.
driver.execute_script("$('div.arrow.up').click()")
For the page which has multiple upvote fields then you can use an index to choose which one you want to click e.g
driver.execute_script("$('div.arrow.up')[0].click()")
driver.execute_script("$('div.arrow.up')[1].click()")
driver.execute_script("$('div.arrow.up')[2].click()")
and so on..
How to automate webpage button control like having span child by using
driver. findElement_by_xpath ()
on Ubuntu Linux and chromium Autotest
The wesite is: https://www.youtube.com/my_webcam?privacy=public
I would like to click on Start recording button.
HTML source is:
<button class="yt-uix-button yt-uix-button-size-default yt-uix-button-dark yt-uix-button-has-icon webcam-record-button" type="button" onclick=";return false;" id="record-button">
<span class="yt-uix-button-icon-wrapper">
<span class="yt-uix-button-icon yt-uix-button-icon-upload yt-sprite">
</span>
</span>
<span class="yt-uix-button-content">Start recording </span>
</button>
We have tried it by using
driver.find_element_by_css_selector()
driver.find_element_by_id()
driver.find_element_by_xpath ()
However, nothing has worked. Could you please suggest us appropriate solution?
The record button is within an iframe, you'll need to switch that iframe first and then locate the button with your chosen selector.
iframe = self.driver.find_element_by_css_selector("#webcam-container")
self.driver.switch_to_frame(iframe)
record = self.driver.find_element_by_css_selector("#record-button > span.yt-uix-button-content")
record.click()