Develop Reference

Unable to locate "Accept" Button - Selenium - Beginner Web Scraping

I am trying to use Selenium in order to learn different ways of web scraping.
When the code is executed Firefox starts and the "accept cookies" or what ever pops up. I am unable to locate the "accept" button when inspecting the page.
my code so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
import pandas as pd
import time
PATH = "C:/Users/myuser/Desktop/Driver/geckodriver.exe"
driver = webdriver.Firefox(executable_path=PATH)
driver.maximize_window() # For maximizing window
driver.get("https://www.immonet.de/")
button_pos = driver.find_element(by=By.CLASS_NAME, value="sc-gsDKAQ fILFKg")
button_pos.click()
print(driver.title)
input = input()
I get the following error: Unable to locate element: .sc-gsDKAQ fILFKg
My thought was locating the button via the inspect tool as follows:
What am I missing or doing wrong? How would i find the right element?
Thanks!
Pat

First of all, to display this url,accepting the cookies is a must but to accept and click on the cookie button isn't a easy task because cookies button is under shadow root (open) selenium and webdriverWait can do nothing on shadow root,so to execute shadow root you need to apply JavaScript querySelector.
#To execute shadow root and accept cookies
driver.execute_script('''return document.querySelector('div#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')''').click()

Class attribute in the html element can contain multiple classes separated by space. i.e. "sc-gsDKAQ fILFKg", contains two classes, sc-gsDKAQ and fILFKg.
You can user either but both are random and can be changed next time css is recompiled. I recommend to think of xpath using data-testid attribute

How to click Captcha "click to verify" button with selenium

I'm trying to use XPath to click a "click to verify" captcha button and it is not working. It returns the error message:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="1e505deed3832c02c96ca5abe70df9ab"]/div"}
Here's my code:
click_to_verify = wd.find_element_by_xpath('//*[#id="1e505deed3832c02c96ca5abe70df9ab"]/div')
click_to_verify.click()
I'm not worried yet about the captcha that will appear when I click the button. I just want to be able to click the button. Any suggestions?

I would probably say that the captcha you want to click is in iframe. It is very common design to put captcha in an iframe. If it's in iframe you would need to switch the focus of web driver like this :
wait = WebDriverWait(wd, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "iframe xpath")))
click_to_verify = wd.find_element_by_xpath('//*[#id="1e505deed3832c02c96ca5abe70df9ab"]/div')
click_to_verify.click()
and I would not recommend you to use 1e505deed3832c02c96ca5abe70df9ab in xpath, since it may change every time you execute your script .
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

So in the end I just decided to install pyautogui and use it to move the mouse to the specific part of the screen that needed to be clicked. Maybe not the most elegant solution but it worked for me in this situation.
import pyautogui
pyautogui.click(600, 525)

How can I "click" download button on selenium in python

I want to download user data on Google analytics by using crawler so I write some code using selenium. However, I cannot click the "export" button. It always shows the error "no such element". I tried to use find_element_by_xpath, by_name and by_id.
I upload inspect of GA page below.
I TRIED:
driver.find_element_by_xpath("//*[#class='download-link']").click()
driver.find_element_by_xpath('//*[#id="ID-activity-userActivityTable"]/div/div[2]/span[6]/button')
driver.find_element_by_xpath('//*[#class='_GAD.W_DECORATE_ELEMENT.C_USER_ACTIVITY_TABLE_CONTROL_ITEM_DOWNLOAD']')
Python Code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('/Users/parkjunhong/Downloads/chromedriver')
driver.implicitly_wait(3)
usrid = '1021'
url = 'https://analytics.google.com/analytics/web/#/report/app-visitors-user-activity/a113876882w169675624p197020837/_u.date00=20190703&_u.date01=20190906&_r.userId='+usrid+'&_r.userListReportStates=%3F_u.date00=20190703%2526_u.date01=20190906%2526explorer-
table.plotKeys=%5B%5D%2526explorer-table.rowStart=0%2526explorer-
table.rowCount=1000&_r.userListReportId=app-visitors-user-id'
driver.get(url)
driver.find_element_by_name('identifier').send_keys('ID')
idlogin = driver.find_element_by_xpath('//*[#id="identifierNext"]/span/span')
idlogin.click()
driver.find_element_by_name('password').send_keys('PASSWD')
element = driver.find_element_by_id('passwordNext')
driver.execute_script("arguments[0].click();", element)
#login
driver.find_element_by_xpath("//*[#class='download-link']").click()
#click the download button
ERROR:
Message: no such element: Unable to locate element
inspection of GA

your click element is in an iFrame (iFrame id="galaxyIframe" ...). Therefore, you need to tell the driver to switch from the "main" page to said iFrame. If you add this line of code after your #login it should work:
driver.switch_to.frame(galaxyIframe)
(If the frame did not have a name, you would use: iframe = driver.find_element_by_xpath("xpath-to-frame") and then driver.switch_to.frame(iframe)
To get back to your default frame, use:
driver.switch_to.default_content()
Crawling GA is generally a pain. Not just because you have these iFrames everywhere.
Apart from that, I would recommend looking into puppeteer, the new kid on the crawler block. Even though the prospect of switching to javascript from python may be daunting, it is worth it! Once you get into it, selenium will have felt super clunky.

You can try with the text:
If you want to click on 'Export'-
//button[contains(text(),'Export')]

How to click on the element with text as LOGIN as per the website through Selenium and Python?

Windows 10Home x64 /
Python 3.7.0 win64 /
Selenium 3.14.0 /
Geckodriver 3.14.0 /
Firefox 61.0.2.
I'm trying to click on the login button, even though I have copied and pasted the full XPath I am not able to click the Login button.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://library.yonsei.ac.kr/')
linkElem = driver.find_element_by_link_text('LOGIN').click()
username = driver.find_element_by_id('id')
username.send_keys('myuserAbc')
password = driver.find_element_by_id('password')
password.send_keys('mypass123')
linkElem = driver.find_element_by_xpath("/html/body/div[2]/div[2]/div/div[2]/form/fieldset/div[2]/p[#class='loginBtn']/input[#type='submit']").click()
Thank you for your help.
Update1: The script works up to the point of localizing the login button, I don't have problems with Marionette or having the latest Firefox version.
Update2: I added p[#class='loginBtn']/input[#type='submit'] in an effort to make it more specific but still doesn't work.

In my experience i have found that the button click might not work some times on form buttons with type set to action. The best possible way to work through this one will be to use the submit() method.
so something like this should work on your login form
from time import sleep
time.sleep(20)
driver.find_element_by_class_name('loginBtn').submit()
This will submit the form to the server and you will be able to go through the login process.
if that doesn't work you could also try the follwoing
driver.find_element_by_class_name('loginBtn).send_keys(Keys.ENTER)
Now, why Button.click(); not working here could have following
reasons.
1.The button is visible but not enabled.
2.Driver is finding the 2 instances of Button element
.More discussion about this can be found here
Selenium: submit() works fine, but click() does not

As per the HTML you have shared to invoke click() on the element with text as LOGIN you need to induce WebDriverWait for the desired element to be clickable and you can use either of the following solutions:
CSS_SELECTOR:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[title='Login']>span"))).click()
XPATH:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Login']/span"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

python selenium click on button

I am quite new to python selenium and I am trying to click on a button which has the following html structure:
<div class="b_div">
<div class="button c_button s_button" onclick="submitForm('mTF')">
<input class="very_small" type="button"></input>
<div class="s_image"></div>
<span>
Search
</span>
</div>
<div class="button c_button s_button" onclick="submitForm('rMTF')" style="margin-bottom: 30px;">
<input class="v_small" type="button"></input>
<span>
Reset
</span>
</div>
</div>
I would like to be able to click both the Search and Reset buttons above (obviously individually).
I have tried a couple of things, for example:
driver.find_element_by_css_selector('.button .c_button .s_button').click()
or,
driver.find_element_by_name('s_image').click()
or,
driver.find_element_by_class_name('s_image').click()
but, I seem to always end up with NoSuchElementException, for example:
selenium.common.exceptions.NoSuchElementException: Message: u'Unable to locate element: {"method":"name","selector":"s_image"}' ;
I am wondering if I can somehow use the onclick attributes of the HTML to make selenium click?
Any thoughts which can point me in the right direction would be great.
Thanks.

Remove space between classes in css selector:
driver.find_element_by_css_selector('.button .c_button .s_button').click()
# ^ ^
=>
driver.find_element_by_css_selector('.button.c_button.s_button').click()

try this:
download firefox, add the plugin "firebug" and "firepath"; after install them go to your webpage, start firebug and find the xpath of the element, it unique in the page so you can't make any mistake.
See picture:
browser.find_element_by_xpath('just copy and paste the Xpath').click()

For python, use the
from selenium.webdriver import ActionChains
and
ActionChains(browser).click(element).perform()

open a website https://adviserinfo.sec.gov/compilation and click on button to download the file and even i want to close the pop up if it comes using python selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.chrome.options import Options
#For Mac - If you use windows change the chromedriver location
chrome_path = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(chrome_path)
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-popup-blocking")
driver.maximize_window()
driver.get("https://adviserinfo.sec.gov/compilation")
# driver.get("https://adviserinfo.sec.gov/")
# tabName = driver.find_element_by_link_text("Investment Adviser Data")
# tabName.click()
time.sleep(3)
# report1 = driver.find_element_by_xpath("//div[#class='compilation-container ng-scope layout-column flex']//div[1]//div[1]//div[1]//div[2]//button[1]")
report1 = driver.find_element_by_xpath("//button[#analytics-label='IAPD - SEC Investment Adviser Report (GZIP)']")
# print(report1)
report1.click()
time.sleep(5)
driver.close()

I had the same problem using Phantomjs as browser, so I solved in the following way:
driver.find_element_by_css_selector('div.button.c_button.s_button').click()
Essentially I have added the name of the DIV tag into the quote.

The following debugging process helped me solve a similar issue.
with open("output_init.txt", "w") as text_file:
text_file.write(driver.page_source.encode('ascii','ignore'))
xpath1 = "the xpath of the link you want to click on"
destination_page_link = driver.find_element_by_xpath(xpath1)
destination_page_link.click()
with open("output_dest.txt", "w") as text_file:
text_file.write(driver.page_source.encode('ascii','ignore'))
You should then have two textfiles with the initial page you were on ('output_init.txt') and the page you were forwarded to after clicking the button ('output_dest.txt'). If they're the same, then yup, your code did not work. If they aren't, then your code worked, but you have another issue.
The issue for me seemed to be that the necessary javascript that transformed the content to produce my hook was not yet executed.
Your options as I see it:
Have the driver execute the javascript and then call your find
element code. Look for more detailed answers on this on
stackoverflow, as I didn't follow this approach.
Just find a comparable hook on the 'output_dest.txt' that will produce the same result, which is what I did.
Try waiting a bit before clicking anything:
xpath2 = "your xpath that you are going to click on"
WebDriverWait(driver, timeout=5).until(lambda x:
x.find_element_by_xpath(xpath2))
The xpath approach isn't necessarily better, I just prefer it, you can also use your selector approach.

I had the same problem and with Firefox, I got button element with the following steps:
right click button of interest and select "Inspect Accessibility Properties"
this opens the inspector. Right click the highlighted line and click "Print to JSON"
this opens a new tab. Look for nodeCssSelector and copy the value
This allowed me to accept cookies of the website Yahoo by using.
url = "https://yahoo.com"
driver = Firefox(executable_path="geckodriver.exe")
driver.get(url)
driver.find_element_by_css_selector("button.btn:nth-child(5)").click()
I tested this further and it allowed me to accept individual cookies with ease. Simply repeat the mentioned steps from before to get the button names.
url = "https://yahoo.com"
driver = Firefox(executable_path="geckodriver.exe")
driver.get(url)
driver.find_element_by_css_selector("a.btn").click()
driver.find_element_by_css_selector(".firstPartyAds > div:nth-child(2) > label:nth-child(1)").click()
driver.find_element_by_css_selector(".preciseGeolocation > div:nth-child(2) > label:nth-child(1)").click()
driver.find_element_by_css_selector("button.btn").click()
Another method is to
right click button of interest and select "Inspect"
right click the highlighted line and click "Copy -> CSS Selector" or whatever you need (there are multiple options, including XPath)
However, I think the second method may include whitespaces depending on what you copy, so you might need to manually remove (some of) them. The first method seems to be more foolproof, but I don't know if/how it works on other browsers than Firefox. The second method should work for all browsers.

Use This code To Click On Button
# finding the button using ID
button = driver.find_element_by_id(ID)
# clicking on the button
button.click()

e = driver.find_element(By.XPATH, 's_image').click()
sometime it does not work!
you can try:
e = driver.find_element(By.XPATH, 's_image') driver.execute_script("arguments[0].click();", e)