Develop Reference

How to tell google that I accept cookies python

I am trying to accept cookies and search in google. But I am facing a problem that I never faced before.
Webdriver cant find the accept cookies button element. I looked everything. I tried to see if something changes the xpath(Is there any thing triggers any other element so the xpath will change) but I have zero in my hand. I tried using accept.send_keys(Keys.RETURN) and accept.click(). Nothing seems to work.
What I have for now;
def GoogleIt():
path = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(path)
driver.get("https://google.com")
wait(5)
accept = driver.find_element_by_xpath("/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/div[2]")
accept.send_keys(Keys.RETURN)
accept.click()
search = driver.find_element_by_xpath("/html/body/div/div[2]/form/div[2]/div[1]/div[1]/div/div[2]/input")
search.click()
search.send_keys(talk)
Note: wait() is a different name of time.sleep()
Error;
Traceback (most recent call last):
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\tkinter\__init__.py", line 1883, in __call__
return self.func(*args)
File "C:/Users/Teknoloji/Desktop/Phyton/Assistant V1/Assistant.py", line 20, in GoogleIt
accept = driver.find_element_by_xpath("/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/div[2]")
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/div[2]"}
(Session info: chrome=85.0.4183.121)
Process finished with exit code -1
If you need more information I am here.
Thanks...

To click on accept cookies button which is inside an iframe you need to switch to iframe first.
Induce WebDriverWait() and frame_to_be_available_and_switch_to_it() and following css selector.
Induce WebDriverWait() and element_to_be_clickable() and following xpath.
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src^='https://consent.google.com']")))
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[#id='introAgreeButton']"))).click()
You need to import following libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

In case KunduK's solution doesn't work for some of those reading this, use "By.ID" parameter instead of the "By.CSS_SELECTOR" in the frame_to_be_available_and_switch_to_it() method -
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "name_of_the_iframe")))

After the latest update of Selenium, the above solutions did not work for me.
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium import webdriver
from selenium.webdriver.common.by import By
firefox_binary = FirefoxBinary()
browser = webdriver.Firefox(firefox_binary=firefox_binary)
Let's say you want to get this page from Google:
browser.get('https://www.google.com/search?q=cats&source=lnms&tbm=isch')
To continue, you have to click on "Accept all"
Then, you use Selenium to click the button:
browser.find_element(By.XPATH,"//.[#aria-label='Accept all']").click()
Then, you can continue to Google page!

How to click on a text box and type in text using selenium in Python?

I am attempting to try and type into a login page using Selenium to go to the page and click on the login box and type in some text. My code allows me to go to the login page, click the login box, and this is the part where my code breaks.
Code
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe')
driver.get("https://accounts.google.com/signin/v2/identifier?passive=1209600&continue=https%3A%2F%2Fdocs.google.com%2F&followup=https%3A%2F%2Fdocs.google.com%2F&emr=1&flowName=GlifWebSignIn&flowEntry=ServiceLogin")
text_area= driver.find_element_by_xpath('//*[#id="view_container"]/div/div/div[2]/div/div[1]/div/form/span/section/div/div/div[1]/div/div[1]/div/div[2]')
text_area.click()
text_area.send_keys(email_address)
The code opens the page, which in case anyone is wondering or if this might be what is affecting my code is the login page when you go to Google Docs on a guest account, clicks on the login text box, and fails to type any text. At the point in time that the code should be typing in text, I get this error.
Error
Traceback (most recent call last):
File "file.py", line 8, in <module>
text_area.click()
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <div class="i9lrp mIZh1c"></div> is not clickable at point (508, 242). Other element would receive the click: <input type="email" class="whsOnd zHQkBf" jsname="YPqjbf" autocomplete="username" spellcheck="false" tabindex="0" aria-label="Email or phone" name="identifier" value="" autocapitalize="none" id="identifierId" dir="ltr" data-initial-dir="ltr" data-initial-value="">
(Session info: chrome=81.0.4044.92)
Admittedly this is a bit over my head and I am wondering if anyone might know how to fix this error, and how they came to find out how to do so since my next reasonable steps would be to type in the password, select new document, and type text into the Google Doc.

You should try to set the value in input but instead your current code is getting div element and inputbox is intercepting the click on the div element. Instead please, try with the below line of code.
text_area= driver.find_element_by_xpath("//input[#name='identifier']")

You can use waits, i.e.:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path="C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe")
driver.maximize_window()
driver.get("https://accounts.google.com/signin/v2/identifier?passive=1209600&continue=https%3A%2F%2Fdocs.google.com%2F&followup=https%3A%2F%2Fdocs.google.com%2F&emr=1&flowName=GlifWebSignIn&flowEntry=ServiceLogin")
wait = WebDriverWait(driver, 10)
# usr
el = wait.until(EC.visibility_of_element_located((By.ID, "identifierId")))
el.send_keys("username#gmail.com")
el.send_keys(Keys.ENTER)
# pwd
el = wait.until(EC.visibility_of_element_located((By.XPATH, "//input[#type='password']")))
el.send_keys("XXXXX")
el.send_keys(Keys.ENTER)
# you're logged

My scraper throws an error clicking on the first link out of many

I've written a script in python in combination with selenium to click on a tags under class domino-viewentry in a webpage. My script is able to click on the first a tag. However, instead of going for the next ones to click, it throws an error. My script and errors are given below. I hope somebody will take a look and provide me with any solution to make it right.
Link to the webpage: webpage_link
My script:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("above_link")
for item in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".domino-viewentry a"))):
item.click()
wait.until(EC.staleness_of(item))
driver.quit()
The error I'm having:
line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\ar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 501, in _execute
return self._parent.execute(command, params)
File "C:\Users\ar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 308, in execute
self.error_handler.check_response(response)
File "C:\Users\ar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=63.0.3239.84)
(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 6.1.7601 SP1 x86)

If you just want to click all the links on page to expand embedded data you can use below code
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("http://www.siicex-caaarem.org.mx/Bases/TIGIE2007.nsf/4caa80bd19d9258006256b050078593c/")
while True:
try:
link = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[img[#src='/icons/expand.gif'] and not(./following::img[#src='/icons/collapse.gif'])]")))
link.click()
wait.until(EC.staleness_of(link))
except:
break
But note that page is "not scalable": once you reached 01 > 02 > 0207 all following links just disappears, so you cannot continue... Let me know if you need to implement more complex logic
Update
If you want to click only "first level" links you can try:
links_length = len(wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//td[#colspan='9']//a[img]"))))
for index in range(links_length):
driver.find_elements_by_xpath("//td[#colspan='9']//a[img]")[index].click()

So when you click the first link, you go to a different page, and references to the other elements in the first page are no longer valid, unfortunately. You would have to re-query the DOM to find the elements every time you load the page. See also: http://www.seleniumhq.org/exceptions/stale_element_reference.jsp
For your purposes, you might be better off grabbing all the href's, storing those, and visiting them one by one to do your scraping.

Replace placeholder text on a webpage using python and selenium?

The element where I want to replace text:
<input tabindex="1" style="padding-left:119px!important;width:318px!important;background-color:#fff;outline: none;box-sizing: inherit;" id="email" name="mobile" placeholder="Enter your Mobile Number" class="un_s un1_s" value="" onblur="remove_border();" type="text" maxlength="20">
I have tried various ways(all I could think up and google), to replace the placeholder text. Lots of errors later, I thought since the focus is already in the text box(where I want to enter the text may be just send_keys would work. It didn't. Can someone help me out and explain the concept or point to where I might read on where I went wrong?
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = r"C:\Users\-------\Desktop\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("SAMPLE WEBSITE URL") #sorry had to remove the link :(
driver.maximize_window()
action = webdriver.ActionChains(driver)
elm = driver.find_element_by_id("user_sign_in").click()
inputElement = driver.find_element_by_xpath('//*[#id="lfm"]/div[1]/div[2]') #driver.find_elements_by_xpath("//*[contains(text(), 'Enter your Mobile Number')]")
#driver.find_element_by_id("mobile")
inputElement.send_keys('1234567890')
inputElement.submit()
#//*[#id="lfm"]/div[1]/div[2] xpath id for the mobile number element
#code below this is not working, for move mouse
#action = webdriver.ActionChains(driver)
#action.move_to_element((By.XPATH, '//*[#id="user_sign_in"]')).perform()
#For moving the mouse to sign in: Tried the ones below and they didn't work either
#driver.move_to_element(By.XPATH, '//*[#id="user_sign_in"]')
#login_menu = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//*[#id="user_sign_in"]')))
#ActionChains(driver).move_to_element(sign_in).perform()
Errors:
Traceback (most recent call last):
File "C:/Python34/Selenium 2nd Trial.py", line 16, in
inputElement.send_keys('9810307369')
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webelement.py", line 322, in send_keys
self._execute(Command.SEND_KEYS_TO_ELEMENT, {'value': keys_to_typing(value)})
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webelement.py", line 457, in _execute
return self._parent.execute(command, params)
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 233, in execute
self.error_handler.check_response(response)
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot focus element
(Session info: chrome=55.0.2883.87)
(Driver info: chromedriver=2.27.440174 (e97a722caafc2d3a8b807ee115bfb307f7d2cfd9),platform=Windows NT 10.0.14393 x86_64)

You are trying to send keys to the div element, but meant to use an input element:
inputElement = driver.find_element_by_id('email')
inputElement.send_keys('1234567890')
inputElement.submit()
And, be aware that the page might take some time to load and there could be visual effects preventing an element to be located or interactable. If this is the case, use WebDriverWait with an appropriate Expected Condition to make sure element is visible.

Selenium -python element not clickable

I am trying to browse facebook via selenium in python.
Here is my script so far.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
usr = ""# I have put 3 different accounts and tested it. Same error
pwd = ""
driver = webdriver.Chrome('E:\python_libs\chromedriver.exe')
driver.get("https://www.facebook.com")
assert "Facebook" in driver.title
elem = driver.find_element_by_id("email")
elem.send_keys(usr)
elem = driver.find_element_by_id("pass")
elem.send_keys(pwd)
elem.send_keys(Keys.RETURN)
elem = driver.find_element_by_css_selector(".input.textInput")
elem.send_keys("Posted using Python's Selenium WebDriver bindings!")
elem = driver.find_element_by_css_selector(".selected")
elem.click()
time.sleep(5)
driver.close()
This script when run on windows with chromedriver obtained and correctly installed returns an error.
Traceback (most recent call last):
File "C:\Users\Home\Desktop\facebook_post.py", line 28, in <module>
elem.click()
File "C:\Python27\lib\site-packages\selenium-2.42.0- py2.7.egg\selenium\webdriver\remote\webelement.py", line 60, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium-2.42.0- py2.7.egg\selenium\webdriver\remote\webelement.py", line 370, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium-2.42.0-py2.7.egg\selenium\webdriver\remote\webdriver.py", line 172, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium-2.42.0-py2.7.egg\selenium\webdriver\remote\errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'unknown error: Element is not clickable at point (481, 185). Other element would receive the click: <input type="file" class="_n _5f0v" title="Choose a file to upload" accept="image/*" name="file" id="js_0">\n (Session info: chrome=35.0.1916.114)\n (Driver info: chromedriver=2.10.267521,platform=Windows NT 6.1 x86)'
I am unable to make head or tail of this.
Any help would be appreciated.
The Graph API is not feasible here as I want to browse as myself and not as some application.
If however browsing as myself can be done by graph API or some other means, please do tell.

If you're doing this process several times, Selenium has some options in WebDriverWait to not waste too much time and check to see if elements are visible.
Here is the documentation on Waits in Selenium and
this is a section of the Python documentation of Selenium that talks about the Expected Conditions classes. It seems that "clickable" is one conditions that you can check for.
In my case, I wrote a simple function that took care of visibility and clicking and called it every time I needed to click on something dynamic.
My example code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Browser = webdriver.Chrome()
def wait_until_visible_then_click(element):
element = WebDriverWait(Browser,5,poll_frequency=.2).until(
EC.visibility_of(element))
element.click()
EDIT:
The links above seem to be nerfed. This is the new documentation on waits and here are the expected conditions docs.