I have written this code using selenium python so parse a webpage(dynamically loading) so
that it goes till the end and then stop when "load more" button is not clickable anymore.
wait=WebDriverWait(driver,50)
wait.until(EC.element_to_be_clickable((By.ID, 'bottomPager'))) while
EC.element_to_be_clickable((By.ID,'bottomPager')):
driver.find_element_by_xpath('.//div[#id="bottomPager"]').click()
if not driver.find_element_by_id('bottomPager').is_enabled():
break
wait.until(EC.element_to_be_clickable((By.ID,'bottomPager')))
but I am getting the error when it reaches the end of the page
Traceback (most recent call last): File "python_org_search2.py", line 24, in <module>
driver.find_element_by_xpath('.//div[#id="bottomPager"]').click() File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 60, in click
self._execute(Command.CLICK_ELEMENT) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 370, in _execute
return self._parent.execute(command, params) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 166, in execute
self.error_handler.check_response(response) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: u'unknown error: Element is not clickable at point (643, 628)\n (Session info: chrome=34.0.1847.137)\n (Driver info: chromedriver=2.9.248304,platform=Linux 3.13.0-24-generic x86_64)'
What are the changes required in code so as to rectify it ?
In my experience, this exception only occurs when using the Chrome driver with Selenium. Can you use Firefox instead? (I believe Firefox is the primary intended browser for webdriver invocations via Selenium.)
driver = webdriver.Firefox()
That should clear up this exception.
(You can get portable versions of Firefox at this site.)
Related
Here is the code which I use to open the new tab using Selenium Python,
self.driver.execute_script("window.open(arguments[0])", self.listing_url)
self.driver.switch_to.window(self.driver.window_handles[1])
time.sleep(10)
and it works perfectly. But I am planning to use WebDriverWait instead of time.sleep
self.driver.execute_script("window.open(arguments[0])", self.listing_url)
self.driver.switch_to.window(self.driver.window_handles[1])
WebDriverWait(self.driver, 20).until(EC.presence_of_element_located((By.XPATH, '//div[#class="honbun"]')))
and when I am using it, it's throwing an error. Here is the error log:
Traceback (most recent call last):
File "D:\tender_scraper\engine\Spider.py", line 308, in parse_index_page
index_handle = self.load_listing_page(listing, i, index_listing)
File "D:\tender_scraper\spiders\gunmapref_listing_spider.py", line 298, in load_listing_page
WebDriverWait(self.driver, 20).until(EC.presence_of_element_located((By.XPATH, '//div[#class="honbun"]')))
File "D:\scraper\lib\site-packages\selenium\webdriver\support\wait.py", line 78, in until
value = method(self._driver)
File "D:\scraper\lib\site-packages\selenium\webdriver\support\expected_conditions.py", line 64, in _predicate
return driver.find_element(*locator)
File "D:\scraper\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1251, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "D:\scraper\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 430, in execute
self.error_handler.check_response(response)
File "D:\scraper\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: missing or invalid columnNumber
(Session info: headless chrome=104.0.5112.48)
Any idea what's wrong with it & how to solve it? Previously, I used chrome 103 and changed it to 104 but still got the same error. It seems the error only occurs when changing into the new tab.
I'm trying to make a script that logs me into my account automatically and I'm stuck at send_keys.
I have the latest version of selenium, I'm using python 2.7.18.
here is my script:
from selenium import webdriver
from time import sleep
username = 'username'
driver = webdriver.Chrome()
driver.get('https://www.instagram.com/')
sleep(4)
email = '//*[#id="react-root"]/section/main/article/div[2]/div[1]/div/form/div[2]/div/label/input'
login = driver.find_element_by_xpath(email).send_keys(username)
The error is:
Traceback (most recent call last):
File "C:\Users\Jameel\Desktop\selenium1.py", line 14, in <module>
login = driver.find_element_by_xpath(email).send_keys(username)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 479, in send_keys
'value': keys_to_typing(value)})
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 633, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 321, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py",
line 242, in check_response
raise exception_class(message, screen, stacktrace) WebDriverException: Message: unknown error: call function result
missing 'value' (Session info: chrome=83.0.4103.61) (Driver info:
chromedriver=2.31.488763
(092de99f48a300323ecf8c2a4e2e7cab51de5ba8),platform=Windows NT
10.0.17763 x86_64)
I know the script isn't complete.
Generally different sites follow different design approaches, which doesn't allow normal script flow & needs additional steps to be added.
In you case, the default CSS style blocks your input. So, first click on that element( Which makes it as focused) then send your keys.
Element Attribute differences below,
Before click action
After click action
I followed this tutorial for Selenium with Python:
So I did the same as the tutorial:
from selenium import webdriver
driver=webdriver.Chrome()
driver.get('https://www.youtube.com/')
searchbox = driver.find_element_by_xpath('//*[#id="search-input"]')
searchbox.send_keys('cooking')
But I get this error:
Traceback (most recent call last):
File "C:/ANGUS/DATA_ANALYSIS/WEBSCRAPING/TextXXX.py", line 7, in <module>
searchbox.send_keys('cooking')
File "C:\Users\Angus\PycharmProjects\Test\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 477, in send_keys
self._execute(Command.SEND_KEYS_TO_ELEMENT,
File "C:\Users\Angus\PycharmProjects\Test\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "C:\Users\Angus\PycharmProjects\Test\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Angus\PycharmProjects\Test\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
(Session info: chrome=81.0.4044.138)
How can I fix this?
Another way to do that in order not to use the search is by inserting the query itself.
Code:
your_query = "cooking"
driver.get("https://www.youtube.com/results?search_query=" + str(your_query))
time.sleep(5)
In this way, it is like you are going immediately in the output that you want after searching a specific query of yours. So, let's say that instead of typing cooking, you can .get() to the URL you want by adding your query into this URL.
Using chrome 76.0.3809.132 with chromedriver 76.0.3809.126 the testing browser opens but throws a traceback on the driver.get. It isn't able to fetch the page.
Tried couple of arguments, and tried looking for an older version of Chrome with an older chrome driver, which seems impossible.
from selenium import webdriver
driver = webdriver.Chrome("C:/chromedriver/chromedriver.exe")
driver.get("www.google.com")
Output:
DevTools listening on ws://127.0.0.1:57577/devtools/browser/7494ca24-60fb-4b14-9aed-4f25232b94bd
Traceback (most recent call last):
File "slidingwindow.py", line 5, in <module>
driver.get("www.google.com")
File "C:\projects\python\harvester\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\projects\python\harvester\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\projects\python\harvester\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument (Session info: chrome=76.0.3809.132)
Try to insert https://:
from selenium import webdriver
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://www.google.com")
I am running through the Splinter tutorial located here: http://splinter.readthedocs.io/en/latest/tutorial.html#
The code I am using:
from splinter import Browser
browser = Browser()
browser.visit('http://google.com')
browser.fill('q', 'chicago pizza')
button = browser.find_by_name('btnG')
button.click()
Every time I try to get my code to click the search I get the following error:
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
button.click()
File "C:\Python27\lib\site-packages\splinter\driver\webdriver\__init__.py", line 546, in click
self._element.click()
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py", line 77, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
ElementNotInteractableException: Message:
I have used browser.find_by_id, browser.find_by_tag, browser.find_by_text and none have worked, all giving me the same error.
I am running python 2.7.8 Firefox v 54.0.1 (32-bit) and have selenium installed.
Any idea of how to fix this? I think it may have to do with my geckodriver.
I would like to solve the problem and not have to work around it by say switching web browsers.
Everything worked fine except you have entered wrong name of button.
Here is the working code
from splinter import Browser
browser = Browser()
browser.visit('http://google.com')
browser.fill('q', 'chicago pizza')
button = browser.find_by_name('btnK')
button.click()