I'm trying to create a content scraper to gather a list of medical terms and I'm using https://www.merriam-webster.com/browse/medical for that task. All I have in my code right now is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("https://www.merriam-webster.com/browse/medical/a")
driver.close()
The following error message is shown after every run of the program using this or any https://www.merriam-webster.com URL:
Traceback (most recent call last):
File "mw-download.py", line 5, in <module>
driver.get("https://www.merriam-webster.com/browse/medical/a")
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 268, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=https%3A//nop.xpanama.net/if.html%3Fadflag%3D1%26cb%3DKHg6C73w9F&c=UTF-8&f=regular&d=Firefox%20can%E2%80%99t%20find%20the%20server%20at%20nop.xpanama.net.
I have tried changing the URL to different sites, with and without https support to test if it was something with https, but I haven't encountered this error with any other site. I also tried removing the driver.close() command at the end of the script to see if trying to close the driver before the page contents were loaded is what was causing the problem but the problem persists even without that line of code.
Any help in understanding this error and fixing it would be greatly appreciated!
Related
I had recently started learning automation with python using selenium. I had installed chromedriver but when i run the code chrome displays no output. My code:
from selenium import webdriver
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("www.wikipedia.com")
Output
Output :
C:\Users\Admin\PycharmProjects\untitled\venv\Scripts\python.exe C:/Users/Admin/PycharmProjects/untitled/1.py
Traceback (most recent call last):
File "C:/Users/Admin/PycharmProjects/untitled/1.py", line 5, in <module>
driver.get("www.wikipedia.com")
File "C:\Users\Admin\PycharmProjects\untitled\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\Users\Admin\PycharmProjects\untitled\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Admin\PycharmProjects\untitled\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument
(Session info: chrome=88.0.4324.96)
Please help as I'm unable to proceed.
Thanks!
URL should contain scheme. Simply replace
"www.wikipedia.com"
with
"https://www.wikipedia.com"
Using chrome 76.0.3809.132 with chromedriver 76.0.3809.126 the testing browser opens but throws a traceback on the driver.get. It isn't able to fetch the page.
Tried couple of arguments, and tried looking for an older version of Chrome with an older chrome driver, which seems impossible.
from selenium import webdriver
driver = webdriver.Chrome("C:/chromedriver/chromedriver.exe")
driver.get("www.google.com")
Output:
DevTools listening on ws://127.0.0.1:57577/devtools/browser/7494ca24-60fb-4b14-9aed-4f25232b94bd
Traceback (most recent call last):
File "slidingwindow.py", line 5, in <module>
driver.get("www.google.com")
File "C:\projects\python\harvester\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\projects\python\harvester\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\projects\python\harvester\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument (Session info: chrome=76.0.3809.132)
Try to insert https://:
from selenium import webdriver
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://www.google.com")
This is my first web scraping project and I'm using selenium webdriver with Python in order to dynamically generate some csv files after choosing a few options on a website (though I'm not there yet).
However, I'm facing an unexpected timeout when the execution reaches a button click(). The click is performed but it gets stuck in there and does not continue the execution till the timeout.
Any clues on how to solve that?
Thanks!!
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
driver = webdriver.Firefox()
driver.get('http://www8.receita.fazenda.gov.br/SimplesNacional/Aplicacoes/ATBHE/estatisticasSinac.app/Default.aspx')
driver.find_element_by_id('ctl00_ctl00_Conteudo_AntesTabela_lnkOptantesPorCNAE').click()
Select(driver.find_element_by_id("ctl00_ctl00_Conteudo_AntesTabela_ddlColuna")).select_by_visible_text("Município")
filtro_uf = driver.find_element_by_id('ctl00_ctl00_Conteudo_AntesTabela_btnFiltros')
for i in range (1, 28):
filtro_uf.click()
uf = Select(driver.find_element_by_id("ctl00_ctl00_Conteudo_AposTabela_ddlUf"))
uf.options[i].click()
time.sleep(2)
driver.find_element_by_id('chkTodosMunicipios').click()
time.sleep(2)
driver.find_element_by_xpath("//*[contains(text(),'Ok')]").click()
time.sleep(2)
# Here is where my code get stuck and gets a timeout
driver.find_element_by_id('ctl00_ctl00_Conteudo_AntesTabela_btnExibir').click()
The error I get:
Traceback (most recent call last):
File "/home/hissashi/Desktop/Python3/WS_SINAC/download_SINAC.py", line 22, in <module> driver.find_element_by_id('ctl00_ctl00_Conteudo_AntesTabela_btnExibir').click()
File "/home/hissashi/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/hissashi/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 501, in _execute
return self._parent.execute(command, params)
File "/home/hissashi/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 308, in execute
self.error_handler.check_response(response)
File "/home/hissashi/.local/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
**selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 300000ms**
I've found a workaround for the problem.
Apparently, the click() function blocks the code until the page is "completely" loaded. However, for some reason, the page keeps loading forever (without anything else to load) and it holds my code till it reaches the timeout limit.
Instead of using click, I've changed it to key ENTER and the page still keeps loading forever but it doesn't hold the code anymore.
#FROM CLICK
driver.find_element_by_id('ctl00_ctl00_Conteudo_AntesTabela_btnExibir').click()
#TO SENDING ENTER (ue007)
driver.find_element_by_id('ctl00_ctl00_Conteudo_AntesTabela_btnExibir').send_keys(u'\ue007')
I'm working on a script to pull some information from a site that I must login to use. I'm using Python 2.7.12 and Selenium 3.4.3.
#!/usr/bin/python
from selenium import webdriver
browser = webdriver.Firefox(firefox_binary='/usr/bin/firefox', executable_path="./geckodriver")
# Get to the login page
browser.get('https://example.com')
browser.find_element_by_link_text('Application').click()
# Login
browser.find_element_by_id('username').send_keys('notmyusername')
browser.find_element_by_id('password').send_keys('notmypassword')
browser.find_element_by_css_selector('.btn').click()
# Open the application
browser.find_element_by_id('ctl00_Banner1_ModuleList_ctl01_lnkModule').click()
If I copy this code and paste it into the python console, it runs just fine and goes to the page I want. However, when I run the script from the terminal (bash on Linux Mint 18), it errors out. Here's the output with the try and catch statements removed:
Traceback (most recent call last):
File "./script.py", line 14, in <module>
browser.find_element_by_id('ctl00_Banner1_ModuleList_ctl01_lnkModule').click()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 289, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 791, in find_element
'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="ctl00_Banner1_ModuleList_ctl01_lnkModule"]
I don't even know how to go about troubleshooting this. Any help?
What is most probably happening is that when you run the script from bash, the script runs too quickly and the get_by_id operation is started before the browser has finished loading the page, which results in this error.
As #murali-selenium suggested, you should probably add some wait time before starting to look for stuff in the document.
That can be achived this way:
#!/usr/bin/python
from selenium import webdriver
import time
wait_time_sec = 1
browser = webdriver.Firefox(firefox_binary='/usr/bin/firefox', executable_path="./geckodriver")
# Get to the login page
browser.get('https://example.com')
time.sleep(wait_time_sec)
browser.find_element_by_link_text('Application').click()
time.sleep(wait_time_sec)
# Login
browser.find_element_by_id('username').send_keys('notmyusername')
browser.find_element_by_id('password').send_keys('notmypassword')
browser.find_element_by_css_selector('.btn').click()
time.sleep(wait_time_sec)
# Open the application
try:
browser.find_element_by_id('ctl00_Banner1_ModuleList_ctl01_lnkModule').click()
except:
print('failed')
#browser.stop()
Per selenium docs, setting browser.implicity_wait(10) # seconds tells the browser to poll the page for 10 seconds before deciding the element isn't there. By default it is set to 0 seconds. Once it is set, the setting persists for the life of the webdriver object.
There are a lot of other nifty tools to wait for elements to load, documented at readthedocs.io.
I just installed Splinter, input simple code like below but have error. Can someone help me on this? Thanks.
from splinter import Browser
b=Browser()
url = "http://www.google.com"
b.visit(url)
Error messages:
Traceback (most recent call last): File "", line 1, in
File
"/usr/local/lib/python2.7/dist-packages/splinter/driver/webdriver/init.py",
line 184, in visit
self.driver.get(url) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py",
line 245, in get
self.execute(Command.GET, {'url': url}) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py",
line 233, in execute
self.error_handler.check_response(response) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py",
line 194, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: Target URL
www.google.ca is not well-formed. Stacktrace:
at FirefoxDriver.prototype.get (file:///tmp/tmpsIlz0H/extensions/fxdriver#googlecode.com/components/driver-component.js:10636)
at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpsIlz0H/extensions/fxdriver#googlecode.com/components/command-processor.js:12661)
at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpsIlz0H/extensions/fxdriver#googlecode.com/components/command-processor.js:12666)
at DelayedCommand.prototype.execute/< (file:///tmp/tmpsIlz0H/extensions/fxdriver#googlecode.com/components/command-processor.js:12608)
Try this instead:
from splinter import Browser
with Browser() as b:
b.visit("http://www.google.com")
Docs here: https://splinter.readthedocs.org/en/latest/#sample-code
The method above handles quitting the browser. Your method, also in the docs, should work (it works for me). Try it again, I've noticed I get almost the same error as yours, if my url is "www.google.com". Make sure you have "http://" part.
selenium.common.exceptions.WebDriverException: Message: Target URL
www.google.com is not well-formed