I am trying to use selenium to search for courses on https://catalog.swarthmore.edu/ and scrape the results. All the selectors I have attempted to use fail, and when I print them out they return empty arrays. Why did these selectors fail, and what is the correct one? I get the "#keyword" and "span input" from hovering over and clicking on the search field with the chrome SelectorGadget extension, and I obtained the "span.show.clearfix input" and ".show.clearfix input" from examining the HTML with Chrome Devtools.
#import scrapy
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import json
browserOptions = Options()
browserOptions.add_argument('--headless')
driver = webdriver.Chrome('C:/WebDriver/bin/chromedriver.exe', options=browserOptions)
def css(selector):
return driver.find_elements_by_id(selector)
driver.get("https://catalog.swarthmore.edu/")
print(driver.title)
search = css("span.show.clearfix input")#css("#keyword")#get search field
print(search)
print(css("span input"))
print(css(".show.clearfix input"))
print(css("#keyword"))
search.send_keys("ANCH 028")#enter your search.
search.submit()
search.clear()#clear search field
driver.quit()
Error message:
Traceback (most recent call last):
File "getDescWJSON.py", line 31, in
search.send_keys("ANCH 028")#enter your search.
AttributeError: 'list' object has no attribute 'send_keys'
The page has multiple elements with id=keyword so the results is a list. To send the keys, select the first element from the list:
Try this code:
search = css('keyword')[0] #("span.show.clearfix input")#css("#keyword")#get search field
You should probably change the name of the function css since it actually searches by element id.
You could simply do this to send the keys to that element. Wait for the element to come up after page load and send keys to the element.
search=WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//input[#id='keyword']"))).send_keys("ANCH 028")
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
This worked for me:-
driver.get('https://catalog.swarthmore.edu/')
btn = driver.find_element_by_xpath('''/html/body/table/tbody/tr[3]/td[1]/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td/form/fieldset/span[3]/input[1]''')
btn.click() #you have to click it first
btn.send_keys("ANCH 028")
Related
I am trying to fill the form on (https://all-access.wax.io). When I am using Javascript
document.getElementsByName("userName")[0].value = "Hello", then I am able to write text to a form. However, when I am using same concept in selenuim
driver.find_element_by_name("userName").send_keys("Hello"), then I am getting:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
When I am executing:
self.driver.find_element_by_name("userName")[1].send_keys("Hello") this leads to:
TypeError: 'WebElement' object is not subscriptable
I have also tried to wait until content is loaded, as well as use XPath and other selectors. I guess I am doing a simple mistake, but I can`t resolve it for several hours already.
Code to reproduce a problem:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome()
browser.get('https://all-access.wax.io')
browser.find_element_by_name("userName")[1].send_keys("hello")
print()
browser.find_element_by_xpath("(//input[#name='userName'])[2]").send_keys("hello")
You want to send_keys to the second input tag and not the first.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome()
browser.get('https://all-access.wax.io')
browser.find_element_by_name("userName").send_keys("hello")
print()
you are using find_element which returns a webelement and not array , use find_elements or remove the index
The approach with only name attribute did not work for me because when you are trying to access with [1] there are still two elements, so the locator is not unique.
So, I used the parent pannels visible-desktop-only-flex class and also added explicit wait.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
browser = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
browser.get('https://all-access.wax.io')
wait = WebDriverWait(browser, 15)
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='pannels visible-desktop-only-flex']//input[#name='userName']")))
browser.find_element_by_xpath("//div[#class='pannels visible-desktop-only-flex']//input[#name='userName']").send_keys("hello")
I'm trying to get my program (written in python) to open multiple links at the same time. I can open one link – but when I change my command from find_element_by_link_text to find_elements_by_link_text it doesn't work.
For context, I'm trying to open all links on the page with the name 'Annual Report'.
Here's the code I've tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import options
driver = webdriver.Chrome()
driver.get('https://webb-site.com/')
search = driver.find_element_by_name('code')
search.click()
search.send_keys("1830")
search.send_keys(Keys.RETURN)
element = driver.find_element_by_link_text('Financials')
element.click()
element = driver.find_elements_by_link_text('Annual Report')
element.click()
Here's my error
AttributeError: 'list' object has no attribute 'click'
As I said above. By removing the s in 'elements' it works fine but only opens the first 'Annual Report'.
You are getting AttributeError: 'list' object has no attribute 'click' error due to click on a list object. If you want to click multiple items then use a for loop.
driver = webdriver.Chrome()
driver.get('https://webb-site.com/')
search = driver.find_element_by_name('code')
search.click()
search.send_keys("1830")
search.send_keys(Keys.RETURN)
element = driver.find_element_by_link_text('Financials')
element.click()
element = driver.find_elements_by_link_text('Annual Report')
# print(element)
# print(len(element))
# element.click()
for c in element:
c.click()
You are calling click to a list object.
The object element in the last line:
element = driver.find_elements_by_link_text('Annual Report')
is a list.
When you call find_elements you get in return a list.
Just change it to:
element = driver.find_element_by_link_text('Annual Report')
Edit
As #Samsul added, to click on all the elements you'll need to loop through the list.
Hoping you can help. I'm relatively new to Python and Selenium. I'm trying to pull together a simple script that will automate news searching on various websites. The primary focus was football and to go and get me the latest Manchester United news from a couple of places and save the list of link titles and URLs for me. I could then look through the links myself and choose anything I wanted to review.
In trying the the independent newspaper (https://www.independent.co.uk/) I seem to have come up against a problem with element not interactable when using the following approaches:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('chromedriver')
driver.get('https://www.independent.co.uk')
time.sleep(3)
#accept the cookies/privacy bit
OK = driver.find_element_by_id('qcCmpButtons')
OK.click()
#wait a few seconds, just in case
time.sleep(5)
search_toggle = driver.find_element_by_class_name('icon-search.dropdown-toggle')
search_toggle.click()
This throws the selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable error
I've also tried with XPATH
search_toggle = driver.find_element_by_xpath('//*[#id="quick-search-toggle"]')
and I also tried ID.
I did a lot of reading on here and then also tried using WebDriverWait and execute_script methods:
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="quick-search-toggle"]')))
driver.execute_script("arguments[0].click();", element)
This didn't seem to error but the search box never appeared, i.e. the appropriate click didn't happen.
Any help you could give would be fantastic.
Thanks,
Pete
Your locator is //*[#id="quick-search-toggle"], there are 2 on the page. The first is invisible and the second is visible. By default selenium refers to the first element, sadly the element you mean is the second one, so you need another unique locator. Try this:
search_toggle = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="row secondary"]//a[#id="quick-search-toggle"]')))
search_toggle.click()
First you need to open search box, then send search keys:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import os
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
browser = webdriver.Chrome(executable_path=os.path.abspath(os.getcwd()) + "/chromedriver", options=chrome_options)
link = 'https://www.independent.co.uk'
browser.get(link)
# accept privacy
button = browser.find_element_by_xpath('//*[#id="qcCmpButtons"]/button').click()
# open search box
li = browser.find_element_by_xpath('//*[#id="masthead"]/div[3]/nav[2]/ul/li[1]')
search_tab = li.find_element_by_tag_name('a').click()
# send keys to search box
search = browser.find_element_by_xpath('//*[#id="gsc-i-id1"]')
search.send_keys("python")
search.send_keys(Keys.RETURN)
Can you try with below steps
search_toggle = driver.find_element_by_xpath('//*[#class="row secondary"]/nav[2]/ul/li[1]/a')
search_toggle.click()
I need to scrape some text from a table, using selenium. I'm a complete novice, but thanks to Stack overflow, I was able to get to get pretty far. Now I am getting an error when using text() to copy text. Here are my codes.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
driver = webdriver.Chrome('C:/Users/User/Downloads/chromedriver')
url = "https://aca.tampagov.net/citizenaccess/Default.aspx#"
driver.get(url)
# The website I need to check
# Click "search" at the top menu
driver.find_element_by_xpath('//*[#id="main_menu"]/ul/li[2]/a').click()
# Click "building permits" in the menu that follows below
driver.find_element_by_xpath('//[#id="main_menu"]/ul/li[2]/ul/li[1]/a').click()
#change iframes
driver.switch_to.frame("ACAFrame")
# When changing iframes, I was recommended that I should use this,
# WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//*[#id='ACAFrame']")))
# but I'm now having troubles running this line so I changed it to the line like above
Now you can find that there's a table below. I want to copy text using a code such as:
driver.find_element_by_xpath('//tbody/tr[3]/td[4]').text()
But I get, instead:
Traceback (most recent call last):
File "<ipython-input-117-2a4c7f9321b5>", line 1, in <module>
driver.find_element_by_xpath('//tbody/tr[3]/td[4]').text()
TypeError: 'str' object is not callable
What am I doing wrong here?
Use
driver.find_element_by_xpath('//tbody/tr[3]/td[4]').text
instead of
driver.find_element_by_xpath('//tbody/tr[3]/td[4]').text()
The statement driver.find_element_by_xpath('//tbody/tr[3]/td[4]').text returns a string, so you don't have to add the parenthesis at the end. If you add the parenthesis, then syntax means you are calling it, but you can only call callable objects, such as functions. Thus, the error.
I tried below 2 ways to wait before clicking button, but I am getting an error. I want to wait until anchor text stored query is available.
element = WebDriverWait(driver, 200).until(EC.presence_of_element_located((By.ANCHOR,"stored query")))
driver.find_element_by_link_text("[Comma-Delimited Text (CSV)]").click()
or:
wait.until(ExpectedConditions.textToBePresentInElement(By.xpath("path"), "stored query"));
driver.find_element_by_link_text("[Comma-Delimited Text (CSV)]").click()
Error:
wait.until(ExpectedConditions.textToBePresentInElement(By.xpath("path"), "stored query"));
AttributeError: 'module' object has no attribute 'until'
I just went thru web page again searching any element that has id and found one form.
My final code for just waiting page is:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC, wait
# Some code for loading web page
WebDriverWait(driver, 200).until(EC.presence_of_element_located((By.ID, "call_list")))
# Hit button/ perform next action
driver.find_element_by_link_text("[Comma-Delimited Text (CSV)]").click()