Develop Reference

How to select a g-menu-item in selenium (Python)

I am trying to find news articles related to different web searches within a custom time frame. If someone knows a better way to do this than with selenium then please do tell. I tried to use API's but they are all expensive.
My Code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path=r".\chromedriver.exe")
driver.get("https://www.google.com")
cookie_button = driver.find_element_by_id('L2AGLb')
cookie_button.click()
search_bar = driver.find_element_by_name("q")
search_bar.clear()
search_bar.send_keys("Taylor Swift")
search_bar.send_keys(Keys.RETURN)
news_button = driver.find_element_by_css_selector('[data-hveid="CAEQAw"]')
news_button.click()
tools_button = driver.find_element_by_id('hdtb-tls')
tools_button.click()
recent_button = driver.find_element_by_class_name('KTBKoe')
recent_button.click()
custom_range_button = dashboards_button = driver.find_elements_by_tag_name('g-menu-item')[6]
custom_range_button.click()
driver.close()
I am trying to click this button:
I have tried to select, the span, all parent divs and now the g-menu-item itself, in every case I have received the error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
I understand this can happen when the element is not visible or can not be clicked but I also tried adding in sleep(2) between each operation, and I can see in the browser that selenium opens that it gets to the point in the image I have sent and then can't find the custom range button ??
In short:
Do you know how I can click this custom range button,
Or do you know how to scrape a lot of news articles from specific dates (without expensive API's)

Those g-menu are special type of tag,
you can locate them using :
//*[name()='g-menu-item']
this xpath will point to all the nodes.
further, I see that you are interested in 6th item.
(//*[name()='g-menu-item'])[6]
in code :
custom_range_button = driver.find_element_by_xpath("(//*[name()='g-menu-item'])[6]")
custom_range_button.click()

Unable to find element by any means

I'm kinda of a newbie in Selenium, started learning it for my job some time ago. Right now I'm working with a code that will open the browser, enter the specified website, put the products ID in the search box, search, and them open it. Once it opens the product, it needs to extract its name and price and write it in a CSV file. I'm kinda struggling with it a bit.
The main problem right now is that Selenium is unable to open the product after searching it. I've tried by ID, name and class and it still didn't work.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import csv
driver = webdriver.Firefox()
driver.get("https://www.madeiramadeira.com.br/")
assert "Madeira" in driver.title
elem = driver.find_element_by_id("input-autocomplete")
elem.clear()
elem.send_keys("525119")
elem.send_keys(Keys.RETURN)
product_link = driver.find_element_by_id('variant-url').click()
The error I get is usually this:
NoSuchElementException: Message: Unable to locate element: [id="variant-url"]

There are multiple elements with the id="variant-url", So you could use the index to click on the desired element, Also you need to handle I think the cookie pop-Up. Check the below code, hope it will help
#Disable the notifications window which is displayed on the page
option = Options()
option.add_argument("--disable-notifications")
driver = webdriver.Chrome(r"Driver_Path",chrome_options=option)
driver.get("https://www.madeiramadeira.com.br/")
assert "Madeira" in driver.title
elem = driver.find_element_by_id("input-autocomplete")
elem.clear()
elem.send_keys("525119")
elem.send_keys(Keys.RETURN)
#Clicked on the cookie popup button
accept= driver.find_element_by_xpath("//button[#id='lgpd-button-agree']")
accept.click()
OR with explicitWait
accept=WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//button[#id='lgpd-button-agree']")))
accept.click()
#Here uses the XPath with the index like [1][2] to click on the specific element
product_link = driver.find_element_by_xpath("((//a[#id='variant-url'])[1])")
product_link.click()
OR with explicitWait
product_link=WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"((//a[#id='variant-url'])[1])")))
product_link.click()

I used to get this error all the time when starting. The problem is that there are probably multiple elements with the same ID "variant-url". There are two ways you can fix this
By using "driver.find_elementS_by_id"
product_links = driver.find_elements_by_id('variant-url')
product_links[2].click()
This will create an index off all the elements with id 'variant-url' then click the index of 2. This works but it is annoying to find the correct index of the button you want to click and also takes a long time if there are many elements with the same ID.
By using Xpaths or CSS selectors
This way is a lot easier as each element has a specific Xpath of Selector. It will look like this.
product_link = driver.get_element_by_xpath("XPATH GOES HERE").click()
To get an Xpath or selector 1. go into developer mode on your browser by inspecting the element 2. right click the element in the F12 menu 3. hover over copy 4. move to copy Xpath and click on it.
Hope this helps you :D

Python Selenium unable to locate element when it's there?

I am trying to scrape product's delivery date data from a bunch of lists of product urls.
I am running the python file on terminal doing multi-processing so ultimately this opens up multiple chrome browsers (like 10 - 15 of them) which slows down my computer quite a bit.
My code basically clicks a block that contains shipping options, which would would show a pop up box that shows estimated delivery time. I have included an example of a product url in the code below.
I noticed that some of my chrome browsers freeze and do not locate the element and click it like I have in my code. I've incorporated refreshing the page in webdriver into my code just in case that will do the trick but it doesn't seem like the frozen browsers are even refreshing.
I don't know why it would do that as I have set the webdriver to wait until the element is clickable. Do I just increase the time in time.sleep() or the seconds in webdriverwait() to resolve this?
chromedriver = "path to chromedriver"
driver = webdriver.Chrome(chromedriver, options=options)
# Example url
url = "https://shopee.sg/Perfect-Diary-X-Sanrio-MagicStay-Loose-Powder-Weightless-Soft-velvet-Blurring-Face-Powder-With-Cosmetic-Puff-Oil-Control-Smooth-Face-Powder-Waterproof-Applicable-To-Mask-Face-Cinnamoroll-Purin-Gudetama-i.240344695.5946255000?ads_keyword=makeup&adsid=1095331&campaignid=563217&position=1"
driver.get(url)
time.sleep(2)
try:
WebDriverWait(driver,60).until(EC.element_to_be_clickable((By.XPATH,'//div[#class="flex flex-column"]//div[#class="shopee-drawer "]'))).click()
while retries <= 5:
try:
shipping_block = WebDriverWait(driver,60).until(EC.element_to_be_clickable((By.XPATH,'//div[#class="flex flex-column"]//div[#class="shopee-drawer "]'))).click()
break
except TimeoutException:
driver.refresh()
retries += 1
except (NoSuchElementException, StaleElementReferenceException):
delivery_date = None

The element which you desire will get displayed when you hover the mouse, V The type of the element is svg which you need to handle accordingly.
you can take the help of this XPath to hover the mouse
((//*[name()='svg']//*[name()='g'])/*[name()='path'][starts-with(#d, 'm11 2.5c0 .1 0 .2-.1.3l')])
Getting the text from the popUp you need to check with this XPath
((//div[#class='shopee-drawer__contents']//descendant::div[4]))
You can use get_attribute("innerText") to get all the values
You can just check it here the same answer, I hope it will help

First, don't use headless option in webdriver.ChromeOption() to let the webpage window pop up to observe if the element is clicked or not.
Second, your code is just click the element, it doesn't help you to GET anything. After click and open new window, it should do something like this:
items = WebDriverWait(driver, 60).until(
EC.visibility_of_all_elements_located((By.CLASS_NAME, 'load-done'))
)
for item in items:
deliver_date= item.get_attribute('text')
print(deliver_date)

Selenium Webdriver Timeout (Python 2.7)

When scraping data from NASDAQ there are tickers like ACHC that have empty pages. ACHC Empty Field
My program iterates through all ticker symbols and when I get to this one it times out because there is no data to grasp. I am trying to figure out a way to check if there is nothing and if so skip the ticker, but continue the loop. The code is pretty long, so Ill post the most relevant part: the beginning of the loop where it opens the page:
## navigate to income statement annualy page
url = url_form.format(symbol, "income-statement")
browser.get(url)
company_xpath = "//h1[contains(text(), 'Company Financials')]"
company = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, company_xpath))).text
annuals_xpath = "//thead/tr[th[1][text() = 'Period Ending:']]/th[position()>=3]"
annuals = get_elements(browser,annuals_xpath)
Here is a pic of the error message

Selenium doesn't have a built-in method for determining whether an element exists or not, so the most common thing to do is use a try/except block.
from selenium.common.exceptions import TimeoutException
...
try:
company = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, company_xpath))).text
except TimeoutException:
continue
This should keep the loop going without crashing, assuming that continue works as expected with your loop.

You can use libraries like requests or urllib to scrape that web page and check if what you need is there. These libraries are much faster than Selenium because they just fetch the source of the page. If there are particular tags or structures like tables, etc. that you're looking for, you should take a look at beautifulsoup which you can use with requests to identify very specific parts of the page.

Selenium will not find my elements

firebug
console
I have a project that I chose Selenium to open 1-5 links. It's stopping at the 3rd link. I've followed the same methods for the previously successful requests. I've allowed 17 seconds and watched as I can see the page load, before the script continues to run in my console. I'm just not sure why it can't find this link, and I hope it's something I'm simply overlooking...
from selenium import *
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import csv
import time
username = "xxxxxxx"
password = "xxxxxxx"
driver = webdriver.Firefox()
driver.get("https://tm.login.trendmicro.com/simplesaml/saml2/idp/SSOService.php")
assert "Trend" in driver.title
elem1 = driver.find_element_by_class_name("input_username")
elem2 = driver.find_element_by_class_name("input_password")
elem3 = driver.find_element_by_id("btn_logon")
elem1.send_keys(username)
elem2.send_keys(password)
elem3.send_keys(Keys.RETURN)
time.sleep(7)
assert "No results found." not in driver.page_source
elem4 = driver.find_element_by_css_selector("a.float-right.open-console")
elem4.send_keys(Keys.RETURN)
time.sleep(17)
elem5 = driver.find_element_by_tag_name("a.btn_left")
elem5.send_keys(Keys.RETURN)

Well one of the reasons is elem5 is looking for the element by tag name, but you are passing it a css tag. "a.btn_left" is not an html tag name and so your script will never actually find it, because it simply doesn't exist in the dom.
You either need to find it by css_selector or better yet by Xpath. If you want to make this as reliable possible and more future proof I always try and find elements on a page with at least 2 descriptors using Xpath if possible.
Change this:
elem5 = driver.find_element_by_tag_name("a.btn_left")
To this:
elem5 = driver.find_element_by_css_selector("a.btn_left")
You will almost never use tag_name, mostly because it will always retrieve the first tag you pass to it, so "a" will always find the first link and click it, yours however does not exist.

I wound up solving it with this code. I increased time to 20 secs, believe it or not, I did try the find by css, I actually left the a.btn_left, and cycled through all the elements, and none of them worked, fortunately, I could access by tab and key functions so that worked for now.
time.sleep(20)
driver.get("https://wfbs-svc-nabu.trendmicro.com/wfbs-svc/portal/en/view/cm")
elem5 = driver.find_element_by_link_text("Devices")
elem5.send_keys(Keys.ENTER)