Develop Reference

How do I click on an item on my google search page with Selenium Python?

Good time of the day!
Faced with a seemingly simple problem,
But it’s been a while, and I’m asking for your help.
I work with Selenium on Python and I need to curse about
20 items on google search page by random request.
And I’ll give you an example of the elements below, and the bottom line is, once the elements are revealed,
Google generates new such elements
Problem:
Cannot click on the element. I will need to click on existing and then on new, generated elements in this block: click for open see blocks with element
Tried to click on xpath, having collected all the elements:
xpath = '//*[#id="qmCCY_adG4Sj3QP025p4__16"]/div/div/div[1]/div[4]'
all_elements = driver.find_element(By.XPATH, value=xpath)
for element in all_elements:
element.click()
sleep(2)
Important note!
id xpath has constantly changing and is generated by another on the google side
Tried to click on the class
class="r21Kzd"
Tried to click on the selector:
#qmCCY_adG4Sj3QP025p4__16 > div > div > div > div.wWOJcd > div.r21Kzd
Errors
This is when I try to click using xpath:
Message: no such element: Unable to locate element: {"method":"xpath","selector"://*[#id="vU-CY7u3C8PIrgTuuJH4CQ_9"]/div/div[1]/div[4]}
In other cases, the story is almost the same, the driver does not find the element and cannot click on it. Below I apply a scratch tag on which I need to click
screenshot tags on google search
Thanks for the help!

In case iDjcJe IX9Lgd wwB5gf are a fixed class name values of that element all you need is to use CSS_SELECTOR instead of CLASS_NAME with a correct syntax of CSS Selectors.
So, instead of driver.find_element(By.CLASS_NAME, "iDjcJe IX9Lgd wwB5gf") try using this:
driver.find_element(By.CSS_SELECTOR, ".iDjcJe.IX9Lgd.wwB5gf")
(dots before each class name, no spaces between them)

How to switch to the child frame within #shadow-root (open) using Selenium

I'm trying to make a web scraper that downloads an image that's inside of an iframe with a child.
I can't get Selenium for Chrome to find the correct iframe to switch into. The main issue is the iframe in question doesn't have a name or id so I searched by index. I managed to get inside of the parent, but I can't get inside of the sub-child. If I set the index to 1 I get the next iframe in the outermost scope.
From looking into my webdriver object I think the search is limited to Red Rectangle, as thats what's inside the page source attribute of my var "driver".
The Object I want to reach is the img with the id pbk-page in the Green Rectangle
My code so far just gets the url then waits for the page to load using sleep (once I can navigate to the correct element I'll implement WebDriverWait). This is the navigation bit of code:
driver.switch_to.frame(0)
Image_link = driver.find_element(By.ID,'pbk-page')
Oh! I'm using python

Like any other element iframe can be located by XPath or CSS Selector. They can use any attribute value making that locator unique. I believe here you could uniquely locate both the iframes by their src value, but since you marked them out I can't see their values.

As per the given HTML:
The desired element:
<img id="pbk-page"....>
is within the child <iframe> which is within a #shadow-root (open) marked with a blue rectangle.
Solution
To access the desired element you need to:
Switch within the parent iframe
Switch within the shadow-root
Switch within the child iframe
Then locate the element
Effectively, your code block will be:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"parent_iframe_css_selector")))
shadow_host = driver.find_element(By.CSS_SELECTOR, 'mosaic-book')
shadow_root = shadow_host.shadow_root
shadow_content = shadow_root.find_element(By.CSS_SELECTOR, 'child_iframe_css_selector')
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"child_iframe_css_selector")))
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//img[#id='pbk-page']")))

I was stuck doing the exact same thing you were (maybe even scraping the same website?), and this is what worked for me:
My solution:
iframe1 = driver.find_elements(By.XPATH, value="//iframe")[0]
driver.switch_to.frame(iframe1)
iframe2 = driver.execute_script("return document.querySelector(\"body > mosaic-book\").shadowRoot.querySelector(\"iframe\")")
driver.switch_to.frame(iframe2)
img = driver.find_elements(By.ID, value="pbk-page")
I am very much an amateur at using Selenium, but this is my best understanding of how this works: First, we're able to find the parent iframe iframe1, but our driver can't see anything inside of the shadow DOM. However, we can access inside of the shadow DOM using javascript, so starting from the iframe, we can find the shadow host element mosaic-book, enter the shadow DOM, and return/pass out the child iframe iframe2. Then we can switch our driver into this iframe2 and access the image.
There very well might be a more elegant way to do this, but this is what worked for me.
Reading I did to come up with this solution:
What even is a shadow root/shadow DOM or whatever??? https://www.geeksforgeeks.org/what-is-shadow-root-and-how-to-use-it/
How to access elements inside of a shadow DOM using javascript: https://www.youtube.com/watch?v=PQcRaIoc2AM

Python selenium presence_of_element_located with only href

I am trying to click on a link in a forum using Selenium, but I need to wait until the page load, so i thought the better way was to use WebDriverWait. This is my code I used to test it:
driver.get("https://testocolo.forumcommunity.net")
#First click, working
driver.find_element_by_xpath('//a[#href="'+"/?f=9087616"+'"]').click()
try :
element = WebDriverWait(driver, 2).until(
EC.presence_of_element_located(By.XPATH, '//a[#href="'+"/?t=61904616"+'"]')
)
element.click()
except :
print("NO")
This is the element
Brotha
The try except cycle ends up every time printing "NO".
Before that I tried locating by LINK_TEXT instead, with 'Brotha' but in neither way works. Where am I doing wrong?

xpath you can try
//a[contains(#tittle,'discussione inviata il')]
or
//*[text()='Brotha']
Next option you can check is if that element is in iframe?
WebDriverWait(driver, 30).until(
EC.element_to_be_clickable((By.XPATH, "//*[text()='Brotha']")))

How to fix error when changing drop down option

I am very new to web scraping, so I still have lots of trouble. Currently, I am trying to web scrape from https://www.enterprisetrucks.com/truckrental/en_US.html by setting the pickup time by running this code:
pickupTime = d.find_element_by_id('fldPickuptime_msdd')
pickupTime.click();
select = Select(d.find_element_by_id('fldPickuptime'))
select.select_by_value('20:00')
But I get the error saying that the element is not currently visible and may not be manipulated.

The dropdown which is present is not of the Select type, so you cannot use the Select method here. You need to click on the time using the xpath of that element directly.
You can use the xpath:
pickupTime = d.find_element_by_id("fldPickuptime_msdd")
pickupTime.click();
selectTime = d.find_element_by_xpath("//*[#id='fldPickuptime_msdd']//span[text()='8:00 PM']")
selectTime.click();
Code for JavaScriptExecutor Click:
element = driver.find_element_by_xpath("Enter the xpath here")
driver.execute_script("arguments[0].click();", element)

Selenium WebDriver - cannot locate some elements inside iFrame

I'm running test scripts on the Shopify checkout page, and while I am able to switch to the iFrame and get the Card Number element, the same approach doesn't seem to work for the other input elements (id='expiry', id='name' etc)
Form screenshot
Inspect iFrame screenshot
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#class='card-fields-iframe']")))
element12 = WebDriverWait(driver,20).until(EC.presence_of_element_located((By.ID,'number')))
element12.click()
element12.send_keys('1111222233334444')
Have also tried driver.switchTo().defaultContent() and switching back to iFrame again, but same error
element13 = WebDriverWait(driver,20).until(EC.presence_of_element_located((By.ID,'expiry')))
element13.click()
element13.send_keys('1220')
Here's the error I get for element13.
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (489, -1)

Found out the issue - the elements were in separate iFrames. Modified my code to retrieve the iFrames using xpath and a partial match on the different id fields