Develop Reference

Selenium Webdriver cannot find element which loads later than DOM

I've been trying to webscrap this page using selenium, and as you may notice on the first load a pop-up appears about agreeing on cookies. This pop-up appears around a second later that the rest of the DOM.
The problem is, that even after adding a sleep or WebDriverWait the driver still cannot click on the Accept button from the pop-up.
Inspecting the page gives that the Xpath of the element is /html/body/div/div[2]/div[4]/button[1] so I tried doing it as:
driver.get("https://www.planespotters.net/airlines")
time.sleep(5)
driver.find_element(By.XPATH, '/html/body/div/div[2]/div[4]/button[1]').click()
with no effect. I get that the element does not exist.
I tried it even with:
driver.execute_script('document.querySelector("#notice > div.message-component.message-row.button-container > button.message-component.message-button.no-children.focusable.primary.sp_choice_type_11").click()')
and
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '/html/body/div/div[2]/div[4]/button[1]'))).click()
Any suggestions on how to bypass this pop-up?

If you scroll up on devtools you'll notice your modal window is in an iframe. I queried the 'planespotters' header image (because it's the top item) and you can see it quite clearly:
For selenium switch to your iframe first - this finds the iframe wher the url contains 'privacy':
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'privacy')]")))
Then you can complete your action. Be sure to keep that sync in there too.
Then you can switch back to your main window and carry in with your test
driver.switch_to_default_content()

You can try introducing an implicit wait for your driver. This will happen once per script and will affect all objects. This will wait 10 seconds before throwing an error, or continues whenever it is ready:
driver.implicitly_wait(10)

click link automatically and scraping

I try to extract the all the products data from this page:
https://www.shufersal.co.il/online/he/קטגוריות/סופרמרקט/חטיפים%2C-מתוקים-ודגני-בוקר/c/A25
I tried
shufersal = "https://www.shufersal.co.il/online/he/%D7%A7%D7%98%D7%92%D7%95%D7%A8%D7%99%D7%95%D7%AA/%D7%A1%D7%95%D7%A4%D7%A8%D7%9E%D7%A8%D7%A7%D7%98/%D7%97%D7%98%D7%99%D7%A4%D7%99%D7%9D%2C-%D7%9E%D7%AA%D7%95%D7%A7%D7%99%D7%9D-%D7%95%D7%93%D7%92%D7%A0%D7%99-%D7%91%D7%95%D7%A7%D7%A8/c/A25"
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
import time
driver.get(shufersal)
products = driver.find_elements_by_css_selector("li.miglog-prod.miglog-sellingmethod-by_unit"
)
the problem is that the product details is showed only when I click the product:
There any option to click all links automatically and scraping the open windows?

What you want can be achieved, but is a significantly more time consuming processes.
You'll have to:
identify the elements that you need to click in the page (they'll probably have the same class) and then select them all:
butons_to_click = driver.find_elements_by_css_selector({RELEVANT SELECTOR HERE})
Then you should loop through all clickable elements, click on them, wait for the pop up to load, scrape the data, close the popup:
scraped_list = []
for button_instance in buttons_to_click:
button_instance.click()
#scrape the information you need here and append to scraped_list
#find close popup button
driver.find_element_by_xpath({XPATH TO ELEMENT}).click()
For this to work properly it is important to setup selenium's implicitly wait parameter. What that does is that if it doesn't find the required element it'll wait for X seconds until it is loaded, if X passes then it'll throw an error (you can handle the error in your code if it is expected).
In your case, you'll need the wait time because after you click on the product to display the popup, the information might take some seconds to load, in case you don't set implicitly wait, your script will exit with an element not found error. More information on selenium's wait parameters can be found here: https://selenium-python.readthedocs.io/waits.html
#put this line immediatelly after creating the driver object
driver.implicitly_wait(10) # seconds
***Suggestion:
I suggest you to always use Xpath when looking up elements, its syntax can actually emulate all other selenium selectors, is faster and will make an easier transition to C compiled html parsers (which you'll need in case you scale your scraper - recommend lxml a python package that uses a compiled parser)

Why won't Selenium in Python click the pop-up privacy button?

I am having some issues with Selenium not clicking the pop-up privacy button on https://www.transfermarkt.com/
Here is my code so far:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.transfermarkt.com/')
accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button')
accept_button.click()
It comes up saying:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[2]/div[3]/div[2]/button"}
Anyone have any thoughts?

I believe the issue is that the element you are trying to click is inside an iframe. In order to click that element you'll have to first switch to that frame. I noticed the iframe has title="SP Consent Message" so I'll use a CSS Selector to identify it based on that. Your code with the added line:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.transfermarkt.com/')
driver.switch_to.frame(driver.find_element_by_css_selector('iframe[title="SP Consent Message"]'))
accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button')
accept_button.click()
Note that you may have to switch back to the default frame to continue your test, I'm not 100% sure as that iframe has gone away.
Also it seems like some folks don't get that popup when the hit the website, not sure why, may be something you want to look in to to determine how you want to test this.
Like #Prophet says you should improve the xpath for the button (again it seems to have a unique title so I would use CSS Selector 'button[title="ACCEPT ALL"]'), but this works for me to click it.

There are 2 issues here:
You need to add wait / delay before accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button') to let the page load
It's very, very bad practice to use absolute XPaths. You have to define an unique, short and clear related XPath expression.
Additionally, I see no pop-up privacy button when I open that page. So, maybe that element is indeed not there.

Unable to locate element using selenium (Python)

I have been trying to find a button a click on it but no matter what I try it has been unable to locate it. I have tried using all the driver.find_element_by... methods but nothing seems to be working
from selenium import webdriver
import time
driver = webdriver.Chrome(executable_path="/Users/shreygupta/Documents/ComputerScience/PythonLanguage/Automation/corona/chromedriver")
driver.get("https://ourworldindata.org/coronavirus")
driver.maximize_window()
time.sleep(5)
driver.find_element_by_css_selector("a[data-track-note='chart-click-data']").click()
I am trying to click the DATA tab on the screenshot below

You can modify your script to open this graph directly:
driver.get("https://ourworldindata.org/grapher/total-cases-covid-19")
driver.maximize_window()
Then you can add implicitly_wait instead of sleep. An implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available (from python documentation). It'll work way faster because it'll interact with an element as soon as it finds it.
driver.implicitly_wait(5)
driver.find_element_by_css_selector("a[data-track-note='chart-click-data']").click()
Hope this helps, good luck.

Here is the logic that you can use, where the script will wait for max 30 for the Data menu item and if the element is present with in 30 seconds it will click on the element.
url = "https://ourworldindata.org/grapher/covid-confirmed-cases-since-100th-case"
driver.get(url)
driver.maximize_window()
wait = WebDriverWait(driver,30)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"a[data-track-note='chart-click-data']"))).click()

StaleElementReferenceException when I try to click the "n others" in "Liked by someone and n others" of an Instagram post, using Selenium and Python

So as you know, when you open a post on Instagram on Desktop, it will show "Liked by someone and 400 others". I am trying to click the "400 others" button using selenium, so that I could then go on to scrape the followers' names.
I retreived the Xpath of the button as /html/body/div1/section/main/div/div/article/div[2]/section[2]/div/div[2]/button
However, an error arises, which is the StaleElementReferenceException.
Below are my codes:
likes = driver.find_element_by_xpath("/html/body/div[1]/section/main/div/div/article/div[2]/section[2]/div/div[2]/button")
actions.move_to_element(likes).perform()
likes.click()
The error happens on the 2nd line, actions.move_to_element(likes).perform()
I wonder whether anyone knows how I can fix it.

ElementNotInteractableException occurs when an element is found, but you can't be interacted with because for example element is hide by another element or is not displayed for some reason and is not clickable anymore. Also you didn't say on which exactly step it failed with StaleElementReferenceException. Anyway maybe using Wait until an element is visible / clickable could help in this case. First solution could be refresh the page and try again for the same element:
driver.refresh()
Second solution could be wait for the element till it gets available:
from selenium.webdriver.support import expected_conditions as EC
likes = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[1]/section/main/div/div/article/div[2]/section[2]/div/div[2]/button")))
likes.click()
Hope it will help you!