Develop Reference

Selenium Webdriver cannot find element which loads later than DOM

I've been trying to webscrap this page using selenium, and as you may notice on the first load a pop-up appears about agreeing on cookies. This pop-up appears around a second later that the rest of the DOM.
The problem is, that even after adding a sleep or WebDriverWait the driver still cannot click on the Accept button from the pop-up.
Inspecting the page gives that the Xpath of the element is /html/body/div/div[2]/div[4]/button[1] so I tried doing it as:
driver.get("https://www.planespotters.net/airlines")
time.sleep(5)
driver.find_element(By.XPATH, '/html/body/div/div[2]/div[4]/button[1]').click()
with no effect. I get that the element does not exist.
I tried it even with:
driver.execute_script('document.querySelector("#notice > div.message-component.message-row.button-container > button.message-component.message-button.no-children.focusable.primary.sp_choice_type_11").click()')
and
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '/html/body/div/div[2]/div[4]/button[1]'))).click()
Any suggestions on how to bypass this pop-up?

If you scroll up on devtools you'll notice your modal window is in an iframe. I queried the 'planespotters' header image (because it's the top item) and you can see it quite clearly:
For selenium switch to your iframe first - this finds the iframe wher the url contains 'privacy':
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'privacy')]")))
Then you can complete your action. Be sure to keep that sync in there too.
Then you can switch back to your main window and carry in with your test
driver.switch_to_default_content()

You can try introducing an implicit wait for your driver. This will happen once per script and will affect all objects. This will wait 10 seconds before throwing an error, or continues whenever it is ready:
driver.implicitly_wait(10)

Selenium wait for an element not present on load page

I'm trying to use Selenium to scrape google maps unfortunatly its's not quite working,the element is not present on page load and is added after clicking on some button but it seems that the element is not always loaded when looking for it. (I'm talking about the carousel's items that appears after clicking on a shop,restaurant while doing a specific search)
I already tried to use the classic Selenium wait options.
Things I tried:
time.sleep()
WebDriverWait(driver,30).until(EC.visibility_of_element_located(...)
WebDriverWait(driver,30).until(EC.element_to_be_clickable(...)
WebDriverWait(driver,60).until(EC.presence_of_all_elements_located(...)
Even with theses things results are random, I sometimes can access the element via Selenium and sometimes I don't.
It seem's that even with long waits, Selenium can't always access the web element even tho I can click it and see it.

You can make the web driver wait until the element is clickable. Have you tried that?
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))
or a better way of doing this as pointed out here
element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "myXpath")))

Why won't Selenium in Python click the pop-up privacy button?

I am having some issues with Selenium not clicking the pop-up privacy button on https://www.transfermarkt.com/
Here is my code so far:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.transfermarkt.com/')
accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button')
accept_button.click()
It comes up saying:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[2]/div[3]/div[2]/button"}
Anyone have any thoughts?

I believe the issue is that the element you are trying to click is inside an iframe. In order to click that element you'll have to first switch to that frame. I noticed the iframe has title="SP Consent Message" so I'll use a CSS Selector to identify it based on that. Your code with the added line:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.transfermarkt.com/')
driver.switch_to.frame(driver.find_element_by_css_selector('iframe[title="SP Consent Message"]'))
accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button')
accept_button.click()
Note that you may have to switch back to the default frame to continue your test, I'm not 100% sure as that iframe has gone away.
Also it seems like some folks don't get that popup when the hit the website, not sure why, may be something you want to look in to to determine how you want to test this.
Like #Prophet says you should improve the xpath for the button (again it seems to have a unique title so I would use CSS Selector 'button[title="ACCEPT ALL"]'), but this works for me to click it.

There are 2 issues here:
You need to add wait / delay before accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button') to let the page load
It's very, very bad practice to use absolute XPaths. You have to define an unique, short and clear related XPath expression.
Additionally, I see no pop-up privacy button when I open that page. So, maybe that element is indeed not there.

Automate clicking button using python and selenium

I would like to automate clicking on "Refresh" button (See image "Refreshbutton") using python and selenium. I have tried to locate the button using xpath, id, class name but each time, I get the NoSuchElement exception, i.e. it cannot find the element. How can I locate the Refresh button? I don't care if it is by xpath, id or any other means. Thanks in advance!
HTML
RefreshButton
Find by XPATH try
Find By ID try
Class Name try
Python Code Snippet

The problem might not be the id/xpath but the presence of the button on the page. If the click is executed before the HTML page is fully loaded, the element might not be here yet. To make sure, you can use the WebDriverWait as follow:
element_xpath = '//*[#id="btnRefreshCameraList"]'
wait_time = 15 # wait for page to load for max. 15 seconds
WebDriverWait(
driver, wait_time).until(
EC.presence_of_element_located((By.XPATH, element_xpath))
)
)
driver.find_element_by_xpath(element_xpath).click()
Didn't have access to your site, so I couldn't try it out.

python selenium cannot locate clickable play button on mobile version of webpage

I am having a hard time locating the play element on a mobile version of a webpage (my python script is passing a mobile user-agent in the header.)
the website url is below (NOTE: must be accessed with a mobile user-agent else it won't show the correct page and reverts to standard browser page instead)
https://m.soundcloud.com/mbmproductions/sets/mark-berrys-playlist
Using inspect in the browser, I have been unable to work out what exactly is needing to be clicked to start it playing. The entire image area around the play button seems to work if I click on it manually, but no element listed by inspect seems to work when using the script to start it playing.
Can anyone explain how to find the correct clickable element in this link so I can add it to a script in order to click it and start it playing. Ideally using XPATH as per my code below, but class or ID or anything would do if it works, even javascript if I have to.
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, targetelement))
)
element.click()

I make some amendment to your script. Use action instead of click(). Change clickable explicit to presence.
element = WebDriverWait(driver, 10).until(
EC. presence_of_element_located
((By.CLASS_NAME, ‘g-header-artwork-controls’))
)
el = driver.find_elements_by_css_selector(“.playlist__playButton.g-play-button")
webdriver.ActionChains(driver).move_to_element(el).click(el).perform()