Develop Reference

Send keys to window DOM with Selenium in python to bypass captcha

I am trying to get data from a webpage (https://www.educacion.gob.es/teseo/irGestionarConsulta.do). It has a captcha at the entry which I manually solve and move to the results page.
It happens that going back from the results page to the entry page does not modify the captcha if I reach the initial page with the "go back" button of the browser; but if I use the driver.back() instruction of Selenium's WebDriver, sometimes the captcha is modified - which I'd better avoid.
Clearly: I want to get access from Selenium to the DOM window (the browser), rather than to the document (or any element within the html) and send the ALT+ARROW_LEFT keys to the browser (the window).
This, apparently, cannot be done with:
from selenium.webdriver import Firefox
from selenium.webdriver.common.keys import Keys
driver = Firefox()
driver.get(url)
xpath = ???
driver.find_element_by_xpath(xpath).send_keys(Keys.ALT, Keys.ARROW_LEFT)
because send_keys connects to the element on focus, and my target is the DOM window, not any particular element of the document.
I have also tried with ActionChains:
from selenium.webdriver.common.action_chains import ActionChains
action = ActionChains(driver)
action.key_down(Keys.LEFT_ALT).key_down(Keys.ARROW_LEFT).send_keys().key_up(Keys.LEFT_ALT).key_up(Keys.ARROW_LEFT)
action.perform()
This also does not work (I have tried with several combinations). The documentation states that key_down/up require an element to send keys, which if None (the default) is the current focused element. So again here, there is the issue of how to focus on the window (the browser).
I have thought about using the mouse controls, but I assume it will be the same: I know how to make the mouse reach any element in the document, but I want to reach the window/browser.
I have thought of identifying the target element through the driver.current_window_handle, but also fruitlessly.
Can this be done from Selenium? If so, how? Can other libraries do it, perhaps pyppeteer or playwright?

Try with JavaScriptExecutor - driver.execute_script("window.history.go(-1)")

Why won't Selenium in Python click the pop-up privacy button?

I am having some issues with Selenium not clicking the pop-up privacy button on https://www.transfermarkt.com/
Here is my code so far:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.transfermarkt.com/')
accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button')
accept_button.click()
It comes up saying:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[2]/div[3]/div[2]/button"}
Anyone have any thoughts?

I believe the issue is that the element you are trying to click is inside an iframe. In order to click that element you'll have to first switch to that frame. I noticed the iframe has title="SP Consent Message" so I'll use a CSS Selector to identify it based on that. Your code with the added line:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.transfermarkt.com/')
driver.switch_to.frame(driver.find_element_by_css_selector('iframe[title="SP Consent Message"]'))
accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button')
accept_button.click()
Note that you may have to switch back to the default frame to continue your test, I'm not 100% sure as that iframe has gone away.
Also it seems like some folks don't get that popup when the hit the website, not sure why, may be something you want to look in to to determine how you want to test this.
Like #Prophet says you should improve the xpath for the button (again it seems to have a unique title so I would use CSS Selector 'button[title="ACCEPT ALL"]'), but this works for me to click it.

There are 2 issues here:
You need to add wait / delay before accept_button = driver.find_element_by_xpath('/html/body/div/div[2]/div[3]/div[2]/button') to let the page load
It's very, very bad practice to use absolute XPaths. You have to define an unique, short and clear related XPath expression.
Additionally, I see no pop-up privacy button when I open that page. So, maybe that element is indeed not there.

Scroll on a specific DIV Element in Python Selenium

I'm trying to do a simple Python Selenium automation where the script will click a link which opens a dialog box on top of the page (Instagram profile).
The said dialog box will display the list of followers but unfortunately the UL that contains the list will only display the first 12 followers (or LI). It is powered by AJAX to "load more" followers.
Anyway, to simulate loading more followers, i tried this code:
driver.find_element_by_xpath('/html/body/div[3]/div/div[2]/ul').send_keys(Keys.END)
or
driver.find_element_by_tag_name('body').send_keys(Keys.END)
unfortunately, it doesn't work. I was wondering if there is a correct way to do this (scrolling down focused on the active div or any page element)?
Image below is provided to show the html structure of the said page.
Appreciate your help on this, thank you very much!

Can you try something like this?. This will scroll to the div element that you have mentioned.
Python Code
element = driver.find_element_by_xpath("xpath_of_div_element")
driver.execute_script("arguments[0].scrollIntoView(true);", element);
Java sample:
WebElement element = driver.findElement(By.id("id_of_div_element"));
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element);

Use this
liElement = driver.find_element_by_xpath("//div[#role='dialog']//ul//li[text()='required_li_element_visible_text']")
driver.execute_script("arguments[0].scrollIntoView(true);", liElement);

Selenium Webdriver loop over iframe

I am trying to webscrape an iframe object containing a list of links. The ultimate objective is to perform a series of actions, same for each link, that leads to downloading some data in xls format.
For each of the links in the iframe, I want to automate the following actions:
click on the link
select from a drop down list the option for getting data since 1999
click on the excel download button.
My code works for the first of the links, but I am struggling with automating it to go through all the links in the iframe.
This is the code that works for the first link:
#this automates the log in
driver.find_element_by_id('user').send_keys('myusername')
driver.find_element_by_id('password').send_keys('mypassword')
driver.find_element_by_xpath('myxpath').click()
#this switches to iframe and clicks on all the links
driver.switch_to.frame("report-iframe")
driver.find_element_by_tag_name('a').click()
#this performs the required actions for the first link(select drop down, select excel download)
s= Select(driver.find_element_by_id('start-date'))
s.select_by_value('1999-01-01')
driver.find_element_by_xpath('//*[#id="report-excel-download"]')
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, 'report-excel-download')))
element.click()
I want to automate this process for all the links in the iframe (i.e. for each of the links in iframe, click on the link, select the drop down, click the data download button. Does anyone have any advice in how to do this?
I tried:
links=[link.get_attribute('href') for link in driver.find_elements_by_xpath('myxpath')]
for l in links:
#click the link
driver.find_element_by_tag_name('a').click()
#select all years
s= Select(driver.find_element_by_id('report-control-date-start'))
s.select_by_value('1999-01-01')
#click download
driver.find_element_by_xpath('//*[#id="report-excel-download"]')
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, 'report-excel-download')))
element.click()
But this only works for the first link in the iframe, it does not work across all.
Does anyone have any advice?
Thanks!

why is selenium webdriver not finding my element

https://www.mykplan.com/participantsecure_net/TermsAndConditions.aspx
I am doing find by id/xpath/name and they all fail for the accept button. Here is my latest effort
driver.find_element_by_xpath('//*[#id="Accept"]').click()
copied straight from chrome web tool

The button is located inside a frame. Given xpath is correct only inside a frame. I tested xpaths in chrome console and this is what I got:
In case of main page (https://www.mykplan.com/participantsecure_net/TermsAndConditions.aspx) xpath couldn't be located:
$x('//*[#id="Accept"]');
[]
In case of frame contents only (https://www.mykplan.com/participantsecure_net/TermsAndConditionsBottom.aspx) xpath could be found:
$x('//*[#id="Accept"]');
[<input type=​"submit" name=​"Accept" value=​"I agree" id=​"Accept">​]
In selenium, I guess you need to switch to a frame before looking for xpath. I think that web driver function
driver.switch_to_frame("frameName")
should help. In your case, frame with buttons is called "bottomFrame".