I am running a streamlit app, in which I try to run selenium (click a search box and) inside iframe component, but I keep getting an error "NoSuchElementException".
It successfully opens the iframe on streamlit app, but it does not execute the selenium code inside the iframe. The selenium code itself runs fine when I test it without the iframe, but I can't get it to work inside the iframe.
Here is my code:
import streamlit as st
from dateutil.parser import parse
import streamlit.components.v1 as components
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
st.title("Auto Search App")
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
# url = 'https://wego.here.com/'
# driver.get(url)
components.iframe("https://wego.here.com/", height = 500)
## Give time for iframe to load ##
time.sleep(3)
## You have to switch to the iframe like so: ##
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
search_input = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input.input_search")))
search_input.click()
search_input.send_keys('Seattle')
search_input.send_keys(Keys.ENTER)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button.btn"))).send_keys(Keys.ENTER)
internal_search_input = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input#itinerary_item_input_0")))
internal_search_input.click()
internal_search_input.send_keys('Chicago')
internal_search_input.send_keys(Keys.ENTER)
This is the streamlit page with the error message:
EDIT: Based on the answer, I figured that I don't have to switch to iframe in order to access the search input element. I guess the problem is due to the fact that when I run "streamlit run app5.py" on cmd, it generates two separate browsers with an empty browser opening most recently, so that the selenium codes is executing on an empty page not on the streamlit localhost:8501 page. How can I not generate the empty browser and only open the localhost:8501 page, so that the selenium code runs on the streamlit page?
EDIT2: I added more selenium execution code for your reference. The reason that I am using selenium here is to do some automation done on the iframe so that it populates the navigation page at the end. I was able to get it done on a separate chrome by using driver.get(url) but not on streamlit with iframe. This is the final result I want to see in the iframe:
From your code i understand that you are trying to run the selenium code to set the value of the input in the iframe inside the Streamlit page that opens.
This is not possibble. I will try to explain using the order of events when you run this code.
With your current code this is what happens:
you run the streamlit run app.py command.
Streamlit opens this code in a chrome browser.
The selenium code starts running and because you don't use driver.get(url) it runs on an empty page causing the NoSuchElementException.
If you use the driver.get(url) and set the url to the Streamlit app url='http://localhost:8501/' then it also wont work because this is what happens:
you run the streamlit run app.py command.
Streamlit opens this code in a chrome browser.
The selenium code starts running and opens a new chrome tab of the selenium code and you cant make selenium get an already opened tab (session).
I would suggest not using selenium for this and to create a text_input using Streamlit and generate the iframe using the geolocation of the input.
Code:
import streamlit as st
import streamlit.components.v1 as components
from geopy.geocoders import Nominatim
bt = st.text_input("Enter Location")
if bt:
geolocator = Nominatim(user_agent="my_app")
location = geolocator.geocode("Seattle")
components.iframe(f"https://wego.here.com/?map={location.latitude},{location.longitude},18,normal&x=ep", height = 500)
Output:
Edit 1 - if you just want to view direction from Chicago to Seattle this can be done by changing only the url.
code:
import streamlit as st
import streamlit.components.v1 as components
from geopy.geocoders import Nominatim
components.iframe("https://wego.here.com/directions/mix/Chicago/Seattlel", height = 500)
Output:
The input.input_search element is NOT inside any iframe on that page.
There are 4 iframes there, however the search input is not inside any of them.
So you should not switch to iframe in order to access the search input element.
You need to consider a couple of things here.
The desired <input> isn't within any <iframe> as such, so you don't need to switch to any frame.
To click() on any clickable element instead of presence_of_element_located() you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Code Block:
driver.get("https://wego.here.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input_search"))).send_keys("Seattle" + Keys.RETURN)
Browser Snapshot:
Related
I am trying to use Selenium in order to learn different ways of web scraping.
When the code is executed Firefox starts and the "accept cookies" or what ever pops up. I am unable to locate the "accept" button when inspecting the page.
my code so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
import pandas as pd
import time
PATH = "C:/Users/myuser/Desktop/Driver/geckodriver.exe"
driver = webdriver.Firefox(executable_path=PATH)
driver.maximize_window() # For maximizing window
driver.get("https://www.immonet.de/")
button_pos = driver.find_element(by=By.CLASS_NAME, value="sc-gsDKAQ fILFKg")
button_pos.click()
print(driver.title)
input = input()
I get the following error: Unable to locate element: .sc-gsDKAQ fILFKg
My thought was locating the button via the inspect tool as follows:
What am I missing or doing wrong? How would i find the right element?
Thanks!
Pat
First of all, to display this url,accepting the cookies is a must but to accept and click on the cookie button isn't a easy task because cookies button is under shadow root (open) selenium and webdriverWait can do nothing on shadow root,so to execute shadow root you need to apply JavaScript querySelector.
#To execute shadow root and accept cookies
driver.execute_script('''return document.querySelector('div#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')''').click()
Class attribute in the html element can contain multiple classes separated by space. i.e. "sc-gsDKAQ fILFKg", contains two classes, sc-gsDKAQ and fILFKg.
You can user either but both are random and can be changed next time css is recompiled. I recommend to think of xpath using data-testid attribute
I'm trying to pull the airline names and prices of a specific flight. I'm having trouble with the x.path and/or using the right html tags because when I run the code below, all I get back is 14 empty lists.
from selenium import webdriver
from lxml import html
from time import sleep
driver = webdriver.Chrome(r"C:\Users\14074\Python\chromedriver")
URL = 'https://www.google.com/travel/flights/searchtfs=CBwQAhopagwIAxIIL20vMHBseTASCjIwMjEtMTItMjNyDQgDEgkvbS8wMWYwOHIaKWoNCAMSCS9tLzAxZjA4chIKMjAyMS0xMi0yN3IMCAMSCC9tLzBwbHkwcAGCAQsI____________AUABSAGYAQE&tfu=EgYIAhAAGAA'
driver.get(URL)
sleep(1)
tree = html.fromstring(driver.page_source)
for flight_tree in tree.xpath('//div[#class="TQqf0e sSHqwe tPgKwe ogfYpf"]'):
title = flight_tree.xpath('.//*[#id="yDmH0d"]/c-wiz[2]/div/div[2]/div/c-wiz/div/c-wiz/div[2]/div[2]/div/div[2]/div[6]/div/div[2]/div/div[1]/div/div[1]/div/div[2]/div[2]/div[2]/span/text()')
price = flight_tree.xpath('.//span[contains(#data-gs, "CjR")]')
print(title, price)
#driver.close()
This is just the first part of my code but I can't really continue without getting this to work. If anyone has some ideas on what I'm doing wrong that would be amazing! It's been driving me crazy. Thank you!
I noticed a few issues with your code. First of all, I believe that when entering this page, first google will show you the "I agree to terms and conditions" popup before showing you the content of the page, therefore you need to first click on that button.
Also, you should use the find_elements_by_xpath function directly on driver instead of using the page content, as this also allows you to render the javascript content. You can find more info here: python tree.xpath return empty list
To get more info on how to scrape using selenium and python you could check out this guide: https://www.webscrapingapi.com/python-selenium-web-scraper/
I used the following code to scrape the titles. (I also changed the xpaths to do so, by extracting them directly from google chrome. You can do that by right clicking on an element -> inspect and in the elements tab where the element is, you can right click -> copy -> Copy xpath)
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
# I used these for the code to work on my windows subsystem linux
option = webdriver.ChromeOptions()
option.add_argument('--no-sandbox')
option.add_argument('--disable-dev-sh-usage')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=option)
URL = 'https://www.google.com/travel/flights/searchtfs=CBwQAhopagwIAxIIL20vMHBseTASCjIwMjEtMTItMjNyDQgDEgkvbS8wMWYwOHIaKWoNCAMSCS9tLzAxZjA4chIKMjAyMS0xMi0yN3IMCAMSCC9tLzBwbHkwcAGCAQsI____________AUABSAGYAQE&tfu=EgYIAhAAGAA'
driver.get(URL)
driver.find_element_by_xpath('//*[#id="yDmH0d"]/c-wiz/div/div/div/div[2]/div[1]/div[4]/form/div[1]/div/button/span').click() # this is necessary to pres the I agree button
elements = driver.find_elements_by_xpath('//*[#id="yDmH0d"]/c-wiz[2]/div/div[2]/div/c-wiz/div/c-wiz/div[2]/div[3]/div[3]/c-wiz/div/div[2]/div[1]/div/div/ol/li')
for flight_tree in elements:
title = flight_tree.find_element_by_xpath('.//*[#class="W6bZuc YMlIz"]').text
print(title)
I tried the below code, with screen maximized and having explicit waits and could successfully extract the information, please see below :
Sample code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.get("https://www.google.com/travel/flights/searchtfs=CBwQAhopagwIAxIIL20vMHBseTASCjIwMjEtMTItMjNyDQgDEgkvbS8wMWYwOHIaKWoNCAMSCS9tLzAxZjA4chIKMjAyMS0xMi0yN3IMCAMSCC9tLzBwbHkwcAGCAQsI____________AUABSAGYAQE&tfu=EgYIAhAAGAA")
wait = WebDriverWait(driver, 10)
titles = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//div/descendant::h3")))
for name in titles:
print(name.text)
price = name.find_element(By.XPATH, "./../following-sibling::div/descendant::span[2]").text
print(price)
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Output :
Tokyo
₹38,473
Mumbai
₹3,515
Dubai
₹15,846
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# FIRST - GET THE WEBPAGE
url = "https://op.europa.eu/en/web/who-is-who/organization/-/organization/EP/EP"
driver = webdriver.Chrome("C:/Users/XXX/Downloads/chromedriver_win32/chromedriver.exe")
driver.get(url)
# SECOND - CLICK THE BUTTONS
test = driver.find_element_by_css_selector("op-icon op-icon-more tree-hitarea").click()
print(test)
Now my question is how to be able to use the .click() in selenium to extend this button/icon. Although I find some similar problems, it is never about an icon. I am at loss
driver.find_element_by_css_selector("span.op-icon.op-icon-more.tree-hitarea").click()
Multiple class names in css selector or seperated by a .
to expand the button you will needed to use the xpath of element and than to click it.
Install the chrome addon XPath Helper from chrome and then after refresh click the addon and then shift + click on the element what you neded.
After that use driver.find_element_by_xpath().click()
I want to download user data on Google analytics by using crawler so I write some code using selenium. However, I cannot click the "export" button. It always shows the error "no such element". I tried to use find_element_by_xpath, by_name and by_id.
I upload inspect of GA page below.
I TRIED:
driver.find_element_by_xpath("//*[#class='download-link']").click()
driver.find_element_by_xpath('//*[#id="ID-activity-userActivityTable"]/div/div[2]/span[6]/button')
driver.find_element_by_xpath('//*[#class='_GAD.W_DECORATE_ELEMENT.C_USER_ACTIVITY_TABLE_CONTROL_ITEM_DOWNLOAD']')
Python Code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('/Users/parkjunhong/Downloads/chromedriver')
driver.implicitly_wait(3)
usrid = '1021'
url = 'https://analytics.google.com/analytics/web/#/report/app-visitors-user-activity/a113876882w169675624p197020837/_u.date00=20190703&_u.date01=20190906&_r.userId='+usrid+'&_r.userListReportStates=%3F_u.date00=20190703%2526_u.date01=20190906%2526explorer-
table.plotKeys=%5B%5D%2526explorer-table.rowStart=0%2526explorer-
table.rowCount=1000&_r.userListReportId=app-visitors-user-id'
driver.get(url)
driver.find_element_by_name('identifier').send_keys('ID')
idlogin = driver.find_element_by_xpath('//*[#id="identifierNext"]/span/span')
idlogin.click()
driver.find_element_by_name('password').send_keys('PASSWD')
element = driver.find_element_by_id('passwordNext')
driver.execute_script("arguments[0].click();", element)
#login
driver.find_element_by_xpath("//*[#class='download-link']").click()
#click the download button
ERROR:
Message: no such element: Unable to locate element
inspection of GA
your click element is in an iFrame (iFrame id="galaxyIframe" ...). Therefore, you need to tell the driver to switch from the "main" page to said iFrame. If you add this line of code after your #login it should work:
driver.switch_to.frame(galaxyIframe)
(If the frame did not have a name, you would use: iframe = driver.find_element_by_xpath("xpath-to-frame") and then driver.switch_to.frame(iframe)
To get back to your default frame, use:
driver.switch_to.default_content()
Crawling GA is generally a pain. Not just because you have these iFrames everywhere.
Apart from that, I would recommend looking into puppeteer, the new kid on the crawler block. Even though the prospect of switching to javascript from python may be daunting, it is worth it! Once you get into it, selenium will have felt super clunky.
You can try with the text:
If you want to click on 'Export'-
//button[contains(text(),'Export')]
Windows 10Home x64 /
Python 3.7.0 win64 /
Selenium 3.14.0 /
Geckodriver 3.14.0 /
Firefox 61.0.2.
I'm trying to click on the login button, even though I have copied and pasted the full XPath I am not able to click the Login button.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://library.yonsei.ac.kr/')
linkElem = driver.find_element_by_link_text('LOGIN').click()
username = driver.find_element_by_id('id')
username.send_keys('myuserAbc')
password = driver.find_element_by_id('password')
password.send_keys('mypass123')
linkElem = driver.find_element_by_xpath("/html/body/div[2]/div[2]/div/div[2]/form/fieldset/div[2]/p[#class='loginBtn']/input[#type='submit']").click()
Thank you for your help.
Update1: The script works up to the point of localizing the login button, I don't have problems with Marionette or having the latest Firefox version.
Update2: I added p[#class='loginBtn']/input[#type='submit'] in an effort to make it more specific but still doesn't work.
In my experience i have found that the button click might not work some times on form buttons with type set to action. The best possible way to work through this one will be to use the submit() method.
so something like this should work on your login form
from time import sleep
time.sleep(20)
driver.find_element_by_class_name('loginBtn').submit()
This will submit the form to the server and you will be able to go through the login process.
if that doesn't work you could also try the follwoing
driver.find_element_by_class_name('loginBtn).send_keys(Keys.ENTER)
Now, why Button.click(); not working here could have following
reasons.
1.The button is visible but not enabled.
2.Driver is finding the 2 instances of Button element
.More discussion about this can be found here
Selenium: submit() works fine, but click() does not
As per the HTML you have shared to invoke click() on the element with text as LOGIN you need to induce WebDriverWait for the desired element to be clickable and you can use either of the following solutions:
CSS_SELECTOR:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[title='Login']>span"))).click()
XPATH:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Login']/span"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC