Scraping data from high charts - python

I have two problems: first, I cannot click on the show all bottom; and, second, I cannot get the data from the high chart.
I saw some examples for the high chart on Stack Overflow; however, I did not get how people figure our which JS code to execute.
I tried the following code to achieve that:
from selenium import webdriver
DRIVER_PATH = r"C:\Users\XX\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
url = "https://siterankdata.com/wsj.com"
driver.get(url)
driver.find_element_by_xpath('//*[#id="smallchart"]/div/div/svg/g[17]/g/text/tspan').click() # Does not work I try to click on the show all button.
I would appreciate any help!

To click on show all button Use WebDriverWait() and wait for element_to_be_clickable() and following xpath
driver.get(url)
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[#id='smallchart']//*[name()='svg']/*[name()='g'][17]/*[name()='g']/*[name()='text']/*[name()='tspan']"))).click()
You need to import below libraries
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

The element with text as Show all is a svg element so to click on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://siterankdata.com/wsj.com")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#smallchart svg g text[text-anchor='start'] tspan"))).click()
Using XPATH:
driver.get("https://siterankdata.com/wsj.com")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id="smallchart"]//*[name()='svg']//*[name()='g']//*[name()='text']//*[name()='tspan' and text()='Show all']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on interacting with SVG element in:
How to access to 'rect' type element through Selenium-Python
Clicking on svg using selenium python

Related

Unable to find element using Selenium in Python

I'm trying to click on a button, but selenium can't find the element.
I'm using the xpath:
//button[#title="Monitor Chart(Ctrl+Alt+G)"]
but selenium can't locate. I can find the xpath from Chrome manualy using the find(Ctrl+F) tool, as we can see in the image.
Here is my code and error. The xpath does't have an ID.
Code Snapshot:
To click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.eui-btn.eui-btn-default.eui-btn-normal[title^='Monitor Chart']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='eui-btn eui-btn-default eui-btn-normal' and starts-with(#title, 'Monitor Chart')]"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
hiding_button = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[title ='Monitor Chart(Ctrl+Alt+G)']")))
hiding_button.click()
Thanks to all. But I figure it out the issue. The button was inside an iFrame. So I hadd to switch to the iFrame first.
Code used:
iframe = driver.find_element_by_id('Access_Performance_Monitor')
driver.switch_to.frame(iframe)

How to locate the first channel points button/icon on twitch.tv/esl_csgo using Selenium and Python

On twitch.tv/esl_csgo I want to click on the channel points button/icon but it keeps giving me the Error
Message: no such element: Unable to locate element
I have searched and tried various methods of finding the element for over 4 hours and I have not found a way to click on the element
This is my code but it can not click on the button I want to, help would really be appreciated.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("--user-data-dir=C:\\Users\\Me\\Desktop\\UserData")
options.page_load_strategy = 'normal'
driver = webdriver.Chrome(options=options)
driver.get("https://twitch.tv/esl_csgo")
time.sleep(10)
element = driver.find_element_by_xpath('//*[#id="c7037441c8fd58e7e0ac6326babcf03d"]/div/div[1]/div/div/div/div/div/section/div/div[5]/div[2]/div[2]/div[1]/div/div/div/div[1]/div[2]/button/div/div/div/div[2]/span')
element.click()
To click on the first channel points button/icon you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class^='InjectLayout-sc'] > div span[data-test-selector]"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[starts-with(#class, 'InjectLayout-sc')]/div//span[#data-test-selector]"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Can't seem to select from drop down menu - Python Selenium

As titled, seems like I just can't select the drop down menu from this website no matter what.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
driver=webdriver.Chrome()
driver.get('https://assessing.nashuanh.gov/search.asp')
time.sleep(1)
select=Select(driver.find_element_by_xpath('//*[#id="cboSearchType"]'))
select.select_by_value('2')
First you have to handdle the frames in your page. Also looks like there is no value 2 inside this dropdown, so you have to pass a valid value.
driver.switch_to.frame("middle")
select = Select(driver.find_element_by_xpath('//*[#id="cboSearchType"]'))
select.select_by_value('Parcel')
The <option> element with value/text as Owner within the html-select is within an <frame> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR and select_by_value():
driver.get('https://assessing.nashuanh.gov/search.asp')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='middle']")))
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#cboSearchType")))).select_by_value("Owner")
Using XPATH and select_by_visible_text():
driver.get('https://assessing.nashuanh.gov/search.asp')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//frame[#name='middle']")))
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//select[#id='cboSearchType']")))).select_by_visible_text("Owner")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python

Finding an element and clicking on it using Selenium and Python

I am making a script that will navigate through a website and perform some actions for me. But I am stuck trying to make my script click on the following elemnt:
<img src="../Images/copy.gif">
As there is no ID or class name for this element, i tried to find the element by xpath and clicking on it by using the following code:
import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
....
copy = driver.find_element_by_xpath('//[#id="divGrid"]/div[2]/div[2]/table/tbody/tr[1]/td[1]/div/span/a[2]')
copy.click()
This isn't working, so I am open to suggestions to how to solve this issue.
The desired element is a JavaScript enabled element so to click on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[title='Copy Trip']>img[src$='/Images/copy.gif']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Copy Trip' and contains(#href, 'OnCopy')]/img[contains(#src, '/Images/copy.gif')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to click the button on the following website?

I am new to the world of python and I am trying to select a couple of options on the following website and then click the search button to update results. However, I cannot get the button to respond.
I tried using search button.click() and .submit() and I have tried to implicitly wait. I have also used the code below to wait until the button is clickable. When executing the code, it highlights the button but doesn't seem to release the click; almost like a half click.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Safari()
driver.get('https://leasing.com/personal/car-leasing/')
element = driver.find_element_by_id('selUpfront')
select = Select(element)
select.select_by_value("3")
element = driver.find_element_by_id('selMileage')
select = Select(element)
select.select_by_value("8000")
searchbutton = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "search-button")))
searchbutton.click()
I would expect the search results to be updated with the conditions above.
Seems you were close. To click() on the element you have to induce WebDriverWait for the element to be clickable and you can use either of the following solutions:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#search-button>i.fa.fa-search#search-button-icon"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='search-button']/i[#class='fa fa-search' and #id='search-button-icon']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
There are 2 elements with the search-button you need to use the xpath to locate specific
searchbutton = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='search-button']")))
searchbutton.click()

Categories

Resources