I am trying to determine the number of pages of data generated by the Indian Central Pollution Controal Board. Here is an example of output.
Following https://github.com/RachitKamdar/Python-Scraper, I used selenium/python
maxpage = int(browser.find_elements(By.XPATH,"//*[#id='DataTables_Table_0_paginate']/span/a")[-1].text)
but this produces an empty array. I am really not sure what I am doing wrong. Any help would be greatly appreciated. Thanks
You have to add expected condition to wait until the page loaded the data.
You can wait for visibility of element you are using and after that get it's text, like this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//*[#id='DataTables_Table_0_paginate']/span/a")))
maxpage = int(browser.find_elements(By.XPATH,"//*[#id='DataTables_Table_0_paginate']/span/a")[-1].text)
You might want to try getattribute('textContent')
In your case:
maxpage=browser.find_element_by_xpath("(//*[#id='DataTables_Table_0_paginate']/span/a)[last()]").getattribute('textContent')
Related
I would like to click on one of these tabs but I am not able to access it.
How can I access the data-toggle with Selenium in python?
Please try something like this:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class=='section-tabs']//a[contains(#href,'timeseries-tab')]"))).click()
The code above should be able to select the timeseries tab.
You did not share the xml of the other tab, but I guess this should work:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class=='section-tabs']//a[contains(#href,'heatplot-tab')]"))).click()
This can be also done with CSS Selectors.
To give better answer we need the link to that page or at least the entire XML of that page as a text, not picture
I was hoping for maybe some clarification or suggestions on how to get selenium working with hyperlink. I have tried selecting it by pretty much every element possible. I have also attached a image with the source to look at and compare...
Example of html:
<div class="x-grid3-cell-inner x-grid3-col-ACTION_COLUMN" id="5005a00001rJ22q_ACTION_COLUMN"><span>Edit</span> | </div>
Examples:
#clicky = driver.find_element_by_xpath('//a[contains(#href,"javascript:srcUp(%27%2F5005a00001pKfzm%3Fisdtp%3Dvw%27);")]').click()
#clicky = driver.find_elements_by_partial_link_text('javascript:srcUp(%27%2F5005a00001pKfzm%3Fisdtp%3Dvw%27);').click()
#clicky = driver.find_element_by_xpath("//a[#href='javascript:srcUp(%27%2F5005a00001pKfzm%3Fisdtp%3Dvw%27);']").click()
Try this:
driver.find_element_by_xpath("//div[.//a[contains(#class,'chatterFollowUnfollowAction')]]//a[contains(#href,'javascript:srcUp')]").click()
In order to add an explicit wait before accessing this element use this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[.//a[contains(#class,'chatterFollowUnfollowAction')]]//a[contains(#href,'javascript:srcUp')]"))).click()
Try this:
someElement = driver.find_element_by_xpath("//a[contains(#class, 'chatterFollowUnfollowAction' and #title,'Follow this case')]")
driver.execute_script("arguments[0].click();",someElement)
Using selenium automation webdriver i'm unable to locate the text box element on a travel website using python. Using locator present in the webdriver such as Id/name/css_selector/class_name or xpath/full xpath.
Below is the screenshot of the python code:
[Code_1][1]
While the first one is located the second one isn't. The corresponding HTML code is
[text_box2][2]
How can i fill(automate) both fileds corresponding flight destinations i.e leaving and going
[1]: https://i.stack.imgur.com/gpoIr.jpg
[2]: https://i.stack.imgur.com/Q7mGs.jpg
Here is a slightly different approach for locating things. I am using waits borrowed from CruisePandey in this thread. I used firefox, but that is adaptable. Some notes of what was hard:
I had to make sure to be on the Flights tab.
I had to click in the from and to fields, which were buttons after all, and then wait to be able to type into the revealed input fields.
Finally, I had to choose the first element from the dropdown list that resulted.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox(executable_path='/usr/bin/geckodriver')
driver.maximize_window()
driver.get('https://www.expedia.co.in')
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Flights")))
driver.find_element_by_link_text('Flights').click()
wait.until(EC.presence_of_element_located((By.ID, "location-field-leg1-origin")))
driver.find_element(By.CLASS_NAME,'uitk-faux-input').click()
fieldInput = driver.find_element_by_id('location-field-leg1-origin')
wait.until(EC.visibility_of(fieldInput))
fieldInput.send_keys("SFO")
wait.until(EC.presence_of_element_located((By.TAG_NAME, "strong")))
driver.find_element_by_tag_name('strong').click()
driver.find_elements(By.CLASS_NAME,'uitk-faux-input')[1].click()
fieldInput = driver.find_element_by_id('location-field-leg1-destination')
wait.until(EC.visibility_of(fieldInput))
fieldInput.send_keys("BOS")
wait.until(EC.presence_of_element_located((By.XPATH,"//strong[contains(text(), 'Boston')]")))
driver.find_element_by_xpath("//strong[contains(text(), 'Boston')]").click()
You may want to use the below xpath for going To input field :
//label[text()='Going to']//following-sibling::input[#name]
Code :
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[text()='Going to']//following-sibling::input[#name]"))).send_keys('something')
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
if you want to automate from 1 place holder to another, i have use this set of code & it works for me
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[#id='location-field-leg1-origin-menu']/div[2]/ul/li[1]/button"))).click()
b = wait.until(EC.element_to_be_clickable((By.XPATH, ".//*[#id='location-field-leg1-destination-menu']/div[1]/button"))).send_keys("NYC")
b = wait.until(EC.element_to_be_clickable((By.XPATH,".//*[#id='location-field-leg1-destination-menu']/div[2]/ul/li[1]/button"))).click()
c = wait.until(EC.element_to_be_clickable((By.XPATH, ".//*[#id='d1-btn']"))).click()
I have a Selenium element with href="/daVinci/sys/systems/SK419114/dvmt
I am trying to click on it and go to the next page, my line of code is:
driver.find_element_by_link_texrt(" ").click()
So, what should be in the link text in the double quote?
Find element by link text wont work in this case as you need to know the text for that element like the below example :
Continue
where the following code will work but in your case you do not have such a text based link:
driver.find_element_by_link_texrt("Continue").click()
For your case can use this:
driver.find_element_by_xpath('//a[#href="/daVinci/sys/systems/SK419114/dvmt"').click();
See if this works:-
driver.find_element_by_xpath("//a[contains(#href,'daVinci/sys/systems')]").click()
You may want to try the same with explicit wait :
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(#href, '/daVinci/sys/systems')]"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I've written a script in python using selenium to get the converted value of a certain amount. The amount produces converted value when the earlier is made to put in a placeholder. The newly produced value is found adjacent to the amount. When I put any amount manually in that placeholder, I get a converted value accordingly but when I do the same programmatically, the value remains unchanged and as a result my scraper gets 0 as value. How can I make it work?
Link to that webpage: weblink
The script I've tried with:
from selenium.webdriver import Chrome
from contextlib import closing
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
with closing(Chrome()) as driver:
wait = WebDriverWait(driver, 10)
driver.get("find_the_link_above")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".OrderForm_input-box_XkGmi input[name='amount']"))).send_keys(100)
item = wait.until(EC.presence_of_element_located((By.CLASS_NAME,"OrderForm_total_6EL8d"))).text
print(item)
When I put any amount to the placeholder manually, the change can be seen like below:
But, when I do the same using the script, this is how it looks like:
I've marked the valuees with black color to let you know what I meant.
Problem is that you are sending the value too early so that value is not reflecting after entering the amount value. Here i am waiting for EUR SPREAD element to load before setting the amount value.You can use the same element or any other of your chose but make sure page loads completely with that object and then send the amount value.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = r"path"
driver = webdriver.Chrome(chrome_path)
wait = WebDriverWait(driver, 10)
driver.get("https://www.gdax.com/trade/LTC-EUR")
wait.until(EC.presence_of_element_located((By.XPATH, "//div[#class='OrderBookPanel_text_33dbp']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//input[#name='amount']"))).send_keys(100)
item = wait.until(EC.presence_of_element_located((By.CLASS_NAME,"OrderForm_total_6EL8d"))).text
print(item)
Hope this will solve your problem.