I am trying to get the google map embed url using selenium. I am able to click the share button and the page shows a modal with share url and an embed url. However i am un able to switch the dialog box.
Here is my code
browser.get('https://www.google.com/maps/place/%s?hl=en'%(code))
time.sleep(3)
share_class = "ripple-container"
buttons = browser.find_elements_by_class_name(share_class)
for but in buttons:
x = but.text
if x == 'SHARE':
but.click()
modal = browser.switch_to.active_element
share = modal.find_element_by_id("modal-dialog")
print(share.text)
here is the image.
You don't need to switch to the modal dialog, you can access it just like you would any other HTML on the page. You can simplify your code to
browser.get('https://www.google.com/maps/place/%s?hl=en'%(code))
browser.find_element_by_xpath("//button/div[.='SHARE']").click()
url = browser.find_element_by_id("last-focusable-in-modal").text
print(url)
But... if you read the dialog, you will see that it states
You can also copy the link from your browser's address bar.
so the URL you are navigating to in the first line is what you are going to copy from the Share link so there's really no point. You already have the URL.
Related
I'm trying to use pd.read_html() to read the current page I'm trying to scrape using Selenium.
The only problem is the web page does not contain a table until you press a few buttons using selenium button click and then the table is displayed.
So when I input an argument:
pd.read_html('html_string')
It gives me an error.
Is there a way to read in the current page after the buttons have been clicked and not just putting in the html string as an argument?
I've also looked at the documentation for this and could not find anything to help.
Thanks for reading/answering
I would try to pass a page source instead of an address when the source is updated:
url = ...
button_id = ...
driver.get(url)
button = driver.find_element(by=button_id)
button.click()
... # wait?
data = pd.read_html(driver.page_source)
I am having some trouble scraping the url below:
http://102.37.123.153/Lists/eTenders/AllItems.aspx
I am using Python with Selenium, but have many "onclick" javascript events to run to get to lowest level of information. Does anyone know how to automate this?
Thanks
url = 'http://102.37.123.153/Lists/eTenders/AllItems.aspx'
chrome_options = Options()
chrome_options.add_argument("--headless")
browser = webdriver.Chrome('c:/Users/AB/Dropbox/ITProjects/Scraping/chromedriver.exe', options=chrome_options)
res = browser.get(url)
time.sleep(10)
source = browser.page_source
soup = BeautifulSoup(source)
for link in soup.find_all('a'):
if link.get('href') == 'javascript:':
print(link)
You don't need selenium with this website, you need patience. Let me explain how you'd approach that.
Click X
Y opens, click Y
Z opens, click Z.
Goes on..........
What happened here is that when you've clicked X, an AJAX request was made to get Y and after you click Y, another AJAX was made to get Z and then this goes on.
So you can just simulate those requests, open the networks tab and see how does it craft the requests then make the same ones in your code then get the response, based on it, do the next request and the cycle will go on till you get to the innermost level of the tree.
This approach has no UI and is technically-speaking, more unfriendly and harder to implement. But it's more efficient, on the other side, you can just select your clickable elements with selenium like
eleme = driver.find_elemnent_by_x('x')
elem.click()
And it will also work
I'd also note that sometimes, links don't AJAX, they just hide the info but it's in the source code. To know what you'll recieve in your response, R-click in the website and choose View page source and note that this is different than inspect element.
I am new to Selenium and I am trying to mimic user actions on a site to fetch data from a built in html page on button click. I am able to populate all the field details, but button click is not working, it looks like js code not running.
I tried many options like adding wait time, Action chain etc but it didnt work, i am providing site and code i have written.
driver = webdriver.Chrome()
driver.get("https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm")
driver.implicitly_wait(10)
assigned values to all the other fields
driver.find_element_by_id('rdDateToDate').click()
Dfrom = driver.find_element_by_id('fromDate')
Dfrom.send_keys("02-Oct-2020")
Dto = driver.find_element_by_id('toDate')
Dto.send_keys("08-Oct-2020")
innerHTML = driver.execute_script("document.ready")
sleep(5)
getdata_btn = driver.find_element_by_id('getButton')
ActionChains(driver).move_to_element(getdata_btn).click().click().perform()
I recommend using a full xpath.
chrome.get("https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm")
time.sleep(2)
print("click")
fullxpath = "/html/body/div[2]/div[3]/div[2]/div[1]/div[3]/div/div[1]/div/form/div[19]/p[2]/input"
chrome.find_element_by_xpath(fullxpath).click()
I have tried the button clicking and it worked with XPath ... I though its because of someone used the ID twice on a website, but I can not find it ... so i have no idea whats going wrong there ...
Good luck :)
I am trying to scrap something from website (example facebook(not using graph api just doing for learning), so I successfully login and land on front page, where I want to scrap some data, but the problem is when I land on front page, then facebook shows a layer and a box which says "turn on notification", now without click on any button between "Not Now" or "turn on" I can't do anything with splinter, and when I tried to click splinter doesn't do anything because the link of those button are "#"
when hovering on button footer shows this :
and inspect element shows this :
I tried with other account but that shows this layer as first thing after login :
Now I have question how to click on these 2 types of button via splinter or selenium :
first type of button which shows "#" as href
second which chrome shows for block, allow things
My code is :
from selenium import webdriver
from splinter import Browser
web_driver=webdriver.Chrome('/Users/paul/Downloads/chromedriver/chromedriver')
url = "https://www.example.com"
browser = Browser("chrome")
visit_browser = browser.visit(url)
email_box = '//*[#id="email"]'
find_1 = browser.find_by_xpath(email_box)
find_1.fill("example#gmail.com")
password_box = '//*[#id="pass"]'
find_2 = browser.find_by_xpath(password_box)
find_2.fill("example12345")
button_sub = '//*[#id="u_0_5"]'
find_3 = browser.find_by_xpath(button_sub)
find_3.click()
for testing purpose you can try on "see more button" in trending section on facebook, that also shows "#" how to click that ?
Not letting me comment because I don't have enough rep ... but have you tried to select the element by class and then performing .click() on it? That might do the trick as the href being "#" probably means the button has another purpose.
I have solved my problem , since that link was "#" so if i was click via css or other method it was just reloading the page and that layer appear again and again after every reload , But i tried little different solution and i click it by javascript :
First i tried and find the right element for clicking via js console in chrome :
document.getElementsByClassName('layerCancel _4jy0 _4jy3 _517h _51sy _42ft')[0].click();
This is working perfect in js console so now i used splinter method "browser.execute_script()" and pass that script as argument to this method.
browser.execute_script("document.getElementsByClassName('layerCancel _4jy0 _4jy3 _517h _51sy _42ft')[0].click()")
And its working perfect now as i wanted. But still i have not found a way how to click on browser push notification 'Allow" , "Block" etc
Thanks :)
I am trying to automate a booking in process on a travel site using
splinter and having trouble clicking on a css element on the page.
This is my code
import splinter
import time
secret_deals_email = {
'user[email]': 'adf#sad.com'
}
browser = splinter.Browser()
url = 'http://roomer-qa-1.herokuapp.com'
browser.visit(url)
click_FIND_ROOMS = browser.find_by_css('.blue-btn').first.click()
time.sleep(10)
# click_Book_button = browser.find_by_css('.book-button-row.blue-btn').first.click()
browser.fill_form(secret_deals_email)
click_get_secret_deals = browser.find_by_name('button').first.click()
time.sleep(10)
click_book_first_room_list = browser.find_by_css('.book-button-row-link').first.click()
time.sleep(5)
click_book_button_entry = browser.find_by_css('.entry-white-box.entry_box_no_refund').first.click()
The problem is whenever I run it and the code gets to the page where I need to click the sort of purchase I would like. I can't click any of the option on the page.
I keep getting an error of the element not existing no matter what should I do.
http://roomer-qa-1.herokuapp.com/hotels/atlanta-hotels/ramada-plaza-atlanta-downtown-capitol-park.h30129/44389932?rate_plan_id=1&rate_plan_token=6b5aad6e9b357a3d9ff4b31acb73c620&
This is the link to the page that is causing me trouble please help :).
You need to whait until the element is present at the website. You can use the is_element_not_present_by_css method with a while loop to do that
while not(is_element_not_present_by_css('.entry-white-box.entry_box_no_refund')):
time.sleep(50)