Develop Reference

Webscraping Click Button Selenium

I am trying to webscrape indeed.com to search for jobs using python, with selenium and beautifulsoup. I want to click next page but cant seem to figure out how to do this. Looked at many threads but it is unclear to me which element I am supposed to perform on. Here is the web page html and the code marked with grey comes up when I inspect the next button.
Also just to mention I tried first to follow what happens to the url when mousedown is executed. After reading the addppurlparam function and adding the strings in the function and using that url I just get thrown back to page one.
Here is my code for the class with selenium meant to click on the button:
from selenium import webdriver
from selenium.webdriver import ActionChains
driver = webdriver.Chrome("C:/Users/alleballe/Downloads/chromedriver.exe")
driver.get("https://se.indeed.com/Internship-jobb")
print(driver.title)
#assert "Python" in driver.title
elem = driver.find_element_by_class_name("pagination-list")
elem = elem.find_element_by_xpath("//li/a[#aria-label='Nästa']")
print(elem)
assert "No results found." not in driver.page_source
assert elem
action = ActionChains(driver).click(elem)
action.perform()
print(elem)
driver.close()

The indeed site is formatted so that it shows 10 per page.
Your photo shows the wrong section of HTML instead you can see the links contain start=0 for the first page, start=10 for the second, start=20 for the third,...
You could use this knowledge to do a code like this:
while True:
i = 0
driver.get(f'https://se.indeed.com/jobs?q=Internship&start={i}')
# code here
i = i + 10
But, to directly answer to your question you should do:
next_page_link = driver.find_element_by_xpath('/html/head/link[6]')
driver.get(next_page_link)
This will find the link and then get it.

its work. paginated to next page.
driver.find_element_by_class_name("pagination-list").find_element_by_tag_name('a').click()

Is there a way to block/close iframes using seleium?

I am creating a downloader from Instagram. This program gets URLs from top Instagram posts in a given hashtag and inputs them into a downloader. The issue is that a popup iframe ad always appears over the download button when the website is first loaded. This throws an error that the button cannot be clicked because the iframe will be clicked.
This is for Python Selenium running Chrome driver. I have tried to run a filter that finds iframes and goes back to the main page:
all_iframes = self.browser.find_elements_by_tag_name("iframe")
if len(all_iframes) > 0:
self.browser.switch_to.default_content()
This did not work, I also tried to get the XPath to the X-button on the ad, but the ID changes every time, so I cannot be clicked or identified.
#get the website
for link in self.links:
self.browser.get('https://downloadgram.com/')
time.sleep(5)
#this is the x button click iframe I tried, but the XPath changes
all_iframes = self.browser.find_elements_by_tag_name("iframe")
if len(all_iframes) > 0:
xButton = self.browser.find_element_by_xpath('//div[#id="id3019a64023cross3019a64023"]')
xButton.click()
#inputs the URL from array links[] into download box
input = self.browser.find_element_by_xpath('//input[#name="url"]')
input.clear()
input.send_keys(link)
time.sleep(1)
#clicks download button
download = self.browser.find_element_by_xpath("//input[#type='submit']")
download.click()
time.sleep(1)
#clicks confirm button
actuallyDownload = self.browser.find_element_by_xpath("//a[#target='_blank']")
actuallyDownload.click()
time.sleep(1)
I expect the code to download the pictures at the url, but I get:
selenium.common.exceptions.WebDriverException: Message: unknown error: Element <input type="submit" value="Download" class="button"> is not clickable at point (451, 446). Other element would receive the click:
the website (turn off adblock to see add) https://downloadgram.com/

I am able to close the pop up with following code.Please try that.
browser.get('https://downloadgram.com/')
time.sleep(5)
element=browser.find_element_by_xpath("//div[starts-with(#id,'id')]" and "//div[starts-with(#style,'position:absolute !important;height:20px !important;width:20px !important;top:3px !important;left:3px !important;background-image:url(data:image/png;')]")
arrt=element.get_attribute("id")
print(arrt)
browser.execute_script("arguments[0].click();", element)
Let me know if it works.Good Luck.

How to click the icon using selenium

I am trying to automate my certain activities using selenium. I was launching a webpage and a security windows popup appears for the login credentials.
I have automated that using Autoit. Now after login I need to click on the option and I have tried it based on the find_element_by_text and find_element_by_id.
I was getting an error as Unable to find the element with CSS selector and I have seen some other post in the StackOverflow with the same issue but I could not able to fix it by myself.
Here is how my HTML looks like. Could you please guide me on this and also please share any document for further checking. Thanks.
driver = webdriver.Ie(r"C:\\IEDriverServer\\IEDriverServer.exe")
driver.get('URL of the page')
#driver.implicitly_wait(10) # seconds
autoit.win_wait("Windows Security")
# now make that crap active so we can do stuff to it
autoit.win_activate("Windows Security")
# type in the username
autoit.send('username')
# tab to the password field
autoit.send("{TAB}")
# type in the password
autoit.send('password')
# kill!
autoit.send("{ENTER}")
driver.maximize_window()
driver.implicitly_wait(120) # seconds
#submit_button_incidents = driver.find_element_by_link_text("3-Normal Incidents")
submit_button_incidents= driver.find_element_by_id("nodeImgs13pm")
submit_button_incidents.click()
driver.implicitly_wait(10)
++ updating the info
I have tried to copy the whole HTML but the page was restricted so I cant able to view the full HTML page other than the basic templates. Adding some more screenshots of the developer tools.
Here how my webpage looks like.

try with this code :
submit_button_incidents = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'My Group'])")))
submit_button_incidents.click()
and do not use implicit wait too many times in code.
Implicit wait is set for life time of web driver instance.
Reference :
Selenium official document for wiats
Xpath tutorial
cssSelector tutorial
UPDATE :
As OP has shared the HTML code. As per the requirement you can go ahead with this code.
As there are two elements with text as My Groups's queue.
For first one you can write XPATH as :
//a[#class='scbdtext']/span[contains(text(),'My Group')]
Code:
submit_button_incidents = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='scbdtext']/span[contains(text(),'My Group')]")))
submit_button_incidents.click()
For second one you can write XPATH as :
//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]
Code :
submit_button_incidents = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]")))
submit_button_incidents.click()
Hope this will help.

Use ActionChains with double click for this to work in Python.
from selenium.webdriver import ActionChains
# Get the element however you want
element = driver.find_element_by_xpath("//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]")
ActionChains(driver).double_click(settings_icon).perform()

Not able to click a button in Selenium (Booking.com)

I am programming a python scraper with help of Selenium. The first few steps are:
goes on booking.com, insert a city name, selects the first date and then tries to open the check-out calendar.
Here is where my problem occurs. I am not able to click the check-out calendar button (The important are of the website).
I tried to click every element regarding to the to check-out calendar (The elements of check-out calendar) with element.click(). I also tried the method
element = self.browser.find_element_by_xpath('(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]') self.browser.execute_script("arguments[0].click();", element)
It either does nothing (in case of execute.script() and click() on div elements) or it throws following exception when directly clicking the button:
Element <button class="sb-date-field__icon sb-date-field__icon-btn bk-svg-wrapper"
type="button"> is not clickable at point (367.5,316.29998779296875)
because another element <div class="sb-date-field__display"> obscures it
Here is a short code to test it:
browser = webdriver.Firefox()
browser.get("https://www.booking.com/")
wait = WebDriverWait(browser, 5)
element = wait.until(EC.presence_of_element_located((
By.XPATH, '(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')))
element = wait.until(EC.element_to_be_clickable((
By.XPATH, '(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')))
element.click()
I have a temporarily solution for my problem but I am not satisfied with it.
element = browser.find_element_by_xpath('(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')
hov = ActionChains(browser).move_to_element(element)
hov.click().perform()
This will open the calendar by hovering over the object and clicking it. This strangely opens the calendar.
The methods mentioned above still don't work.

Define clicka as an xpath. Now use executescript to click the element.
driver.execute_script("arguments[0].click();", clicka)

I'm not 100% sure that I got everything you posted, because the layout is a bit messy.
However, I tried to test the issue with both Selenium Java and Firefox Scratchpad (a Web Developer tool that allows to run JavaScript scripts) and it worked perfectly - the button was clickable on both of them.
If you're interested in further testing using this tool, this is the code I've used:
In JavaScript:
function getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
var myElement = getElementByXpath('(//div[contains(#class,"checkout-field")]//button[#aria-label="Open calendar"])[1]')
myElement.click()
and in Java:
FirefoxDriver driver = new FirefoxDriver();
WebDriverWait wait = new WebDriverWait(driver, 10);
driver.navigate().to("https://www.booking.com");
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("(//div[contains(#class,'checkout-field')]//button[#aria-label='Open calendar'])[1]")));
driver.findElement(By.xpath("(//div[contains(#class,'checkout-field')]//button[#aria-label='Open calendar'])[1]")).click();
System.out.println("success");

if your are having the control check out button on all the web site managing with explicit wait required lots of codding you can use implicit wait below is in the java.
System.setProperty("webdriver.chrome.driver",
"G:\\TopsAssignment\\SampleJavaExample\\lib\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);

Python+Selenium, can't click the 'button' wrapped by span

I am new to selenium here. I am trying to use selenium to click a 'more' button to expand the review section everytime after refreshing the page.
The website is TripAdvisor. The logic of more button is, as long as you click on the first more button, it will automatically expand all the review sections for you. In other words, you just need to click on the first 'more' button.
All buttons have a similar class name. An example is like taLnk.hvrIE6.tr415411081.moreLink.ulBlueLinks. Only the numbers part changes everytime.
The full element look like this:
<span class="taLnk hvrIE6 tr413756996 moreLink ulBlueLinks" onclick=" var options = {
flow: 'CORE_COMBINED',
pid: 39415,
onSuccess: function() { ta.util.cookie.setPIDCookie(2247); ta.call('ta.servlet.Reviews.expandReviews', {type: 'dummy'}, ta.id('review_413756996'), 'review_413756996', '1', 2247);; window.location.hash = 'review_413756996'; }
};
ta.call('ta.registration.RegOverlay.show', {type: 'dummy'}, ta.id('review_413756996'), options);
return false;
">
More </span>
I have tried several ways to get the button click. But since it is an onclick event wrapped by span, I can't successfully get it clicked.
My last version looks like this:
driver = webdriver.Firefox()
driver.get(newurl)
page_source = driver.page_source
soup = BeautifulSoup(page_source)
moreID = soup.find("span", class_=re.compile(r'.*\bmoreLink\b.*'))['class']
moreID = '.'.join(moreID[0:(len(moreID)+1)])
moreButton = 'span.' + moreID
button = driver.find_element_by_css_selector(moreButton)
button.click()
time.sleep(10)
However, I keep getting the error message like this:
WebDriverException: Message: Element is not clickable at point (318.5,
7.100006103515625). Other element would receive the click....
Can you advise me on how to fix the problem? Any help will be appreciated!

WebDriverException: Message: Element is not clickable at point (318.5, 7.100006103515625). Other element would receive the click....
This error to be occur when element is not in the view port and selenium couldn't click due to some other overlay element on it. In this case you should try one of these following solution :-
You can try using ActionChains to reach that element before click as below :-
from selenium.webdriver.common.action_chains import ActionChains
button = driver.find_element_by_css_selector(moreButton)
ActionChains(button).move_to_element(element).click().perform()
You can try using execute_script() to reach that element before click as :-
driver.execute_script("arguments[0].scrollIntoView(true)", button)
button.click()
You can try using JavaScript::click() with execute_script() but this JavaScript::click() defeats the purpose of the test. First because it doesn't generate all the events like a real click (focus, blur, mousedown, mouseup...) and second because it doesn't guarantee that a real user can interact with the element. But to get rid from this issues you can consider it as an alternate solution.
driver.execute_script("arguments[0].click()", button)
Note:- Before using these options make sure you're trying to interact with correct element using with correct locator, otherwise WebElement.click() would work well after wait until element visible and clickable using WebDriverWait.

Try using an ActionChains:
from selenium.webdriver.common.action_chains import ActionChains
# Your existing code here
# Minus the `button.click()` line
ActionChains(driver).move_to_element(button).cli‌​ck().perform()
I have used this technique when I need to click on a <div> or a <span> element, rather than an actual button or link.