I should start by saying that I'm not a web developer but I have been using Selenium with Python for a few weeks and think I've gotten the basics down.
What I have is a page with a weekly calendar on it. Image as follows:
What happens is that you can click on any of the coloured boxes which will bring up a register for a class. It features items that you can click on which bring up new information. The problem I have is that I can't click on the items - or, rather, I don't know how to automate the click. I have automated clicking with Selenium before in other situations.
That looks as follows:
As you can see, the calendar still appears in the background. That is to say, the item we click on doesn't take us to a new page but, I suppose, runs some kind of method which shows the register and populates it with data.
My problem is this: I want to automate the process so that I can click on each of these items then scrape the information. In order to do that I need to be able to automate the clicking for each of the items.
So what have I done? When I've done this before, I've searched through the web html for the relevant part and then grabbed the xPath to the element I needed. But here I can't do that. Why not? Well, firstly, I just can't find that element!
Take a look at this close up of the first column:
It's divided into columns, but then I'd expect the clickable area to be an element within that. As you can see, it's not. Furthermore, the clickable area is just the coloured box itself, but if you look closely you can see the element goes outside of that area. I have gone very close with my mouse cursor to see exactly what's clickable, and it definitely is just the coloured box.
So I've not been able to get the element at all.
I thought I might be able to just find out where we went after I clicked the button, but when I got the link address, it just said it was the same page with no differences.
I appreciate I'm asking quite a broad question here, but the problem is that I don't really know where to start. If someone could give me at least that much, I would be grateful. Like if I could just click on each of these one at a time... I've found where the populated data is so I could grab that without a problem.
Well, here's to hoping.
Edit: I should add that there are some JavaScript items (tag type script, type='text/javascript'). I presume that the answer is in there somewhere, but there is a lot of Javascript and I'm not adept at reading it. It's hard for me to tell what script does what. If I could at least figure out what script runs when I click the item then I think I'd be onto something, but I have no idea. Even that would help me.
I had encountered similar problem when scraping for Instagram followers in mobile view where it was floating box when showing the accounts followers name. The approach I took was identifying the name of floating dialog box and clicking the element in it. It might different in your case of html.
Trying looking at this link. Selenium Scroll inside of popup div
Hard to say without the HTML. Maybe try Katalon Recorder (chrome extension) and see if that can detect the xpath for you? It might also be you have to use some kind of javascript to invoke the method for the element
Related
I am writing a program that answers questions from a database. I am currently on the last question type (ordering questions). Here is an image of what the question looks like. The user needs to put the elements in the box in order.
As you can see, the bottom of the question gets cut off. This caused errors when trying to click certain elements that were below the page. To try and solve this I scrolled to the bottom of the page but selenium scrolled back to the top before clicking which kept the error in place. And after trying a few other things I decided to try and zoom out on the webpage to see them all at once.
This is the HTML of the list where the <p> tag is what I am trying to click on. Actually, I am trying to right-click but same thing. The rest of the <div>s with the ellipses are the rest of the boxes and are the same as the first one.
newheader = self.driver.find_element(By.XPATH, "//div[contains(#class, 'responses-container')]")
action = ActionChains(self.driver)
action.context_click(WebDriverWait (newheader, 10).until(EC.presence_of_element_located((By.XPATH, "//p[contains(text(), '"+ dictionary_.get(prompt)[r] +"')]")))).perform()
# dictionary.get(prompt)[r] contains the values we are looking for (ovulation, fertilization, etc)
# I have also tried EC.element_to_be_clickable and had the same results
I have tested this on a question that does not overflow the page and it works perfectly. The only time I am getting errors is when I try to zoom out. I have tried refreshing the page before starting the clicking and after zooming out. I have also tried resetting the driver by calling
self.driver.get(self.driver.current_url)
Each of those elements cannot be left-clicked on to select. They can only be right-clicked or tabbed to. Tabbing requires calculations that are complex since each element is being moved dynamically so I chose to right-click instead. It does right-click but in seemingly random places now that it's zoomed out.
Any help is appreciated! Thanks!
Apologies if this question was answered before, I want to click on an area in a browser with plain text using Selenium Webdriver in python
The code I'm using is:
element_plainText = driver.find_elements(By.XPATH, '//*[contains(#class, "WgFkxc")]')
element_plainText.click()
However this is returning "ElementNotInteractableException". Can anyone help me out with this?
Selenium is trying to be helpful here, by telling you why it won't click on the element; ElementNotInteractableException means it thinks that what you're trying to click on isn't clickable.
This usually happens because either:
The element isn't actually visible, or is disabled
Another element is "overlapping" the element, possibly invisibly
You're clicking something Selenium thinks won't do anything, like plain text
There's two things I'd try to get around this. Firstly, Actions. Selenium has an Action API you can use to cause specific UI events to occur. I'd suggest finding the co-ordinates of the text, then making Selenium click those co-ordinates instead of telling it to click the element. Read more about that API here.
Secondly, try clicking it with Javascript, using a Javascript Executor. That can often give you the same outcome as using Selenium directly, without it being so "helpful".
I am trying to grab a bunch numbers that are presented in a table on a web page that I’ve accessed using python and Selenium running headless on a Raspberry Pi. The numbers are not in the page source, rather they are deeply embedded in complex html served by several URLs called by the main page (the numbers update every few seconds). I know I could parse the html to get the numbers I want, but the numbers are already sitting on the front page in perfect format all in one place. I can select and copy the numbers when I view the web page in Chrome on my PC.
How can I use python and get Selenium webdriver to get me those numbers? Can Selenium simply provide all the visible text on a page? How? (I've tried driver.page_source but the text returned does not contain the numbers). Or is there a way to essentially copy text and numbers from a table visible on the screen using python and Selenium? (I’ve looked into xdotool but didn’t find enough documentation to help). I’m just learning Selenium so any suggestions will be much appreciated!
Well, I figured out the answer to my question. It's embarrassingly easy. This line gets just what I need - all the text that is visible on the web page:
page_text = driver.find_element_by_tag_name('body').text
So, there are some different situations why you can not get some info on the page:
Information doesn't loaded yet. You must waiting for some time to get your information ready. You may watch this theme for the better understanding. Some times you get dynamically added page elements with JS and so on, which loading is very slowly.
Information may consists of different type of data. For example you are waiting for a text with numbers, but you may get picture with numbers on the page. In this situation you must change your programming tactics and use another functions to get what you need.
I would like to know if there is a way to get locator for any text in robot framework ? I am using Robot Framework with Selenium2Library.
Scenario example : Suppose I have message "Hello" on my page and its position keeps on changing when new message appears.I want to click on this "Hello" word to show complete message. How can I do that. Please help.
I've seen locations change a lot with Angular webpages, so I understand what you mean. IDs can be generated anew every time the page loads and may change depending on the order in which you click on elements. Worse, they can affect dom- and xpath-based locators as well. The short answer to your question is no, it doesn't exist within standard Selenium, Selenium2, ExtendedSelenium2, or any other standard library.
The medium answer to your question is solved in my currently-most-advanced Click Element by Text custom keyword. It generates an xpath based on your inputs and if it can be found with a simple text search with no extra parameters, it'll do it. It is testy at times and may require argument injections to work, but in general it works very well and is easy to use when xpaths are remotely possible.
Click Element by Text
# EXAMPLE USAGE
# Click Element by Text "text on the element" id="midlevellocationoftext" button
# NOTE: Does not account for extra spaces at the beginning or end, text must be exact
# NOTE: Allows for injections on purpose to allow user to be more exact with their location
[Arguments] ${text} ${location}=* ${elementtype}=*
Click Element xpath=//*[#${location}]//${elementtype}[text()=${text}]
The long answer is that I'm working on something better. Here's what I know so far. The elements to do this do exist out there in the void. JavaScript can turn a webpage into a list of elements. I can call .innerHTML on those elements to get their text. I haven't worked out the details, but I honestly think it's possible and when I get it figured out, the code will be on Code Review for others to see.
I am trying to use Selenium with Python to click on a text field, which opens a pop-up panel, select the text entry area of that popup, and enter text to it.
switch_to_window and switch_to_frame don't seem to be working. In a previous question I asked about Selenium, someone told me to pause the program until the element I need is available. The solution worked for that problem, but not this one, so I'm assuming I have a different issue and I'm too new to Selenium to understand what it is.
This is what the original box I'm trying to click on looks like:
And the Inspect Element for this box:
When that description box is clicked, it should open this window:
And select this element to enter text into:
So in my code I have:
descriptionBox = driver.find_element_by_id('kiadvany_fulszoveg_text')
descriptionBox.click()
That does not error the program, but it also doesn't seem to actually be clicking on that element. To make matters more confusing, I got this to work exactly ONCE, where it opened the correct Description text box as pictured above, but it has since not worked at all even when I try the exact same thing.
The panel's ID is:
As I mentioned, switching to this panel ID using switch_to_frame or switch_to_window was the first thing I tried, but I'm getting a No Such Element error.
Because I saw the description box open correctly once, but never again, I'm assuming that's where the problem is. I wish, the one time it did pop up, that I'd tried to put the text into the field to see if that would work too, but I hadn't gotten there yet at that point, so I don't know if that would have worked.
Thank you in advance to anyone who can help with this!
Try this
descriptionBox = driver.find_element_by_id('kiadvany_fulszoveg_text')
driver.execute_script('arguments[0].click();', descriptionBox)
or
actions = ActionChains(driver)
actions.move_to_element(descriptionBox)
actions.click(descriptionBox)
actions.perform()