I'm searching for a python module that simulates a webbrowser and can handle html frames. I want to use the chatbot Brain http://www.thebot.de/ with python. If you know any tutorials that explain how to use your suggested module in relation with forms and frames give me a links to them please.
Tim,
I suggest you take a look at Selenium. By default it opens and manipulates Firefox to navigate the web. Its primary use case is testing, however in a pinch I've used it in some of my scripts to get past sites with a lot of javascript or, as in this case, iframes.
The basic usage is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
driver.close()
To access an iframe you can use the following code:
elm = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(elm)
Then when you want to switch out of the iframe:
driver.switch_to.default_content()
You can even utilize Selenium headless with xvfbwrapper like this (code from github.com/cgolberg/xvfbwrapper):
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
# launch stuff inside virtual display here
vdisplay.stop()
Related
I am trying to use Selenium to log into a printer and conduct some tests. I really do not have much experience with this process and have found it somewhat confusing.
First, to get values that I may have wanted I opened the printers web page in Chrome and did a right click "view page source" - This turned out to be not helpful. From here I can only see a bunch of <script> tags that call some .js scripts. I am assuming this is a big part of my problem.
Next I selected the "inspect" option after right clicking. From here I can see the actual HTML that is loaded. I logged into the site and recorded the process in Chrome. With this I was able to identify the variables which contain the Username and Password. I went to this part of the HTML did a right click and copied the Xpath. I then tried to use the Selenium find_element_by_xpath but still no luck. I have tried all the other methods to (find by ID, and name) however it returns an error that the element is not found.
I feel like there is something fundamental here that I am not understanding. Does anyone have any experience with this???
Note: I am using Python 3.7 and Selenium, however I am not opposed to trying something other than Selenium if there is a more graceful way to accomplish this.
My code looks something like this:
EDIT
Here is my updated code - I can confirm this is not just a time/wait issue. I have managed to successfully grab the first two outer elements but the second I go deeper it errors out.
def sel_test():
chromeOptions = Options()
chromeOptions.add_experimental_option("useAutomationExtension", False)
browser = webdriver.Chrome(chrome_options=chromeOptions)
url = 'http://<ip address>/'
browser.get(url)
try:
element = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="ccrx-root"]')))
finally: browser.quit()
The element that I want is buried in this tag - Maybe this has something to do with it? Maybe related to this post
<frame name="wlmframe" src="../startwlm/Start_Wlm.htm?arg11=">
As mentioned in this post you can only work with the current frame which is seen. You need to tell selenium to switch frames in order to access child frames.
For example:
browser.switch_to.frame('wlmframe')
This will then load the nested content so you can access the children
Your issue is most likely do to either the element not loading on the page until after your bot searches for it, or a pop-up changing the xpath of the element.
Try this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
delay = 3 # seconds
try:
elementUsername = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.xpath, 'element-xpath')))
element.send_keys('your username')
except TimeoutException:
print("Loading took too much time!")
you can find out more about this here
I have some problems trying to automate repetitive tasks in my company's intranet website.
While I can properly find elements in other websites (clicking, sending keys etc.), I can't do the same in this website (the error I always get is "unable to find element with xpath == ... ").
I tried to do the same tasks recording them with Selenium IDE, and it works well.
If I export the Selenium IDE test case as python code and I compare the xpath command with mine, the xpath is the same, and it's the same that I get using the firefox selector to the point I want to click.
Since for other reasons I must work with explorer (so I have the explorer webdriver etc.), I haven't tried to run the code with python+(webdriver for Firefox).
Code:
from selenium import webdriver
import time
driver=webdriver.IE("D:\myfolder\IEDriverServer.exe")
driver.implicitly_wait(30)
base_url="https://companywebsite.ext")
driver.get(base_url) #it's ok until now, I can open the webpage
time.sleep(20)
element=driver.find_element_by_xpath("/html/body/div[2]/div/table/tbody/tr[3]")
time.sleep(10)
Here are my questions:
1) Why can I do things right in other websites and not in this?
2) Why Selenium IDE is able to do that and, with the same commands, it doesn't work with my python script?
3) Do I have to do particular setups?
Thank you very much.
You should try using wait's:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver=webdriver.IE("D:\myfolder\IEDriverServer.exe")
driver.implicitly_wait(30)
base_url="https://companywebsite.ext")
driver.get(base_url)
wait = WebDriverWait(driver, 10)
element = wait.until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[2]/div/table/tbody/tr[3]")))
Hope this helps!
So something interesting is happening - unfortunately, I cannot post a url or something you guys can access, so maybe someone just has an idea of what the cause could be. I'd assume it would be something in development. Either way...
I have a test that logs in to an application, which will then load up a dashboard and search for some data. In Firefox, everything works fine. However, IE11 is having an issue once it logs in. So, Selenium works fine up until it loads the dashboard, where it will then find NOTHING. Not even "//html". It gives the basic element not found error. I'm assuming the developers have written something that prevents access, but the one I talked to wasn't sure.
Any idea of what could be causing this so that I could go suggest something to the developers? Again, this doesn't happen in Firefox, which could also suggest something's up with IE11 (though Selenium works fine on other applications and up until the Dashboard screen on this one). If you need more information, let me know - just be aware that I might not have much to give.
A couple things to note:
1. it is https protocol, but so is the login page
2. at the end of the url bar, it says there is a certificate error that will not go away
no matter what I do (I have added the supposedly correct certificates).
What I'm using:
Selenium 2.4.5
32 bit IEDriverServer
Python 2.7.8
IE11 (other IE versions are not supported, so I cannot compare)
Windows 7
EDIT:
I talked with a developer, and the best thing he could think of was the fact that the dashboard heavily uses polyfills, and that maybe that is the reason that Selenium isn't finding anything. With that being said, is there a way to find elements using Selenium when polyfills are involved? Or is Selenium just not going to work for it.
MORE:
Here is some code that equivalent to what my test would do. Assume that every exception would normally be caught and wouldn't kill the script...
from pdb import set_trace
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
url = 'server.url'
capabilities = webdriver.DesiredCapabilities().INTERNETEXPLORER
capabilities['acceptSslCerts'] = True
# use path due to having both a 64 bit and 32 bit version of the driver - specifies which to use
driver = webdriver.Ie("path-to-iedriver.exe", capabilities=capabilities)
driver.maximize_window()
driver.get("https://%s" % url)
# click ie's continue to website
driver.get("javascript:document.getElementById('overridelink').click();")
time.sleep(1)
# clicks 'I Agree' on a page before login
driver.find_element_by_xpath("//button[text()='I Agree']").click()
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'username')))
# logs in
driver.find_element_by_id("username").send_keys("username")
driver.find_element_by_name("j_password").send_keys("password")
driver.find_element_by_xpath("//button[contains(text(), 'Log on')]").click()
# click continue when a successful login box pops up
ret = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.ID, "continueBtn")))
ret.click()
# this waits for an element on the dashboard page - it will fail
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, "//span[contains(text(), 'banner')]")))
# everything from here on will fail due to element not found (only in IE)
driver.find_element_by_xpath("//html")
driver.find_element_by_xpath("anything")
set_trace()
Again, the elements will be found in Firefox and Chrome, but not IE.
You might need to add an Explicit Wait to wait for the data to be loaded. For example, you may wait for a specific element to become present:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
# proceed with locating the elements containing the desired data
Old answer
It sounds like this is a certificate issue, print out the .title and the .page_source once the page is loaded and check what is there.
You can also let it accept the certificate by setting acceptSslCerts desired capability to True:
from selenium import webdriver
capabilities = webdriver.DesiredCapabilities().INTERNETEXPLORER
capabilities['acceptSslCerts'] = True
driver = webdriver.Ie(capabilities=capabilities)
# ...
Also see:
How to deal with certificates using Selenium?
Thank you for answering my previous question but as one is solved another is found apparently.
Interacting with the flash game itself is now the problem. I have tried researching how to do it in Selenium but it can't be done. I've seen FlashSelenium, Sikuli, and AutoIT.
I can only find documentation for FlashSelenium in Java, It's easier for me to use AutoIT rather than Sikuli as I'd have to learn to use Jpython to create the kind of script I want to, which I am not straying away from learning just trying to finish this as fast as possible. As for AutoIT, the only problem with it is that I don't undertsand how to use it with seleium
from selenium import webdriver
import autoit
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://na58.evony.com/s.html?loginid=747970653D74727947616D65&adv=index")
driver.maximize_window()
assert "Evony - Free forever " in driver.title
So far I have this and It's doing what is suppose to do which is create a new account using that "driver.get" but when I reach to the page, it is all flash and I can not interact with anything in the webpage so I have to use AutoIT but I don't know how to get it to "pick-up" from where selenium left off. I want it to interact with a button on the webpage and from viewing a previous post on stackoverflow I can use a (x,y) to specify the location but unfortunately that post didn't explain beyond that. Any and all information would be great, thanks.
Yes, you can use any number of scraping libraries (scrapy and beautiful soup are both easy to use and very powerful). Personally though, I like Selenium and its python bindings because they're the most flexible. Your final script would look something like this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://xx.yy.zz")
# Click the "New comer, click here to play!" button
elem = driver. find_element_by_partial_link_text("Click here to play")
elem.send_keys(Keys.RETURN)
Can you post what the source of the page looks like (maybe using a Pastebin)?
Edit: updated to show you how to click the "Click here to play" button.
I'm having an issue coding a click action using Selenium and the Chrome Web Driver in Python. I've spent some time googling around and found that I have to use another selenium process in order to make a click in Google Chrome, which doesn't make sense to me (Wouldn't it be something while calling webdrive.Chrome?). Though I can't find any other method to make a click, either online or by going through seleniums modules.
Here's what I have, any help is appreciated! Thanks!
EDIT: So I found the ActionChains module in Selenium, can't seem to get this to work either. Updated my code, a bit, still stuck. Does the ChromeDriver really just not support clicks?
import selenium
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
import time
chromeOps = webdriver.ChromeOptions()
chromeOps._binary_location = "C:\\Applications\\Browser\\Chrome.exe"
chromeOps._arguments = ["--enable-internal-flash"]
browser = webdriver.Chrome("C:\\Applications\\Browser\\chromedriver.exe", port=4445, chrome_options=chromeOps)
time.sleep(3)
browser.get("http://example.com")
##selenium.selenium("127.0.0.1", 4445,'*Chrome.exe', 'https://example.com').click("//a[contains(#href,'http://example.com/link')]")
webdriver.ActionChains(browser).click(on_element='//a[contains(#href,"http://example.com/link")]')
I hate it when such simple things are right infront of you.
clickme = browser.find_element_by_xpath('//a[contains(#href,"http://example.com/link")]')
clickme.click()