Develop Reference

Why my selenium IDE works good while the same python command doesn't?

I have some problems trying to automate repetitive tasks in my company's intranet website.
While I can properly find elements in other websites (clicking, sending keys etc.), I can't do the same in this website (the error I always get is "unable to find element with xpath == ... ").
I tried to do the same tasks recording them with Selenium IDE, and it works well.
If I export the Selenium IDE test case as python code and I compare the xpath command with mine, the xpath is the same, and it's the same that I get using the firefox selector to the point I want to click.
Since for other reasons I must work with explorer (so I have the explorer webdriver etc.), I haven't tried to run the code with python+(webdriver for Firefox).
Code:
from selenium import webdriver
import time
driver=webdriver.IE("D:\myfolder\IEDriverServer.exe")
driver.implicitly_wait(30)
base_url="https://companywebsite.ext")
driver.get(base_url) #it's ok until now, I can open the webpage
time.sleep(20)
element=driver.find_element_by_xpath("/html/body/div[2]/div/table/tbody/tr[3]")
time.sleep(10)
Here are my questions:
1) Why can I do things right in other websites and not in this?
2) Why Selenium IDE is able to do that and, with the same commands, it doesn't work with my python script?
3) Do I have to do particular setups?
Thank you very much.

You should try using wait's:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver=webdriver.IE("D:\myfolder\IEDriverServer.exe")
driver.implicitly_wait(30)
base_url="https://companywebsite.ext")
driver.get(base_url)
wait = WebDriverWait(driver, 10)
element = wait.until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[2]/div/table/tbody/tr[3]")))
Hope this helps!

Selenium can't find any elements on IE11 once a dashboard is loaded

So something interesting is happening - unfortunately, I cannot post a url or something you guys can access, so maybe someone just has an idea of what the cause could be. I'd assume it would be something in development. Either way...
I have a test that logs in to an application, which will then load up a dashboard and search for some data. In Firefox, everything works fine. However, IE11 is having an issue once it logs in. So, Selenium works fine up until it loads the dashboard, where it will then find NOTHING. Not even "//html". It gives the basic element not found error. I'm assuming the developers have written something that prevents access, but the one I talked to wasn't sure.
Any idea of what could be causing this so that I could go suggest something to the developers? Again, this doesn't happen in Firefox, which could also suggest something's up with IE11 (though Selenium works fine on other applications and up until the Dashboard screen on this one). If you need more information, let me know - just be aware that I might not have much to give.
A couple things to note:
1. it is https protocol, but so is the login page
2. at the end of the url bar, it says there is a certificate error that will not go away
no matter what I do (I have added the supposedly correct certificates).
What I'm using:
Selenium 2.4.5
32 bit IEDriverServer
Python 2.7.8
IE11 (other IE versions are not supported, so I cannot compare)
Windows 7
EDIT:
I talked with a developer, and the best thing he could think of was the fact that the dashboard heavily uses polyfills, and that maybe that is the reason that Selenium isn't finding anything. With that being said, is there a way to find elements using Selenium when polyfills are involved? Or is Selenium just not going to work for it.
MORE:
Here is some code that equivalent to what my test would do. Assume that every exception would normally be caught and wouldn't kill the script...
from pdb import set_trace
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
url = 'server.url'
capabilities = webdriver.DesiredCapabilities().INTERNETEXPLORER
capabilities['acceptSslCerts'] = True
# use path due to having both a 64 bit and 32 bit version of the driver - specifies which to use
driver = webdriver.Ie("path-to-iedriver.exe", capabilities=capabilities)
driver.maximize_window()
driver.get("https://%s" % url)
# click ie's continue to website
driver.get("javascript:document.getElementById('overridelink').click();")
time.sleep(1)
# clicks 'I Agree' on a page before login
driver.find_element_by_xpath("//button[text()='I Agree']").click()
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'username')))
# logs in
driver.find_element_by_id("username").send_keys("username")
driver.find_element_by_name("j_password").send_keys("password")
driver.find_element_by_xpath("//button[contains(text(), 'Log on')]").click()
# click continue when a successful login box pops up
ret = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.ID, "continueBtn")))
ret.click()
# this waits for an element on the dashboard page - it will fail
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, "//span[contains(text(), 'banner')]")))
# everything from here on will fail due to element not found (only in IE)
driver.find_element_by_xpath("//html")
driver.find_element_by_xpath("anything")
set_trace()
Again, the elements will be found in Firefox and Chrome, but not IE.

You might need to add an Explicit Wait to wait for the data to be loaded. For example, you may wait for a specific element to become present:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
# proceed with locating the elements containing the desired data
Old answer
It sounds like this is a certificate issue, print out the .title and the .page_source once the page is loaded and check what is there.
You can also let it accept the certificate by setting acceptSslCerts desired capability to True:
from selenium import webdriver
capabilities = webdriver.DesiredCapabilities().INTERNETEXPLORER
capabilities['acceptSslCerts'] = True
driver = webdriver.Ie(capabilities=capabilities)
# ...
Also see:
How to deal with certificates using Selenium?

Using AutoIT with Selenium

Thank you for answering my previous question but as one is solved another is found apparently.
Interacting with the flash game itself is now the problem. I have tried researching how to do it in Selenium but it can't be done. I've seen FlashSelenium, Sikuli, and AutoIT.
I can only find documentation for FlashSelenium in Java, It's easier for me to use AutoIT rather than Sikuli as I'd have to learn to use Jpython to create the kind of script I want to, which I am not straying away from learning just trying to finish this as fast as possible. As for AutoIT, the only problem with it is that I don't undertsand how to use it with seleium
from selenium import webdriver
import autoit
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://na58.evony.com/s.html?loginid=747970653D74727947616D65&adv=index")
driver.maximize_window()
assert "Evony - Free forever " in driver.title
So far I have this and It's doing what is suppose to do which is create a new account using that "driver.get" but when I reach to the page, it is all flash and I can not interact with anything in the webpage so I have to use AutoIT but I don't know how to get it to "pick-up" from where selenium left off. I want it to interact with a button on the webpage and from viewing a previous post on stackoverflow I can use a (x,y) to specify the location but unfortunately that post didn't explain beyond that. Any and all information would be great, thanks.

Yes, you can use any number of scraping libraries (scrapy and beautiful soup are both easy to use and very powerful). Personally though, I like Selenium and its python bindings because they're the most flexible. Your final script would look something like this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://xx.yy.zz")
# Click the "New comer, click here to play!" button
elem = driver. find_element_by_partial_link_text("Click here to play")
elem.send_keys(Keys.RETURN)
Can you post what the source of the page looks like (maybe using a Pastebin)?
Edit: updated to show you how to click the "Click here to play" button.

searching web-client for python that can handle frames

I'm searching for a python module that simulates a webbrowser and can handle html frames. I want to use the chatbot Brain http://www.thebot.de/ with python. If you know any tutorials that explain how to use your suggested module in relation with forms and frames give me a links to them please.

Tim,
I suggest you take a look at Selenium. By default it opens and manipulates Firefox to navigate the web. Its primary use case is testing, however in a pinch I've used it in some of my scripts to get past sites with a lot of javascript or, as in this case, iframes.
The basic usage is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
driver.close()
To access an iframe you can use the following code:
elm = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(elm)
Then when you want to switch out of the iframe:
driver.switch_to.default_content()
You can even utilize Selenium headless with xvfbwrapper like this (code from github.com/cgolberg/xvfbwrapper):
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
# launch stuff inside virtual display here
vdisplay.stop()

Selenium takes a long time to find an element.Is there something I can do?

I have been trying to write a selenium script to login to my Quora account.
This is the script I have written.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import getpass
import time
email=raw_input("email: ")
password=getpass.getpass("Password: ")
driver = webdriver.Firefox()
driver.get("http://www.quora.com")
#time.sleep(5)
Form=driver.find_element_by_xpath("//div[#class='form_inputs']")
Form.find_element_by_name("email").send_keys(email)
#time.sleep(4)
Form.find_element_by_name("password").send_keys(password)
#time.sleep(4)
Form.find_element_by_xpath("//input[#value='Login']").click()
The statement
Form=driver.find_element_by_xpath("//div[#class='form_inputs']") takes very long to find the element. In fact, all the find_element statements take very long to do their job.(This could be because of some Javascript snippet to increase the load on selenium, but I could not understand much from the page source)
Is there any way I could do it faster? Similar scripts have worked well for me in Facebook and Google.
EDIT:
Removed the time.sleep() calls. It still takes around 6-8 minutes to find the element.

The reason why it is taking a while is because you are preforming time.sleep()
You should not do this, it's bad practice. You should be using WebDriver waits. I would personally go with Implicit waits for your scenario.
Please see the documentation

This is something I've seen here asked on SO multiple times, see:
Is Selenium slow, or is my code wrong?
Unable to login to quora using selenium webdriver in python
I've been able to reproduce the slow code execution using Firefox, but the following code works without any delays using Chrome or PhantomJS driver:
import getpass
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
email = raw_input("email: ")
password = getpass.getpass("Password: ")
driver = webdriver.Chrome()
driver.get("http://www.quora.com")
form = driver.find_element_by_class_name('regular_login')
form.find_element_by_name('email').send_keys(email)
form.find_element_by_name('password').send_keys(password + Keys.RETURN)
FYI, for Firefox, it helps to overcome the issue if you fire up Firefox with disabled javascript:
firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("browser.download.folderList",2)
firefox_profile.set_preference("javascript.enabled", False)
driver = webdriver.Firefox(firefox_profile=firefox_profile)
driver.get('http://www.quora.com/')
But, as you would see - you'll quickly get a different set of problems.