Develop Reference

Printing web search results won't work in Selenium script, but works when I type it into the shell

I'm very new and learning web scraping in python by trying to get the search results from the website below after a user types in some information, and then print the results. Everything works great up until the very last 2 lines of this script. When I include them in the script, nothing happens. However, when I remove them and then just try typing them into the shell after the script is done running, they work exactly as I'd intended. Can you think of a reason this is happening? As I'm a beginner I'm also super open if you see a much easier solution. All feedback is welcome. Thank you!
#Setup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
#Open Chrome
driver = webdriver.Chrome()
driver.get("https://myutilities.seattle.gov/eportal/#/accountlookup/calendar")
#Wait for site to load
time.sleep(10)
#Click on street address search box
elem = driver.find_element(By.ID, 'sa')
elem.click()
#Get input from the user
addr = input('Please enter part of your address and press enter to search.\n')
#Enter user input into search box
elem.send_keys(addr)
#Get search results
elem = driver.find_element(By.XPATH, ('/html/body/app-root/main/div/div/account-lookup/div/section/div[2]/div[2]/div/div/form/div/ul/li/div/div[1]'))
print(elem.text)

I haven't used Selenium in a while, so I can only point you in the right direction. It seems to me you need to iterate over the individual entries, and print those, as opposed to printing the entire div as one element.

You should remove the parentheses from the xpath expression
You can shorten the xpath expression as follows:
Code:
elems = driver.find_element(By.XPATH, '//*[#class="addressResults"]/div')
for elem in elems:
print(elem.text)

You are using an absolute XPATH, what you should be looking into are relative XPATHs
Something like this should do it:
elems = driver.find_elements(By.XPATH, ("//*[#id='addressResults']/div"))
for elem in elems:
...

I ended up figuring out my problem - I just needed to add in a bit that waits until the search results actually load before proceeding on with the script. tossing in a time.sleep(5) did the trick. Eventually I'll add a bit that checks that an element has loaded before proceeding with the script, but this lets me continue for now. Thanks everyone for your answers!

Selenium webdriver.get() method doesnt always work

link = "https://www.google.com"
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % str(xxx))
chrome = webdriver.Chrome(chrome_options=chrome_options)
time.sleep(3)
chrome.get(link)
print("po get")
time.sleep(1)
chrome.get(link)
time.sleep(15)
Hello, I have a problem with selenium for a long time and I would like to find a way how to fix it
The problem is that almost everytime I run a script that's opening selenium / even when I use it like
for a test such as :
from selenium import webdriver
chrome = webdriver.Chrome()
chrome.get(https://www.google.com)
It still sometimes doesn't get the website, I thought it's because of how slow selenium opens but even after its nicely open it doesn't get the value, sadly it just gets stuck on an empty browser that had data in an url window... any idea what should I do to fix it?

Okay, after a few hours I decided to give it a little try and change ( "" ) to ( ' ' ) and as I can see it works :D I dont know why it has a problem with string ""
here is edited line of my code :
chrome.get('https://www.google.com')
( I ve tried it with proxies with loop that was starting the webdriver
100 times , and everytime after i've changed it, it passed )

You can use driver.navigate.to("");
Also as I see, You might missing String Double Quotes " " here, chrome.get(https://www.google.com)

I think it's too late for answer, but I want to answer anyway. You may have already solved problem, but if it doesn't work check this out:
Different links navigate you same site or etc. It is not a code error but I want to say that this could cause of selenium.common.exceptions.NoSuchElementException: Message: no such element
You logged a site and if you want to go back login screen, you should log out of your account, otherwise the site will automatically navigate to the home screen because you already logged in.
That's all I wanted to say.

Selenium Python find element partial link text

I'm trying to use Pythons Selenium module to click on an element whose link has the text "xlsx" at the end of it. Below is the code I'm using and the details of the element. Can someone please see why Python is unable to find this element?
driver.find_element_by_partial_link_text('xlsx').click()
Here is the element details:
<a name="URL$2" id="URL$2" ptlinktgt="pt_new" tabindex="43" onfocus="doFocus_win0(this,false,true);" href="http:******/HSC8_CNTRCT_ITEMS_IMPRVD-1479218.xlsx" onclick="window.open('http:********/HSC8_CNTRCT_ITEMS_IMPRVD-1479218.xlsx','','');cancelBubble(event);return false;" class="PSHYPERLINK">HSC8_CNTRCT_ITEMS_IMPRVD-1479218.xlsx</a>
I had to remove some parts of the URL for confidentiality purposes, however, it should not impact the answering of the question.
Thanks.

Thanks for the replies. Turns out, as #Andersson mentioned, the window was in a different frame.
I solved the problem using the following code before the find_element: driver.switch_to.frame('ptModFrame_0').

You can use a CSS selector:
driver.find_element_by_css_selector("a[href*='xlsx']")
If the element still cannot be located, I would suggest using a wait statement, to ensure that the element is visible, before you interact with it.

Please try:
driver.find_element_by_xpath(".//a[contains(#href,'xlsx')]").

You can grab it by class name (class name = PSHYPERLINK).
This should work:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
url = ''
driver = webdriver.Chrome('/path/to/chromedriver/executable')
if __name__=='__main__':
driver.get(url)
time.sleep(3)
driver.find_element_by_class_name('PSHYPERLINK').click()
When finding the attribute, make sure to use a singular '"element". Like:
driver.find_element_by_class_name('PSHYPERLINK').click()
not:
driver.find_elements_by_class_name('PSHYPERLINK').click()
Hope this helps.

Python Splinter clicking button CSS

I'm having trouble selecting a button in my Splinter script using the find_by_css method. The documentation is sparse at best, and I haven't found a lot of good articles out there with examples.
br.find_by_css('div#edit-field-download-files-und-0 a.button.launcher').first.click()
...where br is my browser instance.
I've tried a few different ways of writing it. I'm really not sure how I'm supposed to do it because the documentation doesn't give any hard examples of the syntax.
Here's a screenshot of the element.
Sorry the screenshot kind of sucks.
Does anyone have any experience with this?

The css selector looks alright, just that i am not sure from where have you got find_by_css as a method?
How about this :-
br.find_element_by_css_selector("div#edit-field-download-files-und-0 a.button.launcher").click()
Selenium provides the following methods to locate elements in a page:
find_element_by_id
find_element_by_name
find_element_by_xpath
find_element_by_link_text
find_element_by_partial_link_text
find_element_by_tag_name
find_element_by_class_name
find_element_by_css_selector
To find multiple elements (these methods will return a list):
find_elements_by_name
find_elements_by_xpath
find_elements_by_link_text
find_elements_by_partial_link_text
find_elements_by_tag_name
find_elements_by_class_name
find_elements_by_css_selector

I'm working on something similar where I'm trying to click stuff on a webpage. The documentation for find_by_css() is very poor and you need to type the css path to the element you want to click.
Say we want to go to the about tab on python.org
from splinter import Browser
from time import sleep
with Browser() as browser: #<--Create browser instance (firefox default driver)
browser.visit('http://www.python.org') #<--Visits url string
browser.find_by_css('#about > a').click()
# ^--Put css path here in quotes
sleep(5)
If your connection is good you might not get the chance to see the about tab getting clicked but you should end up on the about page.
The hard part is figuring out the css path to an element. However once you have it, the find_by_css() method looks pretty easy

I like the W3Schools reference for CSS selection parameters: http://www.w3schools.com/cssref/css_selectors.asp
As for your code... I recommend breaking this down into a few steps, at least during debug. The call to br.find_by_css('css_string') returns a list of elements. So you can grab that list and check the count.
elems = br.find_by_css('div#edit-field-download-files-und-0 a.button.launcher')
if len(elems) == 1:
elems.first.click()
If you don't check the length of the returned list and call '.first' on an empty list, you'll get an exception. If len > 1, you're probably getting things you don't expect.
Each id on a page is unique, and you can daisy-chain searches, so you can use a few different statements to make this happen:
id_elems = br.find_by_id('edit-field-download-files-und-0')
if id_elems:
id_elem = id_elems.first
a_elems = id_elem.find_by_tag("a")
for e in a_elems:
if e.has_class("button launcher"):
print('Found it!')
e.click()
This is, of course, just one of many ways to do this.
Lastly, Splinter is a wrapper around Selenium and other webdrivers. It's possible that, even after you find the element to click, the actual click won't do anything. If this happens, you can also try clicking on the wrapped Selenium object, available as e._element. So you could try e._element.click() if necessary.

Selenium scrape with full xpath and Python

I am trying to get the Nasdaq "Most Advanced" list of stocks from here: http://www.nasdaq.com/extended-trading/premarket-mostactive.aspx (click on Most Advanced tab)
What is the best way using Selenium to loop through all the Symbols and put them into a Python list? I have figured out the XPATH to the first Symbol:
/html/body/div[4]/div[3]/div/div[7]/div[2]/table/tbody/tr[2]/td/div/h3/a
but am not sure where to go from there.. I tried:
element=driver.find_elements_by_xpath("/html/body/div[4]/div[3]/div/div[7]/div[2]/table/tbody/tr[2]/td/div/h3/a")
print element.text
..as a start just to see if I can get a value but it obviously doesn't work. Sorry for the stupid question :(

These xpaths containing the full absolute path to the element are very fragile.
Rely on the class name (//div[#class="symbol_links"]):
from selenium.webdriver.firefox import webdriver
driver = webdriver.WebDriver()
driver.get('http://www.nasdaq.com/extended-trading/premarket-mostactive.aspx')
# choose "Most Advanced" tab
advanced_link = driver.find_element_by_id('most-advanced')
advanced_link.click()
# get the symbols
print [symbol.text for symbol in driver.find_elements_by_xpath('//div[#class="symbol_links"]') if symbol.text]
driver.close()
prints:
[u'RNA', u'UBIC', u'GURE', u'DRTX', u'DSLV', u'YNDX', u'QIWI', u'NXPI', u'QGEN', u'ZGNX']
Hope that helps.