Develop Reference

Open multiple URL's from a list in new tabs and get the title of each of them

I am trying to open multiple websites from a list in new tabs and get the title of each page.
It works properly when all the websites are up and running..If incase the website is down, that tab is being replaced by the next url in the list.. I want to have that page open even if the page is not working and next url in the list to be opened in the next tab..
This is the code which i m trying :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
service = Service(
"C:\\Users\\vbred\\Desktop\\Selenium with python\\Selenium\\chrome\\chromedriver_win32\\chromedriver.exe")
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(options=chrome_options)
pages = ['https://www.facebook.com/', 'https://www.commbank.com.au/','https://www.commbank.com'
'https://chercher.tech/python/windows-selenium-python']
for p in range(len(pages)):
try:
driver.get(pages[p])
if p!=len(pages)-1:
driver.execute_script("window.open('');")
windows = driver.window_handles
driver.switch_to.window(windows[-1])
except Exception as e:
print(e)
for p in range(len(pages)):
if p != len(pages):
childwindow = driver.window_handles
driver.switch_to.window(childwindow[p])
print(driver.title)
EXPECTED :
All the 4 urls to be opened in different tabs , even if the the third url is not working..
ACTUAL :
First 2 urls are opened in different tabs, but as the third url is not working , in the third tab 4th url is being opened and i dont see the not working url at all

There should be a comma in your 'pages' list (at the end of the 3rd item). I am not sure if it is a typo or you just miss it in your code.
BTW, I prefer open a new tab before the driver gets urls, which means:
driver.execute_script("window.open('');")
windows = driver.window_handles
driver.switch_to.window(windows[-1])
driver.get(pages[p])
Imaginition:
Open the new blank tab.
Get the windows ID.
Switch to it.
Go to the website by driver.get().
Loop until ends.
Maybe you can have a try. Good luck!

Selenium script searches previous tab's HTML after opening a new tab

After opening a new tab using selenium, I tried to look for an element in the new tab – but the script is still searching on the html script from the previous tab. How do I make it so that after I open a new tab, it searches the open tab's HTML instead of a previous one's?
Below is my code (it doesn't work since I try and search for an element on the new tab, but the script searches for it on the first tab).
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\jack_l\\AppData\\Local\\Google\\Chrome\\User Data")
options.add_argument(r'--profile-directory=Profile 8')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
#Opens the Chrome browser
driver.get("https://www.beatstars.com/")
#Opens a new tab on the browser and goes to that URL
driver.execute_script("window.open('https://www.tunestotube.com/', 'new window')")
#Doesn't work since it's searching for a "tunestotube" element on "beatstars"
theText = driver.find_element(By.XPATH, "/html/body/div[2]/div[2]/div[2]").text
Any help would be greatly appreciated.

driver.execute_script("window.open('https://www.tunestotube.com/', 'new window')")
driver.switch_to.window( driver.window_handles[1])
theText = driver.find_element(By.XPATH, "/html/body/div[2]/div[2]/div[2]").text
When you open a tab you get a handle you need to switch back to the previous handle and then find your element.

Found a solution: You need to switch to the new window
Below is code that gets a website on the first tab, opens a new tab with another website, and then can read/find elements in that new tab.
#Gets a browser and sets the window to a variable
driver.get("https://www.exampleWebsiteForFirstTab.com/")
window_before = driver.window_handles[0]
#Open a new tab and sets the window to a variable
driver.execute_script("window.open('https://www.exampleWebsiteForSecondTab.com/', 'new window')")
window_after = driver.window_handles[1]
#Switches the current window to the new tab
driver.switch_to.window(window_after)
###DO WHATEVER YOU WANT IN THE NEW TAB
#Switches the current window to the old tab
driver.switch_to.window(window_before)
Hopefully this helps anyone who had the same problem I did!

Python Selenium - Get Google search HREF

I have two examples of href values from my google search site:linkedin.com/in/ AND "Software Developer" AND "London":
<br><h3 class="LC20lb DKV0Md"><span>Roxana Andreea Popescu - Software Developer - Gumtree ...</span></h3><div class="TbwUpd NJjxre"><cite class="iUh30 Zu0yb qLRx3b tjvcx">uk.linkedin.com<span class="dyjrff qzEoUe"><span> › roxana-andreea-popescu</span></span></cite></div>
<br><h3 class="LC20lb DKV0Md"><span>Tunji Jabitta - London, Greater London, United Kingdom ...</span></h3><div class="TbwUpd NJjxre"><cite class="iUh30 Zu0yb qLRx3b tjvcx">uk.linkedin.com<span class="dyjrff qzEoUe"><span> › tunjijabitta</span></span></cite></div>
I am creating a LinkedIn scraper and I am having a problem when it comes to getting the href value (Which all differ) for each of the results so I can loop through them.
I tried
linkedin_urls = driver.find_elements_by_xpath('//div[#class="yuRUbf"]//a')
links = [linkedin_url.get_attribute('href') for linkedin_url in linkedin_urls]
for linkedin_url in linkedin_urls:
driver.get(links)
sleep(5)
sel = Selector(text=driver.page_source)
But I get the errror A invalid argument: 'url' must be a string'
Another alternative I have tried was
linkedin_urls = driver.find_elements_by_xpath('//div[#class="yuRUbf"]//a[#href]')
for linkedin_url in linkedin_urls:
url = linkedin_url.get_attribute("href")
driver.get(url)
sleep(5)
sel = Selector(text=driver.page_source)
I managed to get the first link opened but it through an error url = linkedin_url.get_attribute("href") when trying to get the other link
Any help would be greatly appreciated, I have been stuck on this for quite a while.

Your driver is opening the link to the new page but it appears, is discarding the previous page. You may want to consider opening in a new tab or window, then switching to that tab/window, once complete, go back to previous page and continue.
Suggested execution:
1. Create a function to open link (or element) in a new tab – and to switch to that tab:
from selenium.webdriver.common.action_chains import ActionChains
# Define a function which opens your element in a new tab:
def open_in_new_tab(driver, element):
"""This is better than opening in a new link since it mimics 'human' behavior"""
# What is the handle you're starting with
base_handle = driver.current_window_handle
ActionChains(driver) \
.move_to_element(element) \
.key_down(Keys.COMMAND) \
.click() \
.key_up(Keys.COMMAND) \
.perform()
# There should be 2 tabs right now...
if len(driver.window_handles)!=2:
raise ValueError(f'Length of {driver.window_handles} != 2... {len(driver.window_handles)=};')
# get the new handle
for x in driver.window_handles:
if x!= base_handle:
new_handle = x
# Now switch to the new window
driver.switch_to.window(new_handle)
2. Execute + Switch back to the main tab:
import time
# This returns a list of elements
linkedin_urls = driver.find_elements_by_xpath('//div[#class="yuRUbf"]//a[#href]')
# A bit redundant, but it's web scraping, so redundancy won't hurt you.
BASE_HANDLE = driver.current_window_handle # All caps so you can follow it more easily...
for element in linkedin_urls:
# switch to the new tab:
open_in_new_tab(driver, element)
# give the page a moment to load:
time.sleep(0.5)
# Do something on this page
print(driver.current_url
# Once you're done, get back to the original tab
# Go through all tabs (there should only be 2) and close each one unless
# it's the "base_handle"
for x in driver.window_handles:
if x!= base_handle:
driver.switch_to.window(x)
driver.close()
# Now switch to the new window
assert BASE_HANDLE in driver.window_handles # a quick sanity check
driver.switch_to.window(BASE_HANDLE) # this takes you back
# Finally, once you for-loop is complete, you can choose to continue with the driver or close + quit (like a human would)
driver.close()
driver.quit()

how to get the url after click to know whether it has 404 error or downloaded the file in the preferenced place using selenium Firefox python

I am new to web automation using selenium Firefox python environment.
Step 1:- I am hitting a webpage using driver.get. there by I am setting up my preference to download the file in the specific location.
Step2:- Providing data to Decal Number after finding driver.find_element_by_id("DecalNumber"). And click Begin search button.
Step3:- Now again click button by accessing through xpath.
Step4:- Now again click one more button by accessing through xpath. this will open a file in new window (https://www.tdlr.texas.gov/DownLoadableContent/Elevator/Images/1197897/26695455.tif) as an example.
My questions are all follows:-
Since the file is automatically getting saved in the preferenced location, and most of the time I get 404 error no data found and some of the time I get .tiff image data. how do I check whether it as data or 404 error????
If 404 error is not found, then need to rename the file that is being downloaded with its associated Decal Number using Shutil library. but the problem here is, if I have 404 error then no new file is getting downloaded. But shutil.move() is tryin to rename the file which is already downloaded for other decal number. Shutil is trying to change the file name wit current decal number, which should not supposed to do. how to stop this happening?
Any lead are welcome.
from selenium import webdriver
# create webdriver object
fp = webdriver.FirefoxProfile()
fp.set_preference('browser.download.dir', r'C:\Users\Swaminathan Sekar\Documents\Decal_Final')
fp.set_preference('browser.download.folderList', 2)
fp.set_preference('browser.helperApps.neverAsk.saveToDisk', "image/tiff")
fp.set_preference('browser.download.manager.showWhenStarting', False)
fp.set_preference('browser.helperApps.neverAsk.openFile', "image/tiff")
fp.set_preference('browser.helperApps.alwaysAsk.force', False)
fp.set_preference('browser.download.manager.useWindow', False)
fp.set_preference('browser.download.manager.focusWhenStarting', False)
fp.set_preference('browser.download.manager.alertOnEXEOpen', False)
fp.set_preference('browser.download.manager.showAlertOnComplete', False)
fp.set_preference('browser.download.manager.closeWhenDone', False)
fp.set_preference("pdfjs.disabled", True)
driver = webdriver.Firefox(fp)
driver.maximize_window()
driver.implicitly_wait(20)
driver.get("https://www.tdlr.texas.gov/Elevator_SearchApp/Elevator/Search")
window_before = driver.current_window_handle
Decal_test=Decal_Number[75:76]
for i in range(0,len(Decal_test)):
inputElement1 = driver.find_element_by_id("DecalNumber")
inputElement1.clear()
inputElement1.send_keys(Decal_test[i])
driver.find_elements_by_xpath("/html/body/div[1]/div[2]/div[3]/div/div[3]/div/form/div[6]/div/div[2]/button")[0].click()
driver.find_elements_by_xpath("/html/body/div[1]/div[2]/div[3]/div/div[3]/div[2]/div/table/tbody/tr/td[3]/a[1]")[0].click()
driver.find_elements_by_xpath("/html/body/div[1]/div[2]/div[2]/div/div[2]/div[1]/div[1]/table/tbody/tr[6]/td/button")[0].click()
window_after = driver.window_handles[1]
driver.switch_to.window(window_after)
driver.implicitly_wait(30)
driver.refresh()
driver.find_element_by_xpath("//td[text()='Subsequent Inspection']/preceding-sibling::td[1]/a").click()
wait = WebDriverWait(driver, 10)
wait.until(EC.text_to_be_present_in_element(".tiff"))
print(driver.current_url)
time.sleep(20)
driver.switch_to.window(window_before)
driver.back()
driver.back()
Initial_path= r'C:\Users\Swaminathan Sekar\Documents\Decal_Final'
filename = max([Initial_path + "\\" + f for f in os.listdir(Initial_path)],key=os.path.getctime,default=0)
if filename !=0:
shutil.move(filename,os.path.join(Initial_path, str(i)+'_'+str(Decal_test[i]) + ".tiff"))
print('Elasped time =',time.process_time(),'s')

move control to newly opened IE window in selenium webdriver (using python) :no window name , no window handle ,not a alert

Setup :python bindings for selenium 2.45.0 ,IEserver driver2.45.0(x86),python 2.7.9 ,window 7 64 bit
Issue : when i click on this redirect button href= https:www.work.test.co.in:1XXX9/TEST/servlet/MainServlet/home" target="_blank"
a new window opens , unable to click anything on new window as control(focus) remains on previous window (confirmed by closing the previous window).
Tried
1.no name , so cannot try
driver.switch_to_window("windowName")
2.tried to print the handle (so that i can use handle reference ) but i can see only one window handle . used following code
for handle in driver.window_handles:
print "Handle arr = ",handle
driver.switch_to_window(handle)
3.Question1 : why i am getting only one window handle handle , i can see two IE instances in task manager.
4.i tried using index - 0 ,1 etc.
driver.switch_to_window(driver.window_handles[-1])
5.not sure of this thing though tried
driver.SwitchTo().Window(driver.WindowHandles.Last())
6.tried though i am sure that its not an alert window .
alert = driver.switch_to_alert()
SCRIPT :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
driver = webdriver.Ie()
driver.get("https://my intranet site .aspx")
driver.implicitly_wait(2)
elem = driver.find_element_by_xpath("my xpath ")
elem.click()
driver.implicitly_wait(2)
elem = driver.find_element_by_xpath("//*[#id='tab1_2']/div[16]")
elem.click()
handle = driver.current_window_handle
print "Handle main = ",handle
driver.implicitly_wait(5)
elem = driver.find_element_by_xpath("page link button")
elem.click()
sleep(5)
my tried scenarioes here
Suggestions will be highly appreciated
Update - when new window opened directly through link URL , able to perform actions on it like clicking etc
So only issue is when I open it in continuation of first window through script.
Update : Main concern is why not getting second window handle even if task manager showing two instances of IE .

I don't know Python, but in Java I would do it in this way:
// get handles to all opened windows before the click
Set<String> handlesBeforeTheClick = driver.getWindowHandles();
// and now click on the link that opens a new window
findElement( linkThatOpensNewWindow ).click();
// then wait until a new window will be opened
wait.until( ....condition ==> handlesBeforeClick.size() < driver.getWindowHandles().size(); .... )
// then get a handle to a new window
Set<String> handlesAfterClick = driver.getWindowHandles();
handlesAfterClick.removeAll( handlesBeforeClick );
String handleToNewWindow = handlesAfterClick.iterator().next();