Develop Reference

How to scrape the nested data from Linkedin page using Selenium and Python

I am sitting with a project for my masters, where I would like to scrape LinkedIn. As far as I am now, I ran into a problem when I want to scrape the education pages of users (eg. https://www.linkedin.com/in/williamhgates/details/education/)
I would like to scrape all the educations of the users. In this example I would like to scrape "Harvard University" under mr1 hoverable-link-text t-bold, but I can't see to get to it.
Here's the HTML at code from Linkedin:
<li class="pvs-list__paged-list-item artdeco-list__item pvs-list__item--line-separated " id="profilePagedListComponent-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-EDUCATION-VIEW-DETAILS-profile-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-NONE-da-DK-0">
<!----><div class="pvs-entity
pvs-entity--padded pvs-list__item--no-padding-when-nested
">
<div>
<a class="optional-action-target-wrapper
display-flex" target="_self" href="https://www.linkedin.com/company/1646/">
<div class="ivm-image-view-model pvs-entity__image ">
<div class="ivm-view-attr__img-wrapper ivm-view-attr__img-wrapper--use-img-tag display-flex
">
<!----> <img width="48" src="https://media-exp1.licdn.com/dms/image/C4E0BAQF5t62bcL0e9g/company-logo_100_100/0/1519855919126?e=1668643200&v=beta&t=BL0HxGNOasVbI3u39HBSL3n7H-yYADkJsqS3vafg-Ak" loading="lazy" height="48" alt="Harvard University logo" id="ember59" class="ivm-view-attr__img--centered EntityPhoto-square-3 lazy-image ember-view">
</div>
</div>
</a>
</div>
<div class="display-flex flex-column full-width align-self-center">
<div class="display-flex flex-row justify-space-between">
<a class="optional-action-target-wrapper
display-flex flex-column full-width" target="_self" href="https://www.linkedin.com/company/1646/">
<div class="display-flex align-items-center">
<span class="mr1 hoverable-link-text t-bold">
<span aria-hidden="true"><!---->Harvard University<!----></span><span class="visually-hidden"><!---->Harvard University<!----></span>
</span>
<!----><!----><!----> </div>
<!----> <span class="t-14 t-normal t-black--light">
<span aria-hidden="true"><!---->1973 - 1975<!----></span><span class="visually-hidden"><!---->1973 - 1975<!----></span>
</span>
<!----> </a>
<!---->
<div class="pvs-entity__action-container">
<!----> </div>
</div>
<div class="pvs-list__outer-container">
<!----> <ul class="pvs-list
">
<li class=" ">
<div class="pvs-list__outer-container">
<!----><!----><!----></div>
</li>
</ul>
<!----></div>
</div>
</div>
</li>
I have tried the following code:
education = driver.find_element("xpath", '//*[#id="profilePagedListComponent-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-EDUCATION-VIEW-DETAILS-profile-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-NONE-da-DK-0"]/div/div[2]/div[1]/a/div/span/span[1]/').text
print(education)
I keep getting the error:
Message: no such element: Unable to locate element:
Can anybody help? I would love to have a script that loops through the educations, and save place of education and the year of educations.

Thank you everyone!
I ended up with this code under that worked.
get_education_school = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//ul[#class='pvs-list ']/li//span[contains(#class, 'hoverable-link-text')]//span[1]")))]
get_education_years = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//ul[#class='pvs-list ']/li//span[contains(#class, 't-14 t-normal t-black--light')]//span[1]")))]
results_education_school = []
results_education_years = []
for i,j in zip(get_education_school, get_education_years):
results_education_school.append(i)
results_education_years.append(j)
print(results_education_school)
print(results_education_years)

To extract the text Harvard University ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "ul.pvs-list>li span.hoverable-link-text span"))).text)
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//ul[#class='pvs-list ']/li//span[contains(#class, 'hoverable-link-text')]//span"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

I would first get the list for the education section.
education_list = driver.find_element(By.CSS_SELECTOR, 'ul.pvs-list')
# loop through education_list for place and years
# would recommend relative locators for this task.
# find the image and get the first and second span with text inside of them.
I am adding further details to the code now. Please hold.

You can use below properties to identify the school name list:
ancestorClass="optional-action-target-wrapper display-flex flex-column full-width" class="display-flex align-items-center" tag="DIV"
Use these properties to identify the year list:
ancestorClass="optional-action-target-wrapper display-flex flex-column full-width" class="t-14 t-normal t-black--light" tag="SPAN"
You may use above info to compose an XPath to locate the list, or if you don't mind using other python libraries, there is a sample code in GitHub to scrape the school and year.

#Nadia S. you can try the following code. I have provided comments inline inside the code.
#Test
public void linkedInTest() {
driver.get("https://www.linkedin.com");
// You need to enter the credentials for your linkedin below for login
driver.findElement(By.id("session_key")).sendKeys("");
driver.findElement(By.id("session_password")).sendKeys("");
driver.findElement(By.className("sign-in-form__submit-button")).click();
driver.get("https://www.linkedin.com/in/williamhgates/details/education/");
//Wait for the Education details to get populated.
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(7));
wait.until(ExpectedConditions.visibilityOfElementLocated(
By.xpath("//div[#class = 'pvs-list__container']//div[#class = 'scaffold-finite-scroll__content']/ul")));
//Take all elements showing education details in a list
List<WebElement> allEducation = driver.findElements(By
.xpath("//div[#class = 'pvs-list__container']//div[#class = 'scaffold-finite-scroll__content']/ul/li"));
//Extract details of each education item in the list.
//Below the details are directed to console. You can use a collection to store them.
for (WebElement oneEducation : allEducation) {
WebElement education = oneEducation.findElement(
By.xpath(".//*[contains(#class,\"mr1 hoverable-link-text\")]/span[#aria-hidden='true']"));
System.out.print("Education - " + education.getText());
try {
WebElement educationType = oneEducation
.findElement(By.cssSelector(".t-14.t-normal span[aria-hidden='true']"));
System.out.print(" Education Type - " + educationType.getText());
} catch (NoSuchElementException e) {
System.out.print(" Education Type - " + "is Not Specified");
}
try {
WebElement educationYear = oneEducation
.findElement(By.cssSelector(".t-14.t-normal.t-black--light span[aria-hidden='true']"));
System.out.println(" Education Year - " + educationYear.getText());
} catch (NoSuchElementException e) {
System.out.println(" Education Year - " + "is Not Specified");
}
}
}

Unable to Locate Element for Selenium Webdriver Path in Python

I've been trying to reference the two inputs in the following HTML code using Selenium's get_element_by_...() with the purpose of automatically entering a username and password into the fields. I'm running into trouble with accessing the proper HTML elements.
<body>
<div id="app">
<div>
<section id="root" class="page-login">
<section class="main">
<section class="auth-page register-fill">
<div class="page-center">
<a class="switch-lang" href="/cn/login">cn</a>
<a class="btn-back" href="en/login">
<span class="isvg loaded">
<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100" viewBox="0 0 24 24">
<path class="a" d="M0,0H24V24H0Z"></path>
<path class="b" d="M20,11H7.83l5.59-5.59L12,4,4,12l8,8,1.41-1.41L7.83,13H20Z"></path>
</svg>
</span>
</a>
<div class="auth-page-wrapper">
<h1 class="auth-page-title">Welcome Back</h1>
<h2 class="auth-page-subtitle">Login to get started</h2>
<div class="auth-page-fields">
<div class="custom-input">
<input type="text" placeholder="Email">
<label>Email</label>
</div>
<div class="custom-input custom-input-password">
<a class="forgot-link" href="en/forgot-password">Forgot Password?</a>
<input type="password" placeholder="Minimum 6 characters">
<label>Password</label>
</div>
</div>
Note: The HTML may have some mistakes, I just quickly formatted it, but the mistakes should not affect the question.
Tried:
I've tried using all the different functions of the 'form find_element_by_...' and referencing a variety of things, but all tend to return "no such element: unable to locate element ..."
I've done absolute xpaths, relative xpaths, css selectors, ids, and combinations of these.
Some examples:
username = browser.find_element_by_xpath("//input[1]")
username = browser.find_element_by_xpath("//html/body/div/div/section/.../input")
The error begins at this point in the path:
username = browser.find_element_by_xpath("//html/body/div/div/section")
I'm likely making a stupid mistake, so sorry if this question isn't great, but other StackOverflow answers all have ids and such that are referable on the input field. (eg. <input id=username ...>)

To identify the Email field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[placeholder='Email']")))
Using XPATH:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#placeholder='Email']")))
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

There are more than one way to do this. But, this should work browser.find_element_by_css_selector('[placeholder="Email"]')

Can find element within an iframe using Firefox but not Chrome

So I'm trying to write a test for a webpage which has some elements within an iframe. I've been able to successfully run the test using webdriver.Firefox() without any problems but if I switch it over to webdriver.Chrome() I get a timeout exception on the following lines of code:
self.driver.switch_to.frame(0)
self.activity_status = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#overview > div.details.w-66 > div > div.duration-and-status > span.status.stat_incomplete#')))
It'd be great to get a solution to this as I'm all out of ideas.
Thanks for your help.
edit, partial html for the page:
<iframe id="iframe_course_details" allowfullscreen="" src="../Course/Details.aspx?HidePageNav=true&IsInIframe=true"></iframe>
Close
Edit (Inactive)
Edit
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_favourite" class="favourite button tooltipstered" style="display: none;">Favourite</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_basket_dull" class="add-to-basket button delete tooltipstered" style="display: none;">Enrolled (Remove From Enrolments)</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_basket" class="add-to-basket button tooltipstered">Add to Enrolments</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_print" class="print button tooltipstered">Print</span>
</div>
<section id="overview" style="opacity: 1;">
<div id="fullname" class="fullname w-100" style="display: none;">
</div>
<div class="image w-33" style="cursor: pointer;">
<div style="background-image:url(/App_Themes/MainTheme-responsive/Images/Course/webcast.jpg);"></div></div>
<div class="details w-66">
<div class="inner">
<h2>testing activity</h2>
<div class="star-rating-num-ratings">
<div class="star-rating">
<span></span><span></span><span></span><span></span><span></span>
</div>
<span class="num-of-ratings">0 Ratings</span>
</div>
<div class="duration-and-status">
<span class="duration">
<label>
Duration:
</label>
<span>0</span>
</span>
<span class="status stat_incomplete">Started</span>
</div>
Edit 2:
So we've managed to find a solution to this and its even more confusing than the original problem
WebDriverWait(self.driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, 'iframe_course_details')))
time.sleep(0)
self.activity_status = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//*[#id="overview"]/div[3]/div/div[2]/span[2]')))
I'd be really curious to hear some theories on why this works, it times out without the 'time.sleep(0).

If you reference the iframe directly rather then an integer that will work between Firefox/Chrome.
self.driver.switch_to.frame(driver.find_element_by_name("iframe"))
You can find the iframe element any way you wish e.g by css/xpath etc

As the the desired element is within an <iframe> so to invoke click() on the element you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Code Block:
# as per your comment assuming -> there is only one frame on the page
WebDriverWait(self.driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME,"iframe")))
self.element = self.activity_status = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#overview > div.details.w-66 > div > div.duration-and-status > span.status.stat_incomplete#')))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a relevant detailed discussion in:
Ways to deal with #document under iframe

How to jump between two different elements located same level?

The problem is the following, in the same html I have a footer and a div located on the same level first I want to click on the img of the following code:
<footer>
<div layout="row" layout-align="center center" flex="" class="layout-align-center-center layout-row flex">
<img src="assets/images/vtr-icon-tecnico.png" class="icon" ng-click="vm.openModal($event, 'appt')" role="button" tabindex="0">
</div>
</footer>
and then click on an input of a input of the following code:
<div class="md-dialog-container ng-scope" tabindex="-1" style="top: 0px; height: 937px;">
<form name="appsForm" class="ng-pristine ng-invalid ng-invalid-required" style="">
<!-- ngIf: !links -->
<md-content class="md-padding ng-scope layout-column" layout="column" ng-if="!links" style="">
<md-input-container class="md-input-invalid"><label for="input_10557">Clave</label>
<input required="" type="password" name="password" md-maxlength="30" ng-model="key.password" class="ng-pristine md-input ng-empty ng-invalid ng-invalid-required ng-touched" id="input_10557" aria-invalid="true" ng-trim="false" style="">
</md-input-container>
</md-content>
</form>
</div>
Using xpath I tried with:
pass = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class,'ng-scope layout-column flex')]//div[#id='input_10555']")))
But the answer is the following in the console:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[contains(#class,'ng-scope layout-column flex')]//div[#id='input_10555']"}
I use Python, Selenium Web Driver and Chrome Browser
thank you who has better understanding on how to do it

The desired element is an Angular element so to locate and click() on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.ng-pristine.md-input.ng-empty.ng-invalid.ng-invalid-required.ng-touched[id^='input_'][name='password']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#class='ng-pristine md-input ng-empty ng-invalid ng-invalid-required ng-touched' and starts-with(#id,'input_')][#name='password']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Scrapy, Selenium, jQuery Dropdown inconsistent behavior with select tag having style="display: none;"

I'm trying to set dropdowns on this page:
The first dropdown and the fourth dropdown are very similar (brands and country). This is the code I'm using for getting the brand (oem) and country:
oem = Select(wd.find_element_by_css_selector("#alBrandsList"))
oem.select_by_visible_text("Acer")
countries = Select(wd.find_element_by_css_selector("#alCountriesList"))
countries.select_by_visible_text("Albania")
The dropdown is technically hidden, but it somehow seems to work for the device/oem dropdown. For the countries dropdown it is saying that the content isn't visible (which it is). Here's the HTML code it's pulling from:
<select class="pretty-dropdown" datatosent="brand" id="alBrandsList" name="alBrandsList" selectorid="alPhoneModelsList" target="/AdvanceLookup/GetPhoneModels/" style="display: none;">
...
</select>
<button type="button" class="ui-multiselect ui-widget ui-state-default ui-corner-all" aria-haspopup="true" style="width: 232px;">
<span class="ui-icon ui-icon-triangle-2-n-s"></span>
<span>Please select brand(s)</span>
</button>
<select class="pretty-dropdown" datatosent="country" id="alCountriesList" name="alCountriesList" selectorid="alCarriersList" target="/AdvanceLookup/GetCarriers/" style="display: none;">
...
</select>
<button type="button" class="ui-multiselect ui-widget ui-state-default ui-corner-all" aria-haspopup="true" style="width: 232px;">
<span class="ui-icon ui-icon-triangle-2-n-s"></span>
<span>Please select country</span>
</button>
Any idea why it works on the first one but not the second one?

As both the <select> tags for the first and the fourth dropdown on the page https://willmyphonework.net/AdvanceLookup is having the property style="display: none; you can't use Select Class. Instead you need to to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://willmyphonework.net/AdvanceLookup')
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='ui-multiselect ui-widget ui-state-default ui-corner-all']//span[text()='Please select brand(s)']"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//ul[#class='ui-multiselect-checkboxes ui-helper-reset']//li/label/input[#title='Acer']"))).click()
driver.find_element_by_xpath("//ul[#class='ui-helper-reset']//li/a/span[#class='ui-icon ui-icon-circle-close']").click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='ui-multiselect ui-widget ui-state-default ui-corner-all']//span[text()='Please select country']"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//ul[#class='ui-multiselect-checkboxes ui-helper-reset']//li/label/input[#title='Albania']"))).click()
driver.find_element_by_xpath("//div[#class='ui-multiselect-menu ui-widget ui-widget-content ui-corner-all']//following::ul[6]//a[#class='ui-multiselect-close']/span[#class='ui-icon ui-icon-circle-close']").click()
Browser Snapshot: