Only part of a table is parsed - python

Here is the begging of source code:
<div id="users_table" class="security_slick_container slickgrid_300610 ui-widget" style="overflow: hidden; outline: 0px; position: relative;">
<div tabindex="0" hidefocus="" style="position:fixed;width:0;height:0;top:0;left:0;outline:0;"></div>
<div class="slick-header ui-state-default" style="overflow:hidden;position:relative;">
<div class="slick-header-columns" style="left: -1000px; width: 2132px;" unselectable="on">
<div class="ui-state-default slick-header-column slick-header-sortable slick-header-column-sorted" id="slickgrid_300610userName" title="" style="width: 94px;"><span class="slick-column-name"><strong>Username:</strong></span><span class="slick-sort-indicator slick-sort-indicator-asc"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column slick-header-sortable" id="slickgrid_300610firstName" title="" style="width: 89px;"><span class="slick-column-name"><strong>Firstname:</strong></span><span class="slick-sort-indicator"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column slick-header-sortable" id="slickgrid_300610lastName" title="" style="width: 109px;"><span class="slick-column-name"><strong>Lastname:</strong></span><span class="slick-sort-indicator"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column slick-header-sortable" id="slickgrid_300610type" title="" style="width: 124px;"><span class="slick-column-name"><strong>Type:</strong></span><span class="slick-sort-indicator"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column slick-header-sortable" id="slickgrid_300610crew" title="" style="width: 109px;"><span class="slick-column-name"><strong>Crew:</strong></span><span class="slick-sort-indicator"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column slick-header-sortable" id="slickgrid_300610jobTitle" title="" style="width: 109px;"><span class="slick-column-name"><strong>Job title:</strong></span><span class="slick-sort-indicator"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column slick-header-sortable" id="slickgrid_300610defaultPriceClass" title="" style="width: 124px;"><span class="slick-column-name"><strong>Defaultprice class:</strong></span><span class="slick-sort-indicator"></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column" id="slickgrid_300610description" title="" style="width: 129px;"><span class="slick-column-name"><strong>Description:</strong></span>
<div class="slick-resizable-handle"></div>
</div>
<div class="ui-state-default slick-header-column" id="slickgrid_300610language" title="" style="width: 39px;"><span class="slick-column-name"><strong>Language:</strong></span>
<div class="slick-resizable-handle"></div>
</div>
</div>
</div>
<div class="slick-headerrow ui-state-default" style="overflow: hidden; position: relative; display: none;">
<div class="slick-headerrow-columns" style="width: 1115px;"></div>
<div style="display: block; height: 1px; position: absolute; top: 0px; left: 0px; width: 1132px;"></div>
</div>
<div class="slick-top-panel-scroller ui-state-default" style="overflow: hidden; position: relative; display: none;">
<div class="slick-top-panel" style="width:10000px"></div>
</div>
<div class="slick-viewport" style="width: 100%; overflow: auto; outline: 0px; position: relative; height: 567px;">
<div class="grid-canvas" style="height: 16550px; width: 1115px;">
<div class="ui-widget-content slick-row even" style="top:0px">
<div class="slick-cell l0 r0">john.smith</div>
<div class="slick-cell l1 r1">John</div>
<div class="slick-cell l2 r2">Smith</div>
<div class="slick-cell l3 r3">Contractor</div>
<div class="slick-cell l4 r4">Microsoft</div>
<div class="slick-cell l5 r5">Sales manager</div>
<div class="slick-cell l6 r6">A</div>
<div class="slick-cell l7 r7"></div>
<div class="slick-cell l8 r8">en</div>
</div>
<div class="ui-widget-content slick-row odd" style="top:25px">
<div class="slick-cell l0 r0">robert.geits</div>
<div class="slick-cell l1 r1">Robert</div>
<div class="slick-cell l2 r2">Geits</div>
<div class="slick-cell l3 r3">Staff</div>
<div class="slick-cell l4 r4">Google</div>
<div class="slick-cell l5 r5">Project manager</div>
<div class="slick-cell l6 r6">B</div>
<div class="slick-cell l7 r7"></div>
<div class="slick-cell l8 r8">de</div>
</div>
<div class="ui-widget-content slick-row even" style="top:50px">
<div class="slick-cell l0 r0">amir.rooney</div>
<div class="slick-cell l1 r1">Amir</div>
<div class="slick-cell l2 r2">Rooney</div>
<div class="slick-cell l3 r3">Staff</div>
<div class="slick-cell l4 r4">Microsoft</div>
<div class="slick-cell l5 r5">Sales manager</div>
<div class="slick-cell l6 r6">A</div>
<div class="slick-cell l7 r7"></div>
<div class="slick-cell l8 r8">en</div>
</div>
</div>
</div>
<div tabindex="0" hidefocus="" style="position:fixed;width:0;height:0;top:0;left:0;outline:0;"></div>
</div>
I am using following code for parsing the table. Any ideas why only part of table is parsed? There are a lot of records but current code is able to handle names starting with A and partly with B, but then it stops without giving any errors?
from selenium import webdriver
for tr in driver.find_elements_by_xpath('//*[#id=\"users_table\"]'):
tds = tr.find_elements_by_tag_name('div')
print ([td.text for td in tds])
I use selenium because I need to perform login before parsing.

Try with the below logic.
# get information from all rows
for tr in driver.find_elements_by_css_selector('div.slick-row'):
# get the columns in the row
tds = tr.find_elements_by_tag_name("div")
# print all td informatoin
print ([td.text for td in tds])

Related

Having several problems clicking a button with selenium in Google Chrome

I'm trying to click a button but its being really difficult. More precise, I want to do it for several buttons. I have tried several options but none is working properly. Here is the html code for the button:
<button class="rlg-trade__action rlg-trade__bump --bump " type="button" data-alias="bd520a66-cc88-4af8-ba92-30111bbdbd02" data-preventtext="Bumping…">
<svg viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><g style="stroke-linecap:round;stroke-linejoin:round;stroke:#6a717f;fill:none;stroke-miterlimit:10" transform="translate(.5 .5)"><path d="m12 23v-13"></path><path d="m16 14-4-4-4 4"></path><g stroke="#6a717f"><path d="m4 17h-3v-16h22v16h-3"></path><path d="m1 5h22"></path></g></g></svg>
<span>Bump</span>
</button>
And here is the code for scrapping:
trades_column = self.driver.find_element_by_css_selector('.rlg-trading__intersect') #Section where all trades are listed
trades_list = trades_column.find_elements(By.CLASS_NAME,'rlg-trade') #Search each trade element, all of them include the bump button posted above.
for trades in trades_list:
bump = trades.find_element_by_css_selector('.rlg-trade__action.rlg-trade__bump.--bump').click()
print('Trade bumped successfully!')
time.sleep(1)
self.driver.find_element_by_css_selector('i.fa').click() #This is a click in the page to exit a box that appear after the click.
Have tried by xpath, not working either. Any help would be appreciated.
Output error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"button.rlg-trade__action.rlg-trade__bump.--bump"}
<div class="rlg-trade" data-i="0"> its every item in the trade list section rlg-trading__intersect.
<div class="col-3-3 rlg-trading__intersect">
<div class="rlg-trade" data-i="0" style="">
<header class="rlg-trade__header
">
<a href="/player/isaacdl" class="rlg-trade__user">
<div class="rlg-trade__avatar"><img loading="lazy" class="rlg-trade__avatarimage" src="/content/media/users/avatar/128px/b35468f2331659986325.png" alt="isaacdl"></div>
<div class="rlg-trade__meta">
<div class="rlg-trade__username">
isaacdl
</div>
<span class="rlg-trade__info">
<span class="rlg-trade__time">
<span>5 hours ago</span>
<span>5 hours, 20 minutes, 40 seconds ago</span>
</span>
<span class="rlg-trade__delinfo">·
This trade will be deleted in <strong class="rlg-trade__timeleft">14 days</strong> if you don't bump it.</span>
</span>
</div>
</a>
<div class="rlg-trade__platforms">
<a target="_blank" rel="noopener" class="rlg-trade__platform" style="order: 100;" href="https://steamcommunity.com/profiles/76561198120028164" onclick="event.preventDefault();phishingAware('https://steamcommunity.com/profiles/76561198120028164');">
<img class="rlg-trade__platformlogo" src="https://static.rocket-league.com/assets/b81c8860521ff08c3d8194c2eca3491c1b158f13/images/logos/windowspc_black.svg" alt="Windows PC">
<div class="rlg-trade__platformname">
<span>Add on
Steam
</span>
<span>
auchan </span>
</div>
</a>
</div>
</header>
<div class="rlg-trade__content">
<div class="rlg-trade__labels">
<div class="rlg-trade__haslabel">
Has
</div>
<div class="rlg-trade__wantslabel">
Wants
</div>
</div>
<div class="rlg-trade__items">
<div class="rlg-trade__itemshas ">
<div class="rlg-item --very-rare --hover">
<div class="rlg-item__gradient --very-rare"></div>
<img loading="lazy" class="rlg-item__image" src="/content/media/items/avatar/220px/a67e907fb81451699877.png" alt="Cristiano ">
<div class="rlg-item__text">
<h2 class="rlg-item__name">Cristiano</h2>
</div>
<div class="rlg-item-links">
<a class="rlg-btn-primary --small" href="/items/wheels/cristiano">Item details</a>
<a class="rlg-btn-secondary --small" href="/trading/?filterItem=148&filterCertification=0&filterPaint=0&filterPlatform=0&filterItemType=1">Find trades</a>
</div>
</div>
<div class="rlg-item --very-rare --hover">
<div class="rlg-item__gradient --very-rare"></div>
<img loading="lazy" class="rlg-item__image" src="/content/media/items/avatar/220px/a67e907fb81451699877.png" alt="Cristiano ">
<div class="rlg-item__text">
<h2 class="rlg-item__name">Cristiano</h2>
</div>
<div class="rlg-item-links">
<a class="rlg-btn-primary --small" href="/items/wheels/cristiano">Item details</a>
<a class="rlg-btn-secondary --small" href="/trading/?filterItem=148&filterCertification=0&filterPaint=0&filterPlatform=0&filterItemType=1">Find trades</a>
</div>
</div>
</div>
<div class="rlg-trade__wantslabel rlg-trade__wantslabelalt">
Wants
</div>
<div class="rlg-trade__itemswants ">
<div class="rlg-item --premium --hover">
<div class="rlg-item__gradient --premium"></div>
<img loading="lazy" class="rlg-item__image" src="/content/media/items/avatar/220px/da6ecd87091575484054.png" alt="Credits ">
<div class="rlg-item__text">
<h2 class="rlg-item__name">Credits</h2>
</div>
<div class="rlg-item__quantity --quantity-80 --premium">
80 </div>
<div class="rlg-item-links">
<a class="rlg-btn-primary --small" href="/items/misc/credits">Item details</a>
<a class="rlg-btn-secondary --small" href="/trading/?filterItem=2615&filterCertification=0&filterPaint=0&filterPlatform=0&filterItemType=1">Find trades</a>
</div>
</div>
<div class="rlg-item --premium --hover">
<div class="rlg-item__gradient --premium"></div>
<img loading="lazy" class="rlg-item__image" src="/content/media/items/avatar/220px/da6ecd87091575484054.png" alt="Credits ">
<div class="rlg-item__text">
<h2 class="rlg-item__name">Credits</h2>
</div>
<div class="rlg-item__quantity --quantity-80 --premium">
80 </div>
<div class="rlg-item-links">
<a class="rlg-btn-primary --small" href="/items/misc/credits">Item details</a>
<a class="rlg-btn-secondary --small" href="/trading/?filterItem=2615&filterCertification=0&filterPaint=0&filterPlatform=0&filterItemType=1">Find trades</a>
</div>
</div>
</div>
</div>
</div>
<div class="rlg-trade__actions">
<button class="rlg-trade__action rlg-trade__bump --bump " type="button" data-alias="bd520a66-cc88-4af8-ba92-30111bbdbd02" data-preventtext="Bumping…">
<svg viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><g style="stroke-linecap:round;stroke-linejoin:round;stroke:#6a717f;fill:none;stroke-miterlimit:10" transform="translate(.5 .5)"><path d="m12 23v-13"></path><path d="m16 14-4-4-4 4"></path><g stroke="#6a717f"><path d="m4 17h-3v-16h22v16h-3"></path><path d="m1 5h22"></path></g></g></svg>
<span>Bump</span>
</button>
<a class="rlg-trade__action rlg-trade__edit --edit" href="/trade/edit?trade=bd520a66-cc88-4af8-ba92-30111bbdbd02">
<svg viewBox="0 0 12 12" xmlns="http://www.w3.org/2000/svg"><g fill="none" stroke="#6a717f" stroke-linecap="round" stroke-linejoin="round"><path d="m9.5.5 2 2-5 5-3 1 1-3z"></path><path d="m10.5 8.5v2a1 1 0 0 1 -1 1h-8a1 1 0 0 1 -1-1v-8a1 1 0 0 1 1-1h2" stroke="#6a717f"></path></g></svg>
<span>Edit trade</span>
</a>
<a class="rlg-trade__action rlg-trade__disable --disable" href="/functions/disableTrade.php?trade=bd520a66-cc88-4af8-ba92-30111bbdbd02" onclick="return confirm('Are you sure you want to disable this trade? This trade will be permanently removed.')">
<svg height="20" viewBox="0 0 24 24" width="20" xmlns="http://www.w3.org/2000/svg"><g style="stroke-linecap:round;stroke-linejoin:round;stroke-width:1.2;fill:none;stroke:#6a717f;stroke-miterlimit:10"><path d="m20.3 4.7-15.6 15.6"></path><circle cx="12.5" cy="12.5" r="11"></circle></g></svg> <span>Disable trade</span>
</a>
<a href="/trade/bd520a66-cc88-4af8-ba92-30111bbdbd02" class="rlg-trade__action --comments">
<svg viewBox="0 0 18 21" xmlns="http://www.w3.org/2000/svg"><g fill="none" fill-rule="evenodd" stroke="#545454" stroke-linecap="round" stroke-linejoin="round"><path d="m1.421053 1.421053h15.157895v18.526316h-15.157895z"></path><path d="m4.789474 4.789474h8.421053v5.052632h-8.421053z"></path><path d="m4.789474 13.210526h8.421052"></path><path d="m4.789474 16.578947h8.421052"></path></g></svg>
<span>Comments</span>
</a>
</div>
<div class="rlg-trade__note">
Have 2x of them. Please fast. </div>
<button class="rlg-trade__noteexpand" style="display: none;">Show full trade description</button>
</div>
<div class="rlg-trade rlg-trade-placeholder" data-i="1" style="height: 323px; box-shadow: none;"></div>
<div class="rlg-trade rlg-trade-placeholder" data-i="2" style="height: 323px; box-shadow: none;"></div>
<div class="rlg-trade rlg-trade-placeholder" data-i="3" style="height: 323px; box-shadow: none;"></div>
</div>
Try use following xpath to click on the element.
for trades in trades_list:
trades.find_element(By.XPATH,'.//button[.//span[text()="Bump"]]').click()
print('Trade bumped successfully!')
time.sleep(1)

unable to find correct XPATH for selenium drag and drop

I'm building a python script to help upload to beatstars quicker, however I'm having an issue locating the correct element to click on. As well as finding the XPATH. As of now I have no issue opening chrome, moving to beatstars, and pressing the upload button, however once doing so a modal pops up with a drag and drop section:
The issue I have is with locating the correct button to click to open browse or if there's a better way like just copying the file directly into the section. This is the html for this.
<mat-dialog-container
tabindex="-1"
aria-modal="true"
class="mat-dialog-container ng-tns-c25-86 ng-trigger ng-trigger-dialogContainer ng-star-inserted"
id="mat-dialog-5"
role="dialog"
style="transform: none"
><ng-component _nghost-jyk-c105="" class="ng-star-inserted"
><mat-dialog-content
_ngcontent-jyk-c105=""
class="mat-dialog-content upload-dialog-content"
><div _ngcontent-jyk-c105="" class="header">
<h3 _ngcontent-jyk-c105="">Single file upload</h3>
<button _ngcontent-jyk-c105="" class="ng-star-inserted" style="">
<i
_ngcontent-jyk-c105=""
class="vb-icon-close-m-regular-solid"
></i></button
><!---->
</div>
<mat-tab-group
_ngcontent-jyk-c105=""
class="mat-tab-group mat-primary ng-star-inserted"
style=""
><mat-tab-header class="mat-tab-header"
><div
aria-hidden="true"
mat-ripple=""
class="mat-ripple mat-tab-header-pagination mat-tab-header-pagination-before mat-elevation-z4 mat-tab-header-pagination-disabled"
>
<div class="mat-tab-header-pagination-chevron"></div>
</div>
<div class="mat-tab-label-container">
<div
role="tablist"
class="mat-tab-list"
style="transform: translateX(0px)"
>
<div class="mat-tab-labels">
<div
role="tab"
mattablabelwrapper=""
mat-ripple=""
cdkmonitorelementfocus=""
class="mat-ripple mat-tab-label mat-focus-indicator mat-tab-label-active ng-star-inserted"
id="mat-tab-label-4-0"
tabindex="0"
aria-posinset="1"
aria-setsize="4"
aria-controls="mat-tab-content-4-0"
aria-selected="true"
aria-disabled="false"
>
<div class="mat-tab-label-content">
<i
_ngcontent-jyk-c105=""
class="vb-icon-cloud-upload-file-m-regular ng-star-inserted"
></i>
Upload
<!----><!----><!---->
</div>
</div>
<div
role="tab"
mattablabelwrapper=""
mat-ripple=""
cdkmonitorelementfocus=""
class="mat-ripple mat-tab-label mat-focus-indicator ng-star-inserted"
id="mat-tab-label-4-1"
tabindex="-1"
aria-posinset="2"
aria-setsize="4"
aria-controls="mat-tab-content-4-1"
aria-selected="false"
aria-disabled="false"
>
<div class="mat-tab-label-content">
<i
_ngcontent-jyk-c105=""
class="vb-icon-drive-m-regular-solid ng-star-inserted"
></i>
Google Drive
<!----><!----><!---->
</div>
</div>
<div
role="tab"
mattablabelwrapper=""
mat-ripple=""
cdkmonitorelementfocus=""
class="mat-ripple mat-tab-label mat-focus-indicator ng-star-inserted"
id="mat-tab-label-4-2"
tabindex="-1"
aria-posinset="3"
aria-setsize="4"
aria-controls="mat-tab-content-4-2"
aria-selected="false"
aria-disabled="false"
>
<div class="mat-tab-label-content">
<i
_ngcontent-jyk-c105=""
class="vb-icon-dropbox-m-regular-solid ng-star-inserted"
></i>
Dropbox
<!----><!----><!---->
</div>
</div>
<div
role="tab"
mattablabelwrapper=""
mat-ripple=""
cdkmonitorelementfocus=""
class="mat-ripple mat-tab-label mat-focus-indicator ng-star-inserted"
id="mat-tab-label-4-3"
tabindex="-1"
aria-posinset="4"
aria-setsize="4"
aria-controls="mat-tab-content-4-3"
aria-selected="false"
aria-disabled="false"
>
<div class="mat-tab-label-content">
<i
_ngcontent-jyk-c105=""
class="vb-icon-link-m-regular-solid ng-star-inserted"
></i>
Import URL
<!----><!----><!---->
</div>
</div>
<!---->
</div>
<mat-ink-bar
class="mat-ink-bar"
style="visibility: visible; left: 0px; width: 160px"
></mat-ink-bar>
</div>
</div>
<div
aria-hidden="true"
mat-ripple=""
class="mat-ripple mat-tab-header-pagination mat-tab-header-pagination-after mat-elevation-z4 mat-tab-header-pagination-disabled"
>
<div class="mat-tab-header-pagination-chevron"></div></div
></mat-tab-header>
<div class="mat-tab-body-wrapper">
<mat-tab-body
role="tabpanel"
class="mat-tab-body ng-tns-c115-87 mat-tab-body-active ng-star-inserted"
id="mat-tab-content-4-0"
aria-labelledby="mat-tab-label-4-0"
><div
cdkscrollable=""
class="mat-tab-body-content ng-tns-c115-87 ng-trigger ng-trigger-translateTab"
style="transform: none"
>
<!----><!----><!----><!----><!----><!----><!---->
<div
_ngcontent-jyk-c105=""
class="step ng-star-inserted"
style=""
>
<div _ngcontent-jyk-c105="" class="drag-drop-container">
<div _ngcontent-jyk-c105="" class="drag-drop-message">
<div _ngcontent-jyk-c105="" class="icons">
<i
_ngcontent-jyk-c105=""
class="vb-icon-cloud-upload-file-m-regular"
></i>
</div>
<h4 _ngcontent-jyk-c105="">
Drag & Drop, or <a _ngcontent-jyk-c105=""> Browse </a>
</h4>
<label
_ngcontent-jyk-c105=""
class="file-type ng-star-inserted"
><!----><span
_ngcontent-jyk-c105=""
class="ng-star-inserted"
>
.mp3 or .wav </span
><!----><!----><!----><!----><!----><!----><!----><!----></label
><!---->
</div>
<div
_ngcontent-jyk-c105=""
id="uppy-drag-drop"
data-qa="uppy_drag_drop"
class="uppy-box"
>
<button
type="button"
class="uppy-Root uppy-u-reset uppy-DragDrop-container uppy-DragDrop--isDragDropSupported"
style="width: 100%; height: 100%"
>
<input
class="uppy-DragDrop-input"
type="file"
hidden=""
name="files[]"
multiple=""
accept="audio/mp3,audio/x-mp3,audio/mpeg,audio/mpeg3,audio/x-mpeg-3,audio/wav,audio/x-wav,audio/wave,audio/x-wave,audio/vnd.wave,application/octet-stream"
/>
<div class="uppy-DragDrop-inner">
<svg
aria-hidden="true"
focusable="false"
class="uppy-c-icon uppy-DragDrop-arrow"
width="16"
height="16"
viewBox="0 0 16 16"
>
<path
d="M11 10V0H5v10H2l6 6 6-6h-3zm0 0"
fillRule="evenodd"
></path>
</svg>
<div class="uppy-DragDrop-label">
Drop files here or
<span class="uppy-DragDrop-browse">browse</span>
</div>
<span class="uppy-DragDrop-note"></span>
</div>
</button>
</div>
</div>
<div _ngcontent-jyk-c105="" class="my-files ng-star-inserted">
<label _ngcontent-jyk-c105=""> My files </label
><bs-secondary-uploaded-files
_ngcontent-jyk-c105=""
_nghost-jyk-c136=""
class="ng-star-inserted"
style=""
><div _ngcontent-jyk-c136="" class="uploaded-files">
<ul
_ngcontent-jyk-c136=""
class="files-uploaded ng-star-inserted"
>
<li
_ngcontent-jyk-c136=""
id="row-0"
class="ng-star-inserted"
>
<div _ngcontent-jyk-c136="" class="left">
<div _ngcontent-jyk-c136="" class="file-type-icon">
<i
_ngcontent-jyk-c136=""
class="icon-music-4"
></i>
</div>
<div _ngcontent-jyk-c136="" class="file-info">
<span _ngcontent-jyk-c136="" class="file-name"
>j-cole-type-beat</span
>
<div
_ngcontent-jyk-c136=""
class="secondary-info ng-star-inserted"
>
<span _ngcontent-jyk-c136="">26.00 MB</span
><span
_ngcontent-jyk-c136=""
class="ng-star-inserted"
> ∙ </span
><!----><span _ngcontent-jyk-c136=""
>Sep 2, 2022, 6:05 PM</span
>
</div>
<!---->
</div>
</div>
<div _ngcontent-jyk-c136="" class="right">
<bs-menu-more-options
_ngcontent-jyk-c136=""
class="menu-more-options"
_nghost-jyk-c93=""
><div
_ngcontent-jyk-c93=""
aria-haspopup="true"
class="mat-menu-trigger wrapper-button"
style="margin-left: 0px; margin-right: 0px"
>
<button
_ngcontent-jyk-c93=""
mat-icon-button=""
class="mat-focus-indicator mat-tooltip-trigger mat-icon-button mat-button-base ng-star-inserted"
aria-describedby="cdk-describedby-message-9"
cdk-describedby-host=""
>
<span class="mat-button-wrapper"
><i
_ngcontent-jyk-c93=""
class="icon-dots ng-star-inserted"
></i
><!----></span
><span
matripple=""
class="mat-ripple mat-button-ripple mat-button-ripple-round"
></span
><span
class="mat-button-focus-overlay"
></span></button
><!----><!---->
</div>
<!----><mat-menu
_ngcontent-jyk-c93=""
class="ng-star-inserted"
><!----></mat-menu
><!----><!----></bs-menu-more-options
><bs-square-button
_ngcontent-jyk-c136=""
buttontype="button"
class="btn-select ng-star-inserted"
_nghost-jyk-c103=""
><button
_ngcontent-jyk-c103=""
bssquarebuttonfocus=""
type="button"
class="action-element primary ng-star-inserted"
>
Select
<!----></button
><!----><!----><!----></bs-square-button
><!----><!---->
</div>
<div
_ngcontent-jyk-c136=""
class="border-bottom"
></div>
</li>
<li
_ngcontent-jyk-c136=""
id="row-1"
class="ng-star-inserted"
>
<div _ngcontent-jyk-c136="" class="left">
<div _ngcontent-jyk-c136="" class="file-type-icon">
<i
_ngcontent-jyk-c136=""
class="icon-music-4"
></i>
</div>
<div _ngcontent-jyk-c136="" class="file-info">
<span _ngcontent-jyk-c136="" class="file-name"
>island-girl</span
>
<div
_ngcontent-jyk-c136=""
class="secondary-info ng-star-inserted"
>
<span _ngcontent-jyk-c136="">27.10 MB</span
><span
_ngcontent-jyk-c136=""
class="ng-star-inserted"
> ∙ </span
><!----><span _ngcontent-jyk-c136=""
>Aug 31, 2022, 10:47 PM</span
>
</div>
<!---->
</div>
</div>
<div _ngcontent-jyk-c136="" class="right">
<bs-menu-more-options
_ngcontent-jyk-c136=""
class="menu-more-options"
_nghost-jyk-c93=""
><div
_ngcontent-jyk-c93=""
aria-haspopup="true"
class="mat-menu-trigger wrapper-button"
style="margin-left: 0px; margin-right: 0px"
>
<button
_ngcontent-jyk-c93=""
mat-icon-button=""
class="mat-focus-indicator mat-tooltip-trigger mat-icon-button mat-button-base ng-star-inserted"
aria-describedby="cdk-describedby-message-9"
cdk-describedby-host=""
>
<span class="mat-button-wrapper"
><i
_ngcontent-jyk-c93=""
class="icon-dots ng-star-inserted"
></i
><!----></span
><span
matripple=""
class="mat-ripple mat-button-ripple mat-button-ripple-round"
></span
><span
class="mat-button-focus-overlay"
></span></button
><!----><!---->
</div>
<!----><mat-menu
_ngcontent-jyk-c93=""
class="ng-star-inserted"
><!----></mat-menu
><!----><!----></bs-menu-more-options
><bs-square-button
_ngcontent-jyk-c136=""
buttontype="button"
class="btn-select ng-star-inserted"
_nghost-jyk-c103=""
><button
_ngcontent-jyk-c103=""
bssquarebuttonfocus=""
type="button"
class="action-element primary ng-star-inserted"
>
Select
<!----></button
><!----><!----><!----></bs-square-button
><!----><!---->
</div>
<div
_ngcontent-jyk-c136=""
class="border-bottom"
></div>
</li>
<li
_ngcontent-jyk-c136=""
id="row-2"
class="ng-star-inserted"
>
<div _ngcontent-jyk-c136="" class="left">
<div _ngcontent-jyk-c136="" class="file-type-icon">
<i
_ngcontent-jyk-c136=""
class="icon-music-4"
></i>
</div>
<div _ngcontent-jyk-c136="" class="file-info">
<span _ngcontent-jyk-c136="" class="file-name"
>island-girl</span
>
<div
_ngcontent-jyk-c136=""
class="secondary-info ng-star-inserted"
>
<span _ngcontent-jyk-c136="">3.69 MB</span
><span
_ngcontent-jyk-c136=""
class="ng-star-inserted"
> ∙ </span
><!----><span _ngcontent-jyk-c136=""
>Aug 31, 2022, 10:47 PM</span
>
</div>
<!---->
</div>
</div>
<div _ngcontent-jyk-c136="" class="right">
<bs-menu-more-options
_ngcontent-jyk-c136=""
class="menu-more-options"
_nghost-jyk-c93=""
><div
_ngcontent-jyk-c93=""
aria-haspopup="true"
class="mat-menu-trigger wrapper-button"
style="margin-left: 0px; margin-right: 0px"
>
<button
_ngcontent-jyk-c93=""
mat-icon-button=""
class="mat-focus-indicator mat-tooltip-trigger mat-icon-button mat-button-base ng-star-inserted"
aria-describedby="cdk-describedby-message-9"
cdk-describedby-host=""
>
<span class="mat-button-wrapper"
><i
_ngcontent-jyk-c93=""
class="icon-dots ng-star-inserted"
></i
><!----></span
><span
matripple=""
class="mat-ripple mat-button-ripple mat-button-ripple-round"
></span
><span
class="mat-button-focus-overlay"
></span></button
><!----><!---->
</div>
<!----><mat-menu
_ngcontent-jyk-c93=""
class="ng-star-inserted"
><!----></mat-menu
><!----><!----></bs-menu-more-options
><bs-square-button
_ngcontent-jyk-c136=""
buttontype="button"
class="btn-select ng-star-inserted"
_nghost-jyk-c103=""
><button
_ngcontent-jyk-c103=""
bssquarebuttonfocus=""
type="button"
class="action-element primary ng-star-inserted"
>
Select
<!----></button
><!----><!----><!----></bs-square-button
><!----><!---->
</div>
<div
_ngcontent-jyk-c136=""
class="border-bottom"
></div>
</li>
<!---->
</ul>
<!----><!---->
</div></bs-secondary-uploaded-files
><!----><!---->
</div>
<!---->
</div>
<!----><!----><!----><!----><!---->
</div></mat-tab-body
><mat-tab-body
role="tabpanel"
class="mat-tab-body ng-tns-c115-88 ng-star-inserted"
id="mat-tab-content-4-1"
aria-labelledby="mat-tab-label-4-1"
><div
cdkscrollable=""
class="mat-tab-body-content ng-tns-c115-88 ng-trigger ng-trigger-translateTab"
style="transform: translate3d(100%, 0px, 0px); min-height: 1px"
>
<!---->
</div></mat-tab-body
><mat-tab-body
role="tabpanel"
class="mat-tab-body ng-tns-c115-89 ng-star-inserted"
id="mat-tab-content-4-2"
aria-labelledby="mat-tab-label-4-2"
><div
cdkscrollable=""
class="mat-tab-body-content ng-tns-c115-89 ng-trigger ng-trigger-translateTab"
style="transform: translate3d(100%, 0px, 0px); min-height: 1px"
>
<!---->
</div></mat-tab-body
><mat-tab-body
role="tabpanel"
class="mat-tab-body ng-tns-c115-90 ng-star-inserted"
id="mat-tab-content-4-3"
aria-labelledby="mat-tab-label-4-3"
><div
cdkscrollable=""
class="mat-tab-body-content ng-tns-c115-90 ng-trigger ng-trigger-translateTab"
style="transform: translate3d(100%, 0px, 0px); min-height: 1px"
>
<!---->
</div></mat-tab-body
><!---->
</div></mat-tab-group
><!----><!----><!----><!----></mat-dialog-content
></ng-component
><!----></mat-dialog-container
>
I'm new to selenium and can't find the write thing to try and click on:
This is my code:
browser = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver')
browser.get(url)
wait = WebDriverWait(browser, 20)
email = browser.find_element(By.ID, "oath-email").send_keys(username, Keys.ENTER)
password = wait.until(EC.presence_of_element_located((By.ID, "userPassword"))).send_keys(password, Keys.ENTER)
new_track = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body/studio-root/div/ng-component/studio-header/header/div/div/bs-square-button/button"))).click()
upload = wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/studio-root/div/ng-component/studio-page-container/div/form/studio-inventory-form-holder/div/studio-panel/div/mat-tab-group/div/mat-tab-body[1]/div/studio-wrapper-track-files/studio-form-files/div/section[1]/div/studio-form-file-box/div/div/div[2]/bs-upload-button/bs-universal-upload-button/div/div/bs-square-button/button'))).click()
drag_and_drop = wait.until(EC.element_to_be_clickable((By.XPATH, '/html/body/div[4]/div[8]/div/mat-dialog-container/ng-component/mat-dialog-content/mat-tab-group/div/mat-tab-body[1]/div/div/div[1]/div[2]/button'))).click()

How to click on dynamic button using selenium Python?

I am using Selenium Python and trying to access one of the dynamic created buttons which have same names and no ID.
By.XPATH and By.CLASS_NAME not working here. Any suggestions how can I click on this.This is the button I want to click
This is how I am trying
btn=WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="BsGnbProv"]/div[1]/div[13]/div[2]/div[3]/div[3]/button[2]')))
This is xPath
//*[#id="BsGnbProv"]/div[1]/div[13]/div[2]/div[3]/div[3]/button[2]
This is Button element details
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable:
false, click: ViewNetworkFuncModal, visible: $root.IsReadOnly"
style="white-space: normal;">
<span class="mr-1 fa fa-edit">
</span>
<span>View Configuration</span>
</button>
This is the html of this div containing all buttons
<div data-bind="visible: GeneralProperties.NfCount() > 0, css: { 'bs-inactive': $root.IsInactive }" data-csv-category="Network Function Properties" class="card m-1 flex-shrink-0 ns-toggle"><div class="card-header"><label>Network Function Properties</label></div>
<div class="overflow-auto card-body ns-width-md">
<div class="form-group ns-width-md">
<div class="col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label class="d-block text-truncate property-label text-dark" data-placement="top" data-toggle="tooltip" style="text-decoration: underline;" title="" data-original-title="Function Type">Function Type</label>
</div>
<div class="col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label class="d-block text-truncate property-label text-dark" data-placement="top" data-toggle="tooltip" style="text-decoration: underline;" title="" data-original-title="Managed Element ID">Managed Element ID</label>
</div>
</div>
<!-- ko with: GeneralProperties.PuProperties -->
<!-- ko foreach: NetworkFunctionDetailsList -->
<div style="min-width: 480px;" class="form-group ">
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: NfTypeDesc, attr: { title: NfTypeDesc }" data-csv-label="Function Type" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="xPU">xPU</label>
</div>
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: Id, attr: { title: Id }" data-csv-label="Managed Element ID" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="EAB8620065F0">EAB8620065F0</label>
</div>
<div class="col col-12 col-sm-4 col-md-4 col-lg-4 col-xl-4">
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: $root.IsReadOnly, click: LoadNetworkFuncModal, visible: !$root.IsReadOnly()" style="white-space: normal; display: none;" disabled="">
<span class="mr-1 fa fa-edit"></span>
<span>Edit Configuration</span></button>
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: false, click: ViewNetworkFuncModal, visible: $root.IsReadOnly" style="white-space: normal;">
<span class="mr-1 fa fa-edit"></span>
<span>View Configuration</span></button>
</div>
</div>
<!-- /ko -->
<!-- /ko -->
<!-- ko with: GeneralProperties.CuCpProperties -->
<!-- ko foreach: NetworkFunctionDetailsList -->
<div style="min-width: 480px;" class="form-group ">
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: NfTypeDesc, attr: { title: NfTypeDesc }" data-csv-label="Function Type" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="CU-CP">CU-CP</label>
</div>
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: Id, attr: { title: Id }" data-csv-label="Managed Element ID" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="CUCP-at2200-eab8620065f0-1">CUCP-at2200-eab8620065f0-1</label>
</div>
<div class="col col-12 col-sm-4 col-md-4 col-lg-4 col-xl-4">
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: $root.IsReadOnly, click: LoadNetworkFuncModal, visible: !$root.IsReadOnly()" style="white-space: normal; display: none;" disabled="">
<span class="mr-1 fa fa-edit"></span>
<span>Edit Configuration</span></button>
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: false, click: ViewNetworkFuncModal, visible: $root.IsReadOnly" style="white-space: normal;">
<span class="mr-1 fa fa-edit"></span>
<span>View Configuration</span></button>
</div>
</div>
<!-- /ko -->
<!-- /ko -->
<!-- ko with: GeneralProperties.CuUpProperties -->
<!-- ko foreach: NetworkFunctionDetailsList -->
<div style="min-width: 480px;" class="form-group ">
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: NfTypeDesc, attr: { title: NfTypeDesc }" data-csv-label="Function Type" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="CU-UP">CU-UP</label>
</div>
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: Id, attr: { title: Id }" data-csv-label="Managed Element ID" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="CUUP-at2200-eab8620065f0-1">CUUP-at2200-eab8620065f0-1</label>
</div>
<div class="col col-12 col-sm-4 col-md-4 col-lg-4 col-xl-4">
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: $root.IsReadOnly, click: LoadNetworkFuncModal, visible: !$root.IsReadOnly()" style="white-space: normal; display: none;" disabled="">
<span class="mr-1 fa fa-edit"></span>
<span>Edit Configuration</span></button>
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: false, click: ViewNetworkFuncModal, visible: $root.IsReadOnly" style="white-space: normal;">
<span class="mr-1 fa fa-edit"></span>
<span>View Configuration</span></button>
</div>
</div>
<!-- /ko -->
<!-- /ko -->
<!-- ko with: GeneralProperties.DuProperties -->
<!-- ko foreach: NetworkFunctionDetailsList -->
<div style="min-width: 480px;" class="form-group ">
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: NfTypeDesc, attr: { title: NfTypeDesc }" data-csv-label="Function Type" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="DU">DU</label>
</div>
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: Id, attr: { title: Id }" data-csv-label="Managed Element ID" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="DU-at2200-eab8620065f0-1">DU-at2200-eab8620065f0-1</label>
</div>
<div class="col col-12 col-sm-4 col-md-4 col-lg-4 col-xl-4">
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: $root.IsReadOnly, click: LoadNetworkFuncModal, visible: !$root.IsReadOnly()" style="white-space: normal; display: none;" disabled="">
<span class="mr-1 fa fa-edit"></span>
<span>Edit Configuration</span></button>
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: false, click: ViewNetworkFuncModal, visible: $root.IsReadOnly" style="white-space: normal;">
<span class="mr-1 fa fa-edit"></span>
<span>View Configuration</span></button>
</div>
</div>
<!-- /ko -->
<!-- /ko -->
<!-- ko with: GeneralProperties.RuProperties -->
<!-- ko foreach: NetworkFunctionDetailsList -->
<div style="min-width: 480px;" class="form-group ">
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: NfTypeDesc, attr: { title: NfTypeDesc }" data-csv-label="Function Type" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="RU">RU</label>
</div>
<div class="d-inline-flex align-items-center col col-12 col-sm-3 col-md-3 col-lg-3 col-xl-3">
<label data-bind="text: Id, attr: { title: Id }" data-csv-label="Managed Element ID" class="property-label d-block text-truncate text-dark" data-toggle="tooltip" data-placement="bottom" title="" data-original-title="RU-at2200-eab8620065f0-1">RU-at2200-eab8620065f0-1</label>
</div>
<div class="col col-12 col-sm-4 col-md-4 col-lg-4 col-xl-4">
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: $root.IsReadOnly, click: LoadNetworkFuncModal, visible: !$root.IsReadOnly()" style="white-space: normal; display: none;" disabled="">
<span class="mr-1 fa fa-edit"></span>
<span>Edit Configuration</span></button>
<button class="w-100 btn btn-light btn-sm mr-1" data-bind="disable: false, click: ViewNetworkFuncModal, visible: $root.IsReadOnly" style="white-space: normal;">
<span class="mr-1 fa fa-edit"></span>
<span>View Configuration</span></button>
</div>
</div>
<!-- /ko -->
<!-- /ko -->
</div>
</div>
#Atif, thanks for the detail HTML. Can you try the following XPath:
//button/span[text()='View Configuration']
This will give you the five buttons and then you can take action accordingly. I tried this and it is working fine.

how do i get all the href links in this div

i am trying to get the links for each restaurant on the search results page for any city on food.grab.com using selenium. This is the div i need to target to get the links but when i run this
rl_elements = driver.find_elements_by_css_selector("ant-layout")
I get an error about the CSS element not being found. I was thinking I could grab that div then probably loop through or use the get("href") tag. This is the div section with the links of the restaurants
<div class="ant-layout">
<div>
<div class="ant-row-flex RestaurantListRow___1SbZY">
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/mcdonald-s-sta-cruz-church-delivery/AWiD5JwQfYWaYaQC4nY4" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="promoTag___IYhfm">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order McDonald's - Sta Cruz Church" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/AWiD5JwQfYWaYaQC4nY4/hero/20a3b455671746a1b02a2cac627b5a67_1600322490671317355.jpg"></div>
<div class="promoTagHead___1bjRG">Promo</div>
<div class="promoTagTail___2Jy3D"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">McDonald's - Sta Cruz Church</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">Fast Food, Burgers, American, #Combodeals, #ComboDealsBurgers</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.4
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
29 mins • 0.8 km
</div>
</div>
</div>
<div class="basicInfoRow___UZM8d discount___3h-0m">
<div class="medium___3F_Er discountIcon infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-promo-tag.svg");"></div>
<span class="discountText___GQCkj">P150 OFF: Use code PADALOVE, min 600</span>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/jollibee-raon-delivery/2-CZC1EZEYACBTCX" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order Jollibee - Raon" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/2-CZC1EZEYACBTCX/hero/3189897a1a264165af3c554991970723_1634664559764732889.jpg"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">Jollibee - Raon</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">Filipino, Chicken, Fast Food</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.3
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
33 mins • 0.1 km
</div>
</div>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/mang-inasal-tayuman-delivery/2-CZCVUA42FBUTTX" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="promoTag___IYhfm">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order Mang Inasal - Tayuman" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/2-CZCVUA42FBUTTX/hero/40d7b143ab2846a490c52bec6b823cfe_1634227146999336525.png"></div>
<div class="promoTagHead___1bjRG">Promo</div>
<div class="promoTagTail___2Jy3D"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">Mang Inasal - Tayuman</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">Filipino, Chicken, Fast Food</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.5
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
31 mins • 2.3 km
</div>
</div>
</div>
<div class="basicInfoRow___UZM8d discount___3h-0m">
<div class="medium___3F_Er discountIcon infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-promo-tag.svg");"></div>
<span class="discountText___GQCkj">Promo Discount - Paa Large Buddy Size + Pork BBQ Buddy Size</span>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/angel-s-pizza-legarda-available-for-long-distance-delivery-delivery/PHGFSTI000000zw" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order Angel's Pizza - Legarda [Available for LONG-DISTANCE DELIVERY]" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/PHGFSTI000000zw/hero/2166e88cc82b4a8fb3193f1b555002fe_1624797732725799594.jpg"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">Angel's Pizza - Legarda [Available for LONG-DISTANCE DELIVERY]</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">Quick Bites, Pizza, Italian, #ComboDealsPizza</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.6
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
43 mins • 1.4 km
</div>
</div>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/wendy-s-dapitan-available-for-long-distance-delivery-delivery/PHGFSTI0000017w" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="promoTag___IYhfm">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order Wendy's - Dapitan [Available for LONG-DISTANCE DELIVERY]" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/PHGFSTI0000017w/hero/9cf2d2e2170d4c288706e4ec08740199_1604489462614383246.jpeg"></div>
<div class="promoTagHead___1bjRG">Promo</div>
<div class="promoTagTail___2Jy3D"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">Wendy's - Dapitan [Available for LONG-DISTANCE DELIVERY]</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">Fast Food, Burgers, American, #ComboDealsBurgers</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.4
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
30 mins • 1.5 km
</div>
</div>
</div>
<div class="basicInfoRow___UZM8d discount___3h-0m">
<div class="medium___3F_Er discountIcon infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-promo-tag.svg");"></div>
<span class="discountText___GQCkj">P150 OFF: Use code PADALOVE, min 600</span>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/kfc-sta-cruz-delivery/2-CYUZC8BTGJ51GJ" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="promoTag___IYhfm">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order KFC - Sta Cruz" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/2-CYUZC8BTGJ51GJ/hero/680a96e5303d4434af6c9ea5751b211e_1593015057999865942.jpeg"></div>
<div class="promoTagHead___1bjRG">Promo</div>
<div class="promoTagTail___2Jy3D"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">KFC - Sta Cruz</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">American, Chicken, Fast Food</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.1
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
27 mins • 0.8 km
</div>
</div>
</div>
<div class="basicInfoRow___UZM8d discount___3h-0m">
<div class="medium___3F_Er discountIcon infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-promo-tag.svg");"></div>
<span class="discountText___GQCkj">P150 OFF: Use code PADALOVE, min 600</span>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/chowking-sta-cruz-manila-delivery/2-CYMHN7DWAGL2WE" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order Chowking - Sta Cruz Manila" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/2-CYMHN7DWAGL2WE/hero/370cba7acb8645d99e045d93121555cd_1632276886062046084.jpg"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">Chowking - Sta Cruz Manila</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">Chinese, Chicken, Fast Food</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.5
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
31 mins • 0.6 km
</div>
</div>
</div>
</div>
</div>
</a>
</div>
<div class="ant-col-24 RestaurantListCol___1FZ8V ant-col-md-12 ant-col-lg-6">
<a href="/ph/en/restaurant/greenwich-binondo-delivery/2-CYKTRUEHNAUGEX" style="color: inherit; text-decoration: none;">
<div class="ant-row-flex ant-row-flex-start ant-row-flex-top asList___1ZNTr">
<div class="ant-col-24 colPhoto___3vb-o ant-col-md-24 ant-col-lg-24">
<div class="promoTag___IYhfm">
<div class="placeholder___1xbBh restoPhoto___3nncy"><img alt="Order Greenwich - Binondo" class="realImage___2TyNE show___3oA6B" src="https://d1sag4ddilekf6.azureedge.net/compressed/merchants/2-CYKTRUEHNAUGEX/hero/upload-photo-Hero_Photo_66bf96e0fcff47da8b6aacb925f913a3_1554863645049377072.jpeg"></div>
<div class="promoTagHead___1bjRG">Promo</div>
<div class="promoTagTail___2Jy3D"></div>
</div>
</div>
<div class="ant-col-24 colInfo___3iLqj ant-col-md-24 ant-col-lg-24">
<h6 class="name___2epcT">Greenwich - Binondo</h6>
<div class="basicInfoContainer___1DcNs">
<div class="basicInfoRow___UZM8d cuisine___T2tCh">American, Pizza, Fast Food</div>
<div class="basicInfoRow___UZM8d numbers___2xZGn">
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er ratingStar infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-star.svg");"></div>
4.5
</div>
<div class="numbersChild___2qKMV">
<div class="medium___3F_Er deliveryClock infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-clock.svg");"></div>
34 mins • 1.6 km
</div>
</div>
</div>
<div class="basicInfoRow___UZM8d discount___3h-0m">
<div class="medium___3F_Er discountIcon infoItemIcon___23Zvv" role="button" tabindex="0" style="display: inline-block; background-position: center center; background-repeat: no-repeat; background-size: contain; background-image: url("/static/images/icons/icon-promo-tag.svg");"></div>
<span class="discountText___GQCkj">Lasagna Supreme Flash Sale: two (2) Greenwich Lasagna Supreme Small Pans for only P399!</span>
</div>
</div>
</div>
</a>
</div>
</div>
<button type="button" class="ant-btn ant-btn-block"><span>Load More</span></button>
</div>
</div>
I am currently using this, but it gets all the links on the search result page, i want the ones in the div above.
url_elements = driver.find_elements_by_xpath("//a[contains(#href, '/ph/en/restaurant')]")
url_elements_list = []
for url_element in url_elements:
url_elements_list.append(url_element.get_attribute("href"))
sliced_list=url_elements_list[:2]
print(sliced_list)
To get the restaurants elements inside the specific div element you should do as following:
Apply find_elements_by_xpath method on rl_element web element instead of applying it on web driver instance driver.
The XPath locator should start with a dot . meaning "search from this node, not form the beginning of the whole page.
rl_element = driver.find_element_by_class_name("ant-layout")
url_elements = rl_element.find_elements_by_xpath(".//a[contains(#href, '/ph/en/restaurant')]")
url_elements_list = []
for url_element in url_elements:
url_elements_list.append(url_element.get_attribute("href"))
sliced_list=url_elements_list[:2]
print(sliced_list)
Use those line of code Edited:
url_elements_list = []
for url_element in driver.find_elements_by_class_name('ant-layout')::
url_elements_list.append(url_element.get_attribute("href"))
print(url_elements_list)

how to extract data from two html page?

I want to extract data from two html page .As I extact data from one page and going another page some element change ,data are present in list and list changes.
My code for below problem
details_containers = soup_page.findAll("div",{"id":"RESTAURANT_DETAILS"})
details_container = details_containers[0].findAll("div",{"class":"content"})
cuisine = details_container[0].text.strip()
print(cuisine)
meals = details_container[1].text.strip()
print(meals)
hotel_features = details_container[2].text.strip()
print(hotel_features)
From first html I want cuisine ,meals , retaurant_features content values . But there are some extra content values of hours,average prices.
<div id="RESTAURANT_DETAILS" class="content_block details_block scroll_tabs" data-tab="TABS_DETAILS">
<div class="header_with_improve wrap">
<a href="/UpdateListing-g297595-d6384395-Ocellus-Raipur_Raipur_District_Chhattisgarh.html" onclick="ta.setEvtCookie('UpdateListing', 'entry-detail-moreinfo', null, 0, '/UpdateListingRedesign')">
<div class="improve_listing_btn ui_button primary">Improve this listing</div>
</a>
<h3 class="tabs_header">Restaurant Details</h3> </div>
<div class="details_tab">
<div class="table_section">
<div class="row">
<div class="ratingSummary wrap">
<div class="histogramCommon bubbleHistogram wrap">
<div class="colTitle">
Rating summary
</div>
<ul class="barChart">
<li>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Food</span>
</div>
<div class="wrap row part ">
<span class="ui_bubble_rating bubble_45" alt="4.5 of 5 bubbles"></span>
</div>
</div>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Service</span>
</div>
<div class="wrap row part ">
<span class="ui_bubble_rating bubble_35" alt="3.5 of 5 bubbles"></span>
</div>
</div>
</li>
<li>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Value</span>
</div>
<div class="wrap row part ">
<span class="ui_bubble_rating bubble_35" alt="3.5 of 5 bubbles"></span>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="row">
<div class="title">
Average prices
</div>
<div class="content">
<span>₹ 448 -
₹ 768</span>
</div>
</div>
<div class="row">
<div class="title">
Cuisine
</div>
<div class="content">
Indian, Asian, Italian, French, Chinese, International, Vegetarian Friendly
</div>
</div>
<div class="row">
<div class="title">
Meals
</div>
<div class="content">
Breakfast, Lunch, Dinner, Brunch
</div>
</div>
<div class="row">
<div class="title">
Restaurant features
</div>
<div class="content">
Reservations, Seating, Takeout, Private Dining, Waitstaff
</div>
</div>
<div class="row">
<div class="title">
Good for
</div>
<div class="content">
Groups, Business meetings, Child-friendly
</div>
</div>
<div class="row">
<div class="hours title">
Open Hours
</div>
<div class="hours content">
<div class="detail">
<span class="day">Sunday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
<div class="detail">
<span class="day">Monday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
<div class="detail">
<span class="day">Tuesday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
<div class="detail">
<span class="day">Wednesday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
<div class="detail">
<span class="day">Thursday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
<div class="detail">
<span class="day">Friday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
<div class="detail">
<span class="day">Saturday</span>
<span class="hours"><div class="hoursRange">07:00 - 23:00</div></span>
</div>
</div>
</div>
</div>
<div class="additional_info">
<div class="title">
Location and Contact Information </div>
<div class="content">
<ul class="detailsContent">
<li>
<div class="detail">Address:
<span> <span class="format_address"><span class="street-address">G.E. Road</span> | <span class="extended-address">Mayura Hotel</span>, <span class="locality">Raipur 492001, </span><span class="country-name">India</span> </span>
</span>
</div>
</li>
<li>
<div class="detail">Location:
<span> Asia</span>
<span> > India</span>
<span> > Chhattisgarh</span>
<span> > Raipur District</span>
<span> > Raipur</span>
</div>
</li>
<li>
<div class="detail">Phone Number:
<span>+91 77142 00500</span>
</div>
</li>
<li>
<span class="ui_icon email"></span>
<a target="_blank"" href="mailto:banquet#themayurahotels.com" onclick="ta.trackEventOnPage('Eatery_Listing','Email','6384395')">
E-mail </a>
</li>
<!--trkP:waypoint_for_poi_2-->
<!-- PLACEMENT waypoint_for_poi -->
<div id="taplc_waypoint_for_poi_1" class="ppr_rup ppr_priv_waypoint_for_poi" data-placement-name="waypoint_for_poi">
</div>
<!--etk-->
</ul>
</div>
</div>
<!--[if lte IE 9]>
<style>
.details_block .threeColumnList{
height: 350px;
overflow: auto;
}
</style>
<![endif]-->
</div>
</div>
From second html I want cuisine ,meals , retaurant_features content values like above html. But in this extra content values of hours,average prices are not present
<div id="RESTAURANT_DETAILS" class="content_block details_block scroll_tabs" data-tab="TABS_DETAILS">
<div class="header_with_improve wrap">
<a href="/UpdateListing-g297595-d8595502-Barbeque_Nation-Raipur_Raipur_District_Chhattisgarh.html" onclick="ta.setEvtCookie('UpdateListing', 'entry-detail-moreinfo', null, 0, '/UpdateListingRedesign')">
<div class="improve_listing_btn ui_button primary">Improve this listing</div>
</a>
<h3 class="tabs_header">Restaurant Details</h3> </div>
<div class="details_tab">
<div class="table_section">
<div class="row">
<div class="ratingSummary wrap">
<div class="histogramCommon bubbleHistogram wrap">
<div class="colTitle">
Rating summary
</div>
<ul class="barChart">
<li>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Food</span>
</div>
<div class="wrap row part ">
<span class="ui_bubble_rating bubble_45" alt="4.5 of 5 bubbles"></span>
</div>
</div>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Service</span>
</div>
<div class="wrap row part ">
<span class="ui_bubble_rating bubble_45" alt="4.5 of 5 bubbles"></span>
</div>
</div>
</li>
<li>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Value</span>
</div>
<div class="wrap row part ">
<span class="ui_bubble_rating bubble_40" alt="4.0 of 5 bubbles"></span>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="row">
<div class="title">
Cuisine
</div>
<div class="content">
Indian, Barbecue, Asian, Vegetarian Friendly, Vegan Options, Gluten Free Options
</div>
</div>
<div class="row">
<div class="title">
Meals
</div>
<div class="content">
Lunch, Dinner
</div>
</div>
<div class="row">
<div class="title">
Restaurant features
</div>
<div class="content">
Reservations, Seating, Waitstaff, Wheelchair Accessible, Validated Parking
</div>
</div>
<div class="row">
<div class="title">
Good for
</div>
<div class="content">
Groups, Special Occasion Dining, Kids, Child-friendly
</div>
</div>
</div>
<div class="additional_info">
<div class="title">
Location and Contact Information </div>
<div class="content">
<ul class="detailsContent">
<li>
<div class="detail">Address:
<span> <span class="format_address"> | <span class="extended-address">Magneto The Mall, 2nd Floor</span>, <span class="locality">Raipur 429010, </span><span class="country-name">India</span> </span>
</span>
</div>
</li>
<li>
<div class="detail">Location:
<span> Asia</span>
<span> > India</span>
<span> > Chhattisgarh</span>
<span> > Raipur District</span>
<span> > Raipur</span>
</div>
</li>
<li>
<div class="detail">Phone Number:
<span>+91 77160 60008</span>
</div>
</li>
<li>
<span class="ui_icon email"></span>
<a target="_blank"" href="mailto:feedback#barbeque-nation.com" onclick="ta.trackEventOnPage('Eatery_Listing','Email','8595502')">
E-mail </a>
</li>
<!--trkP:waypoint_for_poi_2-->
<!-- PLACEMENT waypoint_for_poi -->
<div id="taplc_waypoint_for_poi_1" class="ppr_rup ppr_priv_waypoint_for_poi" data-placement-name="waypoint_for_poi">
</div>
<!--etk-->
</ul>
</div>
</div>
<!--[if lte IE 9]>
<style>
.details_block .threeColumnList{
height: 350px;
overflow: auto;
}
</style>
<![endif]-->
</div>
</div>
Instead of obtaining a list of all <div class="content"> blocks and selecting several by their index (which is changing from the first page to the second), you can find all <div class="row">, which contain a title and the respective content.
rows = details_container.findAll('div', {'class': 'row'})
# used to store data extracted from HTML <div class="row"> elements
data = {}
for row in rows:
title = row.find('div', {'class': 'title'})
content = row.find('div', {'class': 'content'})
if title and content:
# here I am just formatting the dict key to be more python-ish. totally optional
title = title.text.strip().lower().replace(' ', '-')
data[title] = content
# tested with the HTML from the first page
print data.keys()
#=> [u'cuisine', u'restaurant-features', u'average-prices', u'good-for', u'open-hours', u'meals']
print type(data['cuisine'])
#=> <class 'bs4.element.Tag'>
Now you can extract the content items from the HTML webpage without caring what order they appear in. This code should work on any HTML that has the same general structure as the two pages you provided. I hope this helps!

Categories

Resources