Develop Reference

Extracting full URL from href tag in scrapy

I'm trying to use scrapy to scrape URLs from offers from this site
This is the code I tried:
url = response.css('a[data-tracking="click_body"]::attr(href)').extract()
But my code returns something very different from a URL.
Here is the HTML code of the div I'm interested in.
<div class="offer-item-details">
<header class="offer-item-header">
<h3>
<a href="https://www.otodom.pl/oferta/gdansk-pod-inwestycje-cicha-lokalizacja-ID46DXu.html#ab04badaa0" data-tracking="click_body" data-tracking-data="{"touch_point_button":"title"}" data-featured-name="promo_top_ads">
<strong class="visible-xs-block">42 m²</strong>
<span class="text-nowrap">
<span class="offer-item-title">Gdańsk/ Pod Inwestycje/ Cicha Lokalizacja</span>
</span>
</a>
</h3>
<p class="text-nowrap"><span class="hidden-xs">Mieszkanie na sprzedaż: </span>Gdańsk, Ujeścisko-Łostowice, Łostowice</p>
<div class="vas-list-no-offer">
<a class="button-observed observe-link favourites-button observed-text svg-heart add-to-favourites" data-statkey="ad.observed.list" rel="nofollow" data-id="60688916" href="#" title="Obserwuj">
<div class="observed-text-container" style="display: flex;">
<span class="icon observed-60688916"></span>
<i class="icon-heart-filled"></i>
<div class="observed-label">Dodaj do ulubionych</div>
</div>
</a>
</div>
</header>
<ul class="params
" data-tracking="click_body" data-tracking-data="{"touch_point_button":"body"}">
<li class="offer-item-rooms hidden-xs">2 pokoje</li>
<li class="offer-item-price">
346 000 zł </li>
<li class="hidden-xs offer-item-area">42 m²</li>
<li class="hidden-xs offer-item-price-per-m">8 238 zł/m²</li>
</ul>
</div>
Copied selector of that tag:
#offer-item-ad_id45Wog > div.offer-item-details > header > h3 > a
Copied xPath
//*[#id="offer-item-ad_id45Wog"]/div[1]/header/h3/a
Copied full xPath
/html/body/div[3]/main/section[2]/div/div/div[1]/div/article[1]/div[1]/header/h3/a

Your code gives you a list of the URLs. The extract() method in this case gets a list. To allow scrapy to extract the data you will have to do a for loop and yield statement.
url = response.css('a[data-tracking="click_body"]::attr(href)').extract()
for a in url:
yield{'url', a}

Python Selenium click on input tag by data-filter-tag value

I am trying to make selenium click on input checkbox but facing different errors.
only difference between these checkbox's is data-filter-tag value so I tried this.
driver.find_element_by_css_selector("input[data-filter-tag='am']").click()
ERROR
ElementNotVisibleException: Message: element not interactable
web page is loaded 100%
HTML CODE :
<ul>
<li class="filter-checkbox active-filter">
<div class="custom-checkbox" style="background-position: 0px center;"><input name="ctl00$ctl00$MainContentPlaceHolder$BaseContentPlaceHolder$pmainedge2edge4_0$ctl00$ctl14$rpFilters$ctl01$ctl00" type="checkbox" class="dealer-locator-filter" data-filter-tag="c"></div>
<span data-toggle="popover" data-placement="top" data-trigger="hover" data-content="" data-original-title="" title="">
<i class="c-icon hq-icon hq-icon-crown-dealer hq-icon-small " title=""></i>Crown dealers
</span>
</li>
<li class="filter-checkbox active-filter">
<div class="custom-checkbox" style="background-position: 0px center;"><input name="ctl00$ctl00$MainContentPlaceHolder$BaseContentPlaceHolder$pmainedge2edge4_0$ctl00$ctl14$rpFilters$ctl02$ctl00" type="checkbox" class="dealer-locator-filter" data-filter-tag="r"></div>
<span data-toggle="popover" data-placement="top" data-trigger="hover" data-content="" data-original-title="" title="">
<i class="c-icon hq-icon hq-icon-reseller hq-icon-small " title=""></i>Authorized dealers
</span>
</li>
<li class="filter-checkbox active-filter">
<div class="custom-checkbox" style="background-position: 0px center;"><input name="ctl00$ctl00$MainContentPlaceHolder$BaseContentPlaceHolder$pmainedge2edge4_0$ctl00$ctl14$rpFilters$ctl03$ctl00" type="checkbox" class="dealer-locator-filter" data-filter-tag="cs"></div>
<span data-toggle="popover" data-placement="top" data-trigger="hover" data-content="" data-original-title="" title="">
<i class="c-icon hq-icon hq-icon-servicing-workshop hq-icon-small " title=""></i>Has service
</span>
</li>
<li class="filter-checkbox active-filter">
<div class="custom-checkbox" style="background-position: 0px center;"><input name="ctl00$ctl00$MainContentPlaceHolder$BaseContentPlaceHolder$pmainedge2edge4_0$ctl00$ctl14$rpFilters$ctl04$ctl00" type="checkbox" class="dealer-locator-filter" data-filter-tag="am"></div>
<span data-toggle="popover" data-placement="top" data-trigger="hover" data-content="" data-original-title="" title="">
<i class="c-icon hq-icon hq-icon-category-robotic-lawn-mowers hq-icon-small " title=""></i>Automower® dealers
</span>
</li>
</ul>
</div>
Site : link
Complete code :
from selenium import webdriver
from time import sleep
driver = webdriver.Chrome('./chromedriver/chromedriver')
### website url
driver.get('https://www.husqvarna.com/us/')
sleep(6)
driver.find_element_by_css_selector("input[data-filter-tag='am']").click()

If you select the surrounding div of a checkbox with this rather nasty CSS selector, you can at least click a checkbox without an exception.
checkbox = driver.find_element_by_css_selector("#MainContentPlaceHolder_BaseContentPlaceHolder_pmainedge2edge4_0_ctl00_ctl14_dealerFilters > section:nth-child(1) > div:nth-child(1) > div:nth-child(1) > ul:nth-child(1) > li:nth-child(4) > div:nth-child(1)")
checkbox.click()
There is a lot of JavaScript interfering with webdriver automation. I did not find a better solution yet, but at least you know there is a way to interact with that checkbox.

Can selenium find_element_by_xpath in outlook online?

I'm having trouble finding elements by xpath or id inside of outlook's online html. I'm using selenium and python. Here is what I've wrote.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException #imports
browser = webdriver.Chrome('C:\\Users\\...')
browser.get('https://www.office.com/...') #setup
#then some code to sign into outlook online
browser.find_element_by_xpath('//*[#id="_ariaId_64"]/div/div/div[1]')
browser.find_element_by_xpath('//*[#id="MailFolderPane.FavoritesFolders"]/div[19]')
browser.find_element_by_id('_ariaId_68') #attempts to find an element in outlook
this is the element I'm trying to access. I would post the entire html, but there is just so much. I'm brand new to all things code so go easy on me :)
<div>
<div id="_ariaId_68" aria-expanded="false" draggable="true" dropzone="string:text/plain">
<div autoid="_n_R" class="_n_44 canShowFavoritesAction" role="treeitem" aria-expanded="false" aria-labelledby="_ariaId_68.folder _ariaId_68.ucount" aria-haspopup="true" tabindex="-1">
<div class="_n_S3 nowrap border-color-transparent _n_X3" style="padding-left: 4px;">
<div class="_n_V3 _n_54">
<span autoid="_n_S" class="_n_W3 ms-font-m ms-fwt-sl _n_Y3" id="_ariaId_68.folder" title="Sent Items">Sent Items</span>
<div class="_n_24 ms-bg-color-neutralLighter"> <span autoid="_n_T" class="ms-font-m _n_Z3 ms-fwt-sb ms-fcl-tp" aria-hidden="true"></span> </div>
<span class="ms-font-s ms-fcl-ns" aria-hidden="true" aria-expanded="false" aria-haspopup="true" tabindex="-1"> </span> <button autoid="_n_U" type="button" class="_n_34 ms-fwt-r ms-fcl-ns o365button hidden" style="display: none;" tabindex="-1"></button> <span style="display: none;" aria-hidden="true"></span>
</div>
<div class="_n_14 hidden"><button autoid="_n_V" type="button" class="_n_04 firefoxFavorite o365button" title="Remove from Favorites" aria-labelledby="_ariaId_69"><span class="_fc_3 owaimg ms-Icon--star ms-icon-font-size-18 ms-fcl-ns-b"> </span><span class="_fc_4 o365buttonLabel _fc_2" id="_ariaId_69" style="display: none;"></span></button></div>
</div>
</div>
</div>
</div>

Assuming you're trying to find the SENT ITEMS's - web element.
The SENT ITEM link does not seem to be the visible area [when you have more sub folders inside Inbox, this usually happens] and hence you first make a scroll to view to the element before performing anything on the element.
Here are the options to bring the element to visible area
Try with following xpath
//span[#title='Sent Items']

Using Selenium Webdriver, grabbing data not showing up in innerhtml

I am trying to use selenium to grab text data from a page.
Printing the html attributes:
element = driver.find_element_by_id("divresults")
Results:
print(element.get_attribute('innerHTML'))
<div id="divDesktopResults"> </div>
Results:
print(element.get_attribute('outerHTML'))
<div id="divresults" data-bind="html:resultsContent"><div id="divDesktopResults"> </div></div>
Tried grabbing this element
Results:
driver.find_element_by_css_selector("span[class='glyphicon glyphicon-tasks']")
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span[class='glyphicon glyphicon-tasks']"}
This is the code when copied from the Browser. There is much more below 'divresults' that did not show up in the innerhtml printout
<div id="divresults" data-bind="html:resultsContent">
<div>
<div class="row" style="font-size:8pt;">
<a data-toggle="tooltip" style="text-decoration:underline" href="#pdfviewer?ID=D218101736">
<strong>D218101736 </strong>
<span class="glyphicon glyphicon-new-window"></span>
</a>
<div class="btn-group" style="font-size:8pt;margin-left:10px;" id="btnD218101736">
<span style="display:none;font-size:8pt;" id="lblD218101736"> Added To Cart</span>
<button type="button" style="font-size:8pt;" class="btn btn-primary dropdown-toggle" data-toggle="dropdown"> Add To Cart
<span class="caret"></span>
</button>
<ul class="dropdown-menu" role="menu">
<li> <strong>Regular ($7.00)</strong> </li>
<li> <strong>Certified ($12.00)</strong> </li>
</ul>
</div>
</div> <br>
<ul class="nav nav-tabs compact">
<li class="active">
<a data-toggle="tab" href="#D218101736_Doc">
<span class="glyphicon glyphicon-file"></span>
<span>Doc Info</span>
</a>
</li>
<li class="hidden-xs">
<a data-toggle="tab" href="#D218101736_Thumbnail">
<span class="glyphicon glyphicon-th-large"></span>
<span>Thumbnail</span>
</a>
</li>
....
How to I get data beneath divresults in the instance?

My guess is that it's one of two things:
There is more than one element that matches that locator. To investigate this, try using $$("#divresults") in the dev console and make sure that it returns 1. If it returns more than one, run $$("#divresults")[0] and make sure the element returned is the one you want. If it is, go on to step 2. If it isn't, you will need to find a locator that is more specific. If you want our help, you will need to provide a link to the page or more of the surrounding HTML to the desired element.
You need to add a wait so that the contents of the element can finish loading. You could wait for a locator like #divresults strong or any number of locators to find some of the elements that were missing. You would wait for them to be visible (or at least present). See the docs for more info and options.

Upload a photo in selenium python when file path input is not visible

This is a html code of upload a photo:
<div id="choose-photo" class="controls avatar-settings inline-upload-avatar dropdown center">
<div class="uploader-image uploader-avatar clearfix">
<div class="dropdown-menu">
<div class="dropdown-caret">
<span class="caret-outer"></span>
<span class="caret-inner"></span>
</div>
<ul tabindex="-1" role="menu" aria-hidden="true">
<li id="photo-choose-existing" class="photo-choose-existing upload-photo" role="presentation">
<button type="button" class="dropdown-link" role="menuitem">Prześlij zdjęcie</button>
<div class="photo-selector">
<button class="btn" type="button">
Zmień zdjęcie
</button>
<span class="photo-file-name">Nie wybrano pliku</span>
<div class="image-selector">
<input type="hidden" name="media_file_name" class="file-name">
<input type="hidden" name="media_data_empty" class="file-data">
<label class="t1-label">
<span class="u-hiddenVisually">Dodaj zdjęcie</span>
<input type="file" name="media_empty" class="file-input js-tooltip" tabindex="-1" accept="image/gif,image/jpeg,image/jpg,image/png" data-original-title="Dodaj zdjęcie">
</label>
</div>
</div>
</li>
<li id="photo-choose-webcam" class="u-hidden" role="presentation">
<button type="button" class="dropdown-link">Zrób zdjęcie</button>
</li>
<li id="photo-delete-image" class="u-hidden" role="presentation">
<button type="button" class="dropdown-link" role="menuitem">Usuń</button>
</li>
<li class="dropdown-divider" role="presentation"></li>
<li class="cancel-options" role="presentation">
<button type="button" class="dropdown-link" role="menuitem">Anuluj</button>
</li>
</ul>
</div>
</div>
</div>
I've created a simple method to send text to input (it's not visible on screen):
fileInput = driver.find_element_by_name('media_empty')
fileInput.send_keys(path)
But it doesn't do anything. Also I'm getting not any errors.
So, here's a second method, which may work:
<div class="ProfileAvatarEditing-buttonContainer">
<button class="ProfileAvatarEditing-button u-boxShadowInsetUserColorHover" type="button" tabindex="2">
<div class="ProfileAvatarEditing-addAvatarHelp">
<span class="Icon Icon--cameraPlus"></span>
<p>Dodaj zdjęcie profilowe</p>
</div>
<div class="ProfileAvatarEditing-changeAvatarHelp">
<span class="Icon Icon--camera"></span>
<p>Zmień zdjęcie profilowe</p>
</div>
<div class="ProfileAvatarEditing-dropAvatarHelp">
<span class="Icon Icon--cameraPlus"></span>
<p>Upuść zdięcie profilowe tutaj</p>
</div>
</button>
Here user can drap and drop file. I've found this question: Selenium: Drag and Drop from file system to webdriver? however I still don't know how can I use it in this situation.
So the question is how to send file path to the input to trigger file upload. In this case when you choose a file from file dialog or drag and drop it you'll see confirm window with preview on your photo. So then all what's left to do is to click confirm. But I don't know how to send it in the first place.
Any help will be appreciated.
edit:
I've found a solution (my own answer below):
fileInput = driver.find_element_by_xpath('//*[#id="photo-choose-existing"]/div/div/label/input')
fileInput.send_keys(path)
but there's one more problem: photo is uploaded but file dialog still opens - I don't know how to close it. I tried accesing it:
dialog = driver.switch_to.active_element['value']
but I don't know how to close it.

Strangely enough I found send_keys do indeed work. When I inspected html code in different browser it wasn't "media_empty" anymore, but a different name ("media[]" or something similar). Instead I've used xpath and I was stunned that it actually worked:
fileInput = driver.find_element_by_xpath('//*[#id="photo-choose-existing"]/div/div/label/input')
fileInput.send_keys(path)

try using below code:
fileInput = driver.find_element_by_css_selector("div.image-selector label.t1-label input")
driver.execute_script("arguments[0].setAttribute('value', 'YOUR_PATH_HERE')",fileInput)

Assuming that element is present on page if not explicitly wait for element to exist on page.
then try this:
driver.execute_script("document.getElementById('ID_HERE').setAttribute('value', 'PATH_HERE')");
hope this will help you!