Develop Reference

How to scrape a Handlebar.js script in Python?

I am trying to scrape the Euronorm and the CO2 from a list of cars from an auction website. I have so far succeeded in navigating to the correct auction webpage and downloading that webpage using Selenium. The information that I need is the {{CO2Emission}} and {{EmissionClass}} for all the cars in the following script:
<script id="lot-template" type="text/x-handlebars-template">
<li data-id="{{Id}}">
<a href="{{LotUrl}}">
{{#if IsFollowing}}<figcaption><i class="fa fa-star"></i></figcaption>{{/if}}
<img src="{{ImagePath}}" alt="{{LocaleName}}" />
</a>
<div class="list-info">
<h3>
<a class="car-title" href="{{LotUrl}}">{{LocaleName}}</a>
</h3>
<ul class="item-specs">
<li>Objectnumber: {{Number}}</li>
{{#if EngineSize}}
<li>CC: {{EngineSize}}</li>{{/if}}
<li>Fuel: {{FuelType}}</li>
{{#if PowerKW}}
<li>KW: {{PowerKW}}</li>{{/if}}
{{#if CO2Emission}}
<li>CO2: {{CO2Emission}} g/km</li>{{/if}}
{{#if EmissionClass}}
<li>Euronorm: {{EmissionClass}}</li>{{/if}}
{{#if FirstInUse}}
<li> First Registration: {{date FirstInUse}}</li>{{/if}}
{{#if Mileage}}
<li>Counter: {{Mileage}} {{MileageType}}</li>{{/if}}
{{#if Location}}
<li>Location {{Location}}</li>{{/if}}
{{#if LicensePlate}}
<li>License plate {{LicensePlate}}</li>{{/if}}
</ul>
</div>
<div class="btnrow">
{{#if HasBid}}
<span class="extra">My bid (Excl VAT): <strong>€ {{BidAmount}}</strong></span>
{{/if}}
{{#if IsOpenForBids}}
<i class="fa fa-gavel"></i>{{#if HasBid }}Change Bid{{else}}Bid now{{/if}}
{{/if}}
<a class="btn" href="{{LotUrl}}"><i class="fa fa-arrow-right"></i> details</a>
</div>
</li>
</script>
Is it possible to get the information out of this script? I am a bit stuck now and I would like to know how to proceed. I am new to web scraping, so I am just trying stuff at the moment.
Thank you!

You would not be able to get the information you need from this handlebars template. The template is combined with data to produce HTML, therefore you have two options for extracting the data you need:
Parse the HTML that is generated with this template
Find the source of the data that gets fed into this template
The source of the data may be an API, or in some form that does not require scraping, so I would try that first, then try parsing HTML.
It would be useful to know which site/page you are trying to scrape.

Django: sidebar with dynamic URLs: how to dynamically create URLs which have dynamic folders in the path

I have a problem with dynamic URLs in sidebar navigation in Django and I hope some of you can help me shed some lights on how to solve it. I have looked for similar questions but I couldn't find an answer for my case.
Basically, what I want to achieve is to have a sidebar with links. This sidebar will be reused on many pages, so it sits in a separate sidebar.py file, which is later imported to the pages.
<h6 class="sidebar-heading d-flex justify-content-between align-items-center px-3 mt-4 mb-1 text-muted">
<span>Content</span>
<a class="d-flex align-items-center text-muted" href="#">
<span data-feather="plus-circle"></span>
</a>
</h6>
<ul class="nav flex-column">
<li class="nav-item">
<a class="nav-link active" href="DYNAMIC LINK HERE">
<span data-feather="home"></span>
Status codes</span>
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="#">
<span data-feather="file"></span>
Depth
</a>
</li>
</ul>
The links I want to display are the following:
urls.py
path('<id>/<crawl_id>/dashboard/', ProjectDashboard, name="crawl_dashboard"),
path('<id>/<crawl_id>/dashboard/status-codes/', StatusCodeDashboard, name="status_code_dashboard"),
path('<id>/<crawl_id>/dashboard/url-depth/', UrlDepthDashboard, name="url_depth_dashboard"),
As you can see, they are dynamic URLs which take an id and crawl_id. So, for each crawl dashboard I want the sidebar to link to its relative status_code_dashboard page and url_depth_dashboard page.
As an example:
/22/123/dashboard --> should have a sidebar with links to:
/22/123/dashboard/status-code/
/22/123/dashboard/url-depth/
What I tried to do is to create a context processor like the following:
def get_dashboard_paths(request):
# Get current path
current_path = request.get_full_path()
depths_dashboard = current_path + 'url-depth/'
return {
'depths_dashboard': depths_dashboard
}
...and then in the sidebar.py template use {{depths_dashboard}}...
This works but it's not scalable: when I am in /22/123/dashboard/status-code/ for example, I still want to have the sidebar to link to the other sections. If I use the above context processor, due to the bad solution wrong links would be created like:
/22/123/dashboard/status-code/status-code/
/22/123/dashboard/status-code/url-depth/
Do you have a hint on how I can display the sidebar on all of the above pages with dynamic URLs based on id and crawl_id? Basically the question is, how can I correctly send those parameters dynamically depending on which id and crawl_id context I am in?
Thanks a lot!

Just pass the id and crawl_id into your template. Then in the template:
Dashboard
Status code
URL depth
If you specifically want to use the preprocessor, you can also get these numbers from get_full_path().split('/').

Adding StreamField call-to-actions to base.html using Wagtail CMS

Hi I'm new to using Wagtail and I'm working on a client website. What I aim to do is to dynamically link my wagtail pages to our sidebar, which is currently in our base.html in the main app folder's templates directory, the one with settings.py.
I was wondering if there's a way to render a call to action for the base.html here. Or if I should make a separate app instead and create a base.html there, which extends to all the other templates I'll use for the rest of the website.
Thank you!
edit:
Above is the current home page I'm working with. The sidebar right now is just hard-coded since I haven't worked on that, and I want to know what the rest of the page looks like while I work on the main content.
the sidebar above is coded as so:
<!-- in biodept/templates/base.html -->
{% wagtailuserbar %}
<div class="container main-container">
<div class="row">
<!-- Nav bar not mobile -->
<nav class="nav" id="nav-1">
<a class="nav-link nav-desktop-link nav-desktop-link-active" href="#">HOME</a>
<a class="nav-link nav-desktop-link" href="#">BIOMEDICINE</a>
<a class="nav-link nav-desktop-link" href="#">ECOLOGY & SYSTEMATICS</a>
<a class="nav-link nav-desktop-link" href="#">MOLECULAR BIO & BIOTECH</a>
<a class="nav-link nav-desktop-link" href="#">PROJECTS</a>
<button class="dropdown-btn nav-link nav-desktop-link">PROGRAMS<ion-icon style="float: right; padding-top: 0.25vw;" name="caret-down-outline"></ion-icon></button>
<div id="btn-t" class="dropdown-container">
<a class="nav-link dropdown-nav-desktop-link" href="#">UNDERGRADUATE</a>
<a class="nav-link dropdown-nav-desktop-link" href="#">GRADUATE</a>
</div>
<a class="nav-link nav-desktop-link" href="faculty.html">FACULTY PAGES</a>
<a class="nav-link nav-desktop-link" href="#">BIODIVERSITY LABORATORY</a>
</nav>
{% block content %}{% endblock %}
</div>
</div>
Again the base.html is in the same directory as where the settings.py is. BioDept is the project's name.

Note: Based on the updated question, it looks like this is unrelated to StreamField but it is a question about how to implement a menu based on the Wagtail page structure.
Wagtail does not come with a built in way to render menus, this is because it is going to be something specific to every Wagtail site and any generic solution will likely only cover a small set of cases. However, when getting started this can be a bit confusing.
Wagtail, does come with a way to indicate that a page should be shown in menus though, this is part of every Page model.
You can see this in the model's reference here
https://docs.wagtail.io/en/latest/reference/pages/model_reference.html#wagtail.core.models.Page.show_in_menus
User's can edit this value on the 'promote' tab, plus the docs above let you define what the default value should be (however, existing pages will need to be updated another way).
Implementing a Menu
Here are three ways to implement a menu and use this as a template tag or template include in your project.
View the Bakerydemo code
The bakerydemo is a nice basic reference for a Wagtail implementation, this may not explain why but might be enough for you to get started.
template tag definition - https://github.com/wagtail/bakerydemo/blob/master/bakerydemo/base/templatetags/navigation_tags.py
header include template - https://github.com/wagtail/bakerydemo/blob/master/bakerydemo/templates/includes/header.html
base.html (layout) template - https://github.com/wagtail/bakerydemo/blob/master/bakerydemo/templates/base.html
Follow a tutorial
Googling 'Wagtail navigation' or 'Wagtail menus' can help, but this link below appears to be up to date and walks you through really nicely on how to build a basic Wagtail menu and then enhance it to using an extension (added below)
https://www.accordbox.com/blog/wagtail-tutorial-12-how-create-and-manage-menus-wagtail-application/
Install an extension package
This package appears to give a robust solution, but avoid using it if you can get what you need without adding another dependency (my opinion)
https://wagtailmenus.readthedocs.io/en/stable/overview.html

Using Selenium Webdriver, grabbing data not showing up in innerhtml

I am trying to use selenium to grab text data from a page.
Printing the html attributes:
element = driver.find_element_by_id("divresults")
Results:
print(element.get_attribute('innerHTML'))
<div id="divDesktopResults"> </div>
Results:
print(element.get_attribute('outerHTML'))
<div id="divresults" data-bind="html:resultsContent"><div id="divDesktopResults"> </div></div>
Tried grabbing this element
Results:
driver.find_element_by_css_selector("span[class='glyphicon glyphicon-tasks']")
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span[class='glyphicon glyphicon-tasks']"}
This is the code when copied from the Browser. There is much more below 'divresults' that did not show up in the innerhtml printout
<div id="divresults" data-bind="html:resultsContent">
<div>
<div class="row" style="font-size:8pt;">
<a data-toggle="tooltip" style="text-decoration:underline" href="#pdfviewer?ID=D218101736">
<strong>D218101736 </strong>
<span class="glyphicon glyphicon-new-window"></span>
</a>
<div class="btn-group" style="font-size:8pt;margin-left:10px;" id="btnD218101736">
<span style="display:none;font-size:8pt;" id="lblD218101736"> Added To Cart</span>
<button type="button" style="font-size:8pt;" class="btn btn-primary dropdown-toggle" data-toggle="dropdown"> Add To Cart
<span class="caret"></span>
</button>
<ul class="dropdown-menu" role="menu">
<li> <strong>Regular ($7.00)</strong> </li>
<li> <strong>Certified ($12.00)</strong> </li>
</ul>
</div>
</div> <br>
<ul class="nav nav-tabs compact">
<li class="active">
<a data-toggle="tab" href="#D218101736_Doc">
<span class="glyphicon glyphicon-file"></span>
<span>Doc Info</span>
</a>
</li>
<li class="hidden-xs">
<a data-toggle="tab" href="#D218101736_Thumbnail">
<span class="glyphicon glyphicon-th-large"></span>
<span>Thumbnail</span>
</a>
</li>
....
How to I get data beneath divresults in the instance?

My guess is that it's one of two things:
There is more than one element that matches that locator. To investigate this, try using $$("#divresults") in the dev console and make sure that it returns 1. If it returns more than one, run $$("#divresults")[0] and make sure the element returned is the one you want. If it is, go on to step 2. If it isn't, you will need to find a locator that is more specific. If you want our help, you will need to provide a link to the page or more of the surrounding HTML to the desired element.
You need to add a wait so that the contents of the element can finish loading. You could wait for a locator like #divresults strong or any number of locators to find some of the elements that were missing. You would wait for them to be visible (or at least present). See the docs for more info and options.

Flask + AngularJS - Using url_for()

I am using AngularJS + Flask in my application, and I want to know the best way to "produce" an url, and don't write any hard code url for this. I have this situation:
*considering that I'm using [[ ]] instead of {{ }} for AngularJS.
<dd ng-repeat="item in myList">
<span ng-click="doAction('{{ url_for('my_url', id="[[item.id]]") }}')">
[[item.name]]
</span>
</dd>
This is not going to work, because Jinja2 do the process url_for() before AngularJS, so "[[item.id]]" will not be substituted by AngularJS in time.
The problem is, I don't want to write in hard code like this:
<span ng-click="doAction('/my_url/[[item.id]]')">
[[item.name]]
</span>
I am pretty new in AngularJS, maybe all my approach is wrong, so, does anyone have any idea what is the best way to make an element be clicked, make a request with an URL based on the context of the clicked element?

I just ran across this problem. I ended up using Jinja2's {% set ... %}.
This is how I solved it, adapted for your example.
<dd ng-repeat="item in myList">
{% set url = url_for('my_url', id="[[item.id]]") %}
<span ng-click="doAction('{{ url }}')">
[[item.name]]
</span>
</dd>

In my case,
I was trying to dynamically create urls
I solved my issue as follows (Note: I've swapped out Angular's syntax to {[x]}:
<ul>
<li ng-repeat="x in projects">
{[x.title]}
{% set url = url_for('static',filename="img/") %}
<img src="{{url}}{[x.img]}">
</li>
</ul>