Develop Reference

Python - Download Excel file from clickable "XLS" button of website without using Selenium

I am trying to write a program that goes to the following site and downloads the Excel file that automatically downloads when clicking the XLS button on the bottom of the page. To be honest, I am quite new to programming.
Site: https://echa.europa.eu/assessment-regulatory-needs
At first I tried using Selenium to let the program click itself through the browser and really click on the button. However, I think the website detects the usage of automated software and I cannot bypass the disclaimer that appears when opening the website.
Then I read some answers on the same topic where it is possible by using the requests module. However, I could not get it to work.
One thing I think I understood from this anwer was that you need to get the site/server where the data is requested from by inspecting the button with F12 in the browser. I tried this, and thought I had it, however I cannot get the code to function. I learnt from this answer that you need to give the referer as well, bu I think the referer from this file ist only partially written out, as it is "https://echa.europa.eu/assessment-regulatory-needs".
This answer explained the network process more in detail, howver I am not able to recreate it. Also, to be honest i do not fully understand what the API is and how to search for it.
I also found this answer, but it does not work for me either.
So I am asking for help on this, as I think that my HTML, Java, Website, Python knowledge is too tiny to see what I have to change to be able to download the excel file.

Python Selenium with Salesforce - Cannot Seem to Access Certain Form Elements

Using Selenium to try and automate a bit of data entry with Salesforce. I have gotten my script to load a webpage, allow me to login, and click an "edit" button.
My next step is to enter data into a field. However, I keep getting an error about the field not being found. I've tried to identify it by XPATH, NAME, and ID and continue to get the error. For reference, my script works with a simple webpage like Google. I have a feeling that clicking the edit button in Salesforce opens either another window or frame (sorry if I'm using the wrong terminology). Things I've tried:
Looking for other frames (can't seem to find any in the HTML)
Having my script wait until the element is present (doesn't seem to work)
Any other options? Thank you!

Salesforce's Lighting Experience (the new white-blue UI) is built with web components that hide their internal implementation details. You'd need to read up a bit about "shadow DOM", it's not a "happy soup" of html and JS all chucked into top page's html. Means that CSS is limited to that one component, there's no risk of spilling over or overwriting another page area's JS function if you both declare function with same name - but it also means it's much harder to get into element's internals.
You'll have to read up about how Selenium deals with Shadow DOM. Some companies claim they have working Lightning UI automated tests/ Heard good stuff about Provar, haven't used it myself.
For custom UI components SF developer has option to use "light dom", for standard UI you'll struggle a bit. If you're looking for some automation without fighting with Lighting Experience (especially that with 3 releases/year SF sometimes changes the structure of generated html, breaking old tests) - you could consider switching over to classic UI for the test? It'll be more accessible for Selenium, won't be exactly same thing the user does - but server-side errors like required fields, validation rules should fire all the same.

Extracting info from webpage via python

I'd like to ask somebody with experience with headless browsers and python if it's possible to extract box info with distance from closest strike on webpage below. Till now I was using python bs4 but since everything is driven by jQuery here simple download of webpage doesn't work. I found PhantomJS but I wasn't able extract it too so I am not sure if it's possible. Thanks for hints.
https://lxapp.weatherbug.net/v2/lxapp_impl.html?lat=49.13688&lon=16.56522&v=1.2.0

This isn't really a Linux question, it's a StackOverflow question, so I won't go into too much detail.
The thing you want to do can be easily done with Selenium. Selenium has both a headless mode, and a heady mode (where you can watch it open your browser and click on things). The DOM query API is a bit less extensive than bs4, but it does have nice visual query (location on screen) functions. So you would write a Python script that initializes Selenium, goes to your website and interacts with it. You may need to do some image recognition on screenshots at some point. It may be as simple as finding for a certain query image on the screen, or something much more complicated.
You'd have to go through the Selenium tutorials first to see how it works, which would take you 1-2 days. Then figure out what Selenium stuff you can use to do what you want, that depends on luck and whether what you want happens to be easy or hard for that particular website.
Instead of using Selenium, though, I recommend trying to reverse engineer the API. For example, the page you linked to hits https://cmn-lx.pulse.weatherbug.net/data/lightning/v1/spark with parameters like:
_
callback
isGpsLocation
location
locationtype
safetyMessage
shortMessage
units
verbose
authid
timestamp
hash
You can figure out by trial and error which ones you need and what to put in them. You can capture requests from your browser and then read them yourself. Then construct appropriate requests from a Python program and hit their API. It would save you from having to deal with a Web UI designed for humans.

Program for Google Sites in Python

I must get the URLs of all subpages found within one Google Site in the editor mode. I have a subpage for each Form(1 to 6 Upper) of all classes at school. However, I intend to automate any future changes using Python code. I must be able to access each page and upload photos to the subpages under each one. But, for that, I must get onto the subpage itself.
Basically, the web structure goes like this:
EVERYTHING -> CLASSES -> SUBJECTS
I have tried using Selenium for automation but that idea didn't work out since I cannot log in with Google once it enters automation mode since Selenium is active. I have tried using a program to simulate mouse motion and actually click on the subpages but it is far too complex and after several unsuccessful attempts, I gave up.
I need ideas on what I should do to access each subpage and retrieve its URL. I would appreciate if someone could help me because I am really stuck as I cannot hope to update the entire site manually on a weekly basis.
If someone could show me the code which would perform this task, I would appreciate it too much to express in words. No matter what, thanks very much!

I am trying to scrape this website for all of the documents that are produced from the drop down forms

The site I am trying to scrap has drop-down menus that end up producing a link to a document. The end documents are what I want. I have no experience with web scraping so I don't know where to start on this. I don't know where to start. I have tried adapting this to my needs, but I couldn't get it working. I also tried to adapt this.
I know basically I need to:
for state in states:
select state
for type in types:
select type
select wage_area_radio button
for area in wage_area:
select area
for locality in localities:
select locality
for date in dates:
select date
get_document
I just haven't found anything that works for me yet. Is there a tool better than Selenium for this? I am currently trying to bend it to my will using the the code from my second example as a starter.

Depending on your coding skills and knowledge of HTTP, I would try one of two things. Note that scraping this site appears slightly non-trivial because of the different form options that appear based on what was previously selected, and the fact that there's a lot of AJAX calls happening.
1) Follow the HTTP requests (especially the AJAX ones) that are being made in something like Chrome DevTools. You'll get a good understanding of how the final URL is being formed and how to construct it yourself. In particular, it looks like the last POST to AFWageScheduleYearSelected is the one that generates the final url. Then, you can make these calls yourself in a Python HTTP library to get the documents.
2) Use something like PhantomJS (http://phantomjs.org/) which is a headless browser. I don't have experience scraping with Selenium, but my understanding is that it is more of a testing/automation tool. In any case, PhantomJS is pretty easy to get up and running and you can basically click page elements, fill out forms, etc.
If you do end up using PhantomJS (or any other browser-like tool), you'll run into issues with the AJAX calls that populate the forms. Basically, you'll end up trying to fill out forms that don't yet exist on the page because the data is still being sent over the network. The easiest way to get around this is to just set timeouts (of say 2 seconds) in between each form field that you fill out. The alternative to using timeouts (which may be unreliable and slow) is to continuously poll the page until the AJAX call is finished.