Develop Reference

Sending GET request to URL with fragment returns the content of the main page

I am trying to web-scrap this webpage but I always end up getting the "main" page (same URL but without "#face-a-face" at the end). It's the same problem as this guy encountered, see this forum. He got an answer but I am not able to generalize and apply this for the website I want to scrap.
import requests
from bs4 import BeautifulSoup
url_main = "https://www.lequipe.fr/Football/match-direct/ligue-1/2020-2021/ol-dijon-live/477168"
url_target = url_main + "#face-a-face"
soup_main = BeautifulSoup(requests.get(url_main, verify=False).content, "html.parser")
soup_target = BeautifulSoup(requests.get(url_target, verify=False).content, "html.parser")
print(soup_main == soup_target)
returns True. I would like to get different contents, this is not the case here.
For example, I would like to extract all the "confrontations depuis 2011" in the target webpage. How can I get the final content of this webpage with a GET request (or with another way) ? Thanks !

All the data comes from a highly nested JSON file.
You can get that file and extract the information you need.
Here's how:
import json
import requests
endpoint = "https://iphdata.lequipe.fr/iPhoneDatas/EFR/STD/ALL/V2/Football/Prelive/68/477168.json"
team_data = requests.get(endpoint).json()
specifics = team_data["items"][1]["objet"]["matches"][0]["specifics"]
print(json.dumps(specifics, indent=2))
This should get you a dictionary:
{
"__type": "specifics_sport_collectif",
"vainqueur": "domicile",
"score": {
"__type": "score",
"exterieur": "1",
"domicile": "4"
},
"exterieur": {
"__type": "effectif_sport_collectif",
"equipe": {
"__type": "equipe",
"id": "202",
"url_image": "https://medias.lequipe.fr/logo-football/202/{width}{declinaison}",
"nom": "Dijon",
"url_fiche": "https://www.lequipe.fr/Football/FootballFicheClub202.html"
}
},
"domicile": {
"__type": "effectif_sport_collectif",
"equipe": {
"__type": "equipe",
"id": "22",
"url_image": "https://medias.lequipe.fr/logo-football/22/{width}{declinaison}",
"nom": "Lyon",
"url_fiche": "https://www.lequipe.fr/Football/FootballFicheClub22.html"
}
},
"is_final": false,
"prolongation": false,
"vainqueur_final": "domicile",
"is_qualifier": false
}
And if you, for example, just want the socre, add this line:
just_the_score = specifics["score"]
print(just_the_score)
To get this:
{'__type': 'score', 'exterieur': '1', 'domicile': '4'}

How can I filter API GET Request on multiple variables?

I am really struggling with this one. I'm new to python and I'm trying to extract data from an API.
I have managed to run the script below but I need to amend it to filter on multiple values for one column, lets say England and Scotland. Is there an equivelant to the SQL IN operator e.g. Area_Name IN ('England','Scotland').
from requests import get
from json import dumps
ENDPOINT = "https://api.coronavirus.data.gov.uk/v1/data"
AREA_TYPE = "nation"
AREA_NAME = "england"
filters = [
f"areaType={ AREA_TYPE }",
f"areaName={ AREA_NAME }"
]
structure = {
"date": "date",
"name": "areaName",
"code": "areaCode",
"dailyCases": "newCasesByPublishDate",
}
api_params = {
"filters": str.join(";", filters),
"structure": dumps(structure, separators=(",", ":")),
"latestBy": "cumCasesByPublishDate"
}
formats = [
"json",
"xml",
"csv"
]
for fmt in formats:
api_params["format"] = fmt
response = get(ENDPOINT, params=api_params, timeout=10)
assert response.status_code == 200, f"Failed request for {fmt}: {response.text}"
print(f"{fmt} data:")
print(response.content.decode())

I have tried the script, and dict is the easiest type to handle in this case.
Given your json data output
data = {"length":1,"maxPageLimit":1,"data":[{"date":"2020-09-17","name":"England","code":"E92000001","dailyCases":2788}],"pagination":{"current":"/v1/data?filters=areaType%3Dnation%3BareaName%3Dengland&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22dailyCases%22%3A%22newCasesByPublishDate%22%7D&latestBy=cumCasesByPublishDate&format=json&page=1","next":null,"previous":null,"first":"/v1/data?filters=areaType%3Dnation%3BareaName%3Dengland&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22dailyCases%22%3A%22newCasesByPublishDate%22%7D&latestBy=cumCasesByPublishDate&format=json&page=1","last":"/v1/data?filters=areaType%3Dnation%3BareaName%3Dengland&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22dailyCases%22%3A%22newCasesByPublishDate%22%7D&latestBy=cumCasesByPublishDate&format=json&page=1"}}
You can try something like this:
countries = ['England', 'France', 'Whatever']
return [country for country in data where country['name'] in countries]
I presume the data list is the only interesting key in the data dict since all others do not have any meaningful values.

Why does print(new_data[0]) only print out the first char of the json file?

im trying to print the first object of this json file but it only prints the first char of it.
This is my code:
response = requests.get("http://jsonplaceholder.typicode.com/users")
data = response.json()
new_data = json.dumps(data, indent = 2)
print(str(new_data[0]))
result i was hoping for:
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere#april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
}
actual result:
[

json.dump the first element of the response:
import json
response = requests.get("http://jsonplaceholder.typicode.com/users")
data = response.json()
first_elem = json.dumps(data[0], indent=2)
print(first_elem)

Apparently response.json() already is a dictionary.
So if you try first_element = data[0] you would get what you are looking for.
And then, if you want to make it pretty :
json.dumps(first_element, indent = 2)
If you wish a JSON object to behave like a dictionary , take a look at
json.loads
https://docs.python.org/2/library/json.html
Also :
What's the best way to parse a JSON response from the requests library?

json.dumps results in a string.
You are printing the first word of the string by doing [0]
For the output that you want, do:
print(new_data)

Python requests accessing index of response object

I'm new to python and trying to practice aggregating data from an api call
i have this script
r = requests.get('https://jsonplaceholder.typicode.com/users')
print r.text
which returns an array of objects in this format
[{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere#april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
}]
I've been playing around and tried this to see if I can access the first object
print r.text[0]
And it didn't work. So how do I do this with python

You need to parse the JSON text:
import json
array = json.loads(r)
print array[0]

request.text returns Http response body.
so If you want to get first propery of json,
you should convert string to json object.
This works
result = r.text
print(type) # prints str
import json
result = json.loads(result)
print(result[0]) # (...)

How to add Panda 'to_json' output to a rest call

I have the following JSON data:
x = df.to_json(orient='records')
print(x)
[{"val":"3760","id":"204","quantity":2},{"val":"8221","id":"220","quantity":8}]
I want to add the data to my REST call, but it results in the following string in the payload (note: the single quotes around the square bracket:
'updateVals': '[{"val":"3760","id":"204","quantity":2},{"val":"8221","id":"220","quantity":8}]'}}
The fact that the JASON values are listed as one big string, the REST call results in an HTTP 400 error.
The code is:
url = 'my_url'
payload = {
'client_id': 'my_id',
'api_key': 'my_key',
"data": {
"uuid": "myUUID",
"timeStamp": "2018-09-12T06:17:48+00:00",
"updateVals": x
}
}
How do I plug the JSON into the REST call? I assume I have to split the string, or maybe there is a more straightforward answer?

Still not sure what you need, JSON is really one big string:
x = [{"val":"3760","id":"204","quantity":2},
{"val":"8221","id":"220","quantity":8}]
>>> json.dumps(x)
>>> '[{"val": "3760", "id": "204", "quantity": 2}, {"val": "8221", "id": "220", "quantity": 8}]'