Develop Reference

How can I filter API GET Request on multiple variables?

I am really struggling with this one. I'm new to python and I'm trying to extract data from an API.
I have managed to run the script below but I need to amend it to filter on multiple values for one column, lets say England and Scotland. Is there an equivelant to the SQL IN operator e.g. Area_Name IN ('England','Scotland').
from requests import get
from json import dumps
ENDPOINT = "https://api.coronavirus.data.gov.uk/v1/data"
AREA_TYPE = "nation"
AREA_NAME = "england"
filters = [
f"areaType={ AREA_TYPE }",
f"areaName={ AREA_NAME }"
]
structure = {
"date": "date",
"name": "areaName",
"code": "areaCode",
"dailyCases": "newCasesByPublishDate",
}
api_params = {
"filters": str.join(";", filters),
"structure": dumps(structure, separators=(",", ":")),
"latestBy": "cumCasesByPublishDate"
}
formats = [
"json",
"xml",
"csv"
]
for fmt in formats:
api_params["format"] = fmt
response = get(ENDPOINT, params=api_params, timeout=10)
assert response.status_code == 200, f"Failed request for {fmt}: {response.text}"
print(f"{fmt} data:")
print(response.content.decode())

I have tried the script, and dict is the easiest type to handle in this case.
Given your json data output
data = {"length":1,"maxPageLimit":1,"data":[{"date":"2020-09-17","name":"England","code":"E92000001","dailyCases":2788}],"pagination":{"current":"/v1/data?filters=areaType%3Dnation%3BareaName%3Dengland&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22dailyCases%22%3A%22newCasesByPublishDate%22%7D&latestBy=cumCasesByPublishDate&format=json&page=1","next":null,"previous":null,"first":"/v1/data?filters=areaType%3Dnation%3BareaName%3Dengland&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22dailyCases%22%3A%22newCasesByPublishDate%22%7D&latestBy=cumCasesByPublishDate&format=json&page=1","last":"/v1/data?filters=areaType%3Dnation%3BareaName%3Dengland&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22dailyCases%22%3A%22newCasesByPublishDate%22%7D&latestBy=cumCasesByPublishDate&format=json&page=1"}}
You can try something like this:
countries = ['England', 'France', 'Whatever']
return [country for country in data where country['name'] in countries]
I presume the data list is the only interesting key in the data dict since all others do not have any meaningful values.

Python's request() returning response 400 when trying to get file from API

This is my first time messing with APIs. I wrote some code to get data from USASpending's API using a 'GET' call and it works. I'm now trying to get data from another endpoint that only has 'post' capability. The code below returns the '400' response. I'm not sure what I need to change to get the output to be produced. I don't know if it matters but this endpoint produces a zip file.
import requests
payload = {"award_levels":["prime_awards"],"filters":{"award_types":["contracts","direct_payments","grants","idvs","loans","other_financial_assistance"],"agency":"United States Mint","date_type":"action_date","date_range":{"start_date":"2019-01-01","end_date":"2019-01-31"}},"columns":[],"file_format":"csv"}
response = requests.post('https://api.usaspending.gov/api/v2/bulk_download/awards/', params = payload).json()
Thanks in advance
Update:
This is what I finally used. I have the date as dynamic so I can automate this pull monthly using task scheduler. If you want to pull a specific date, remove the date variable reference in the header declaration and run.
pl_str1 = """{
"filters": {
"prime_award_types": [
"A",
"B",
"C",
"D",
"IDV_A",
"IDV_B",
"IDV_B_A",
"IDV_B_B",
"IDV_B_C",
"IDV_C",
"IDV_D",
"IDV_E",
"02",
"03",
"04",
"05",
"10",
"06",
"07",
"08",
"09",
"11"],
"agency": 54,
"date_type": "action_date","""
# Inserting date_range variable into API call
pl_str2 = '"date_range": {"start_date":' + date_3Months_prior +', "end_date":' + date_today + '}'
pl_str3 = """},
"columns": [],
"file_format": "csv"}"""
desired_payload = pl_str1 + pl_str2 + pl_str3
# $$$$$$$$$$$$$$$$$$$$$$$$$$$ REQUESTING DESIRED INFO FROM THE API HERE $$$$$$
url = 'https://api.usaspending.gov/api/v2/bulk_download/awards/'
headers = {'Content-Type': 'application/json'}
resp = requests.post(url, headers=headers, data=desired_payload)
if resp.status_code == 200:
print('success')
print(resp.content)
else:
print('fail')
# API returns a zip file; grabbing that
# This will turn the API response into a string I can use regex on
test = resp.content.decode('UTF-8')
# Extracting url for zip we want to retrieve
test2 = re.findall(r"file_url\S+\.zip", test)[0]
test3 = re.findall(r"https\S+\.zip", test2)[0]
import zipfile
import io
import time
r = requests.get(test3)
# add sleep in case it takes a while for the API to return stuff; not sure if
# but just in case
time.sleep(5)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall()

params are the query parameters in the URL.
e.g
www.stackoverflow.com?parameter=test
This can be expressed as
payload = {
"parameter": "test"
}
POST request has a body, This is the same data that would've been sent if you were posting from a form, You're sending a body, You have some parameters in requests module function .post that deal with the body, You can either use data=payload or json=payload, See the official documentation on this.

When you run this url (https://api.usaspending.gov/api/v2/bulk_download/awards/) in browser and fill form with your request. You will get respons like this
{
"detail": "Missing one or more required body parameters: prime_award_types or sub_award_types"
}
Add fliters, data_range, data_type , agency,prime_award_types
Your request should be like this:
{
"filters": {
"date_range": {
"start_date": "2019-01-01",
"end_date": "2019-12-31"
},
"date_type": "action_date",
"agency": 50,
"prime_award_types": [
"02",
"03",
"04",
"05",
"A",
"B",
"C",
"D"
],
"award_levels": [
"prime_awards"
],
"filters": {
"award_types": [
"contracts",
"direct_payments",
"grants",
"idvs",
"loans",
"other_financial_assistance"
],
"agency": "United States Mint",
"date_type": "action_date",
"date_range": {
"start_date": "2019-01-01",
"end_date": "2019-01-31"
}
},
"columns": [],
"file_format": "csv"
}
}

Get form data in function and send it an other function with request.
import requests
from flask import Flask, return , request,
#videos.route("/ajaxprocess", methods=["GET","POST"])
def ajaxprocess():
if request.method =="POST":
text = request.form.get('text_text')
font = request.form.get('font_text')
color = request.form.get('color_text')
start_text = request.form.get('start_text')
end_text = request.form.get('end_text')
video_id = request.form.get('vid')
all_data = {'status':'OK','text':text,'font':font,'color':color,'start_text':start_text,'end_text':end_text,'vid':video_id}
r = requests.post("http://127.0.0.1:5000/videos/test_text",json=all_data)
print(r)
return redirect(url_for("videos.show_videos"))
#videos.route("/test_text", methods = ['POST'])
def test_text():
data = request.get_json("")
print(data)
return "success!"

Getting a specific value of JSON data

I'm getting a JSON data from RESTCONF HTTPS request, using the following code
https_request = 'https://' + host + '/restconf/data/' + operation
headers = {'Content-type': 'application/yang-data+json', 'Accept': 'application/yang-data+json'}
r = requests.get(https_request, auth=(user, password), headers=headers, verify=False)
print r.json()
The data I got is the following:
{
"Cisco-IOS-XE-segment-routing:ipv4": {
"prefixes": [
{
"ipprefix": "1.1.1.1/32",
"index": {
"range-start": 333,
"range": 1
}
}
]
}
}
Basically, I want to return the field's "range-start" value which is 333. I tried the following but it did not work.
for element in r:
id = element['range-start']
print(id)
Is there anyway to get that value?

From Python Console:
>>> import json
... data = json.loads('{"Cisco-IOS-XE-segment-routing:ipv4": {"prefixes": [{"ipprefix": "1.1.1.1/32", "index": {"range-start": 333, "range": 1}}]}}')
... print(data['Cisco-IOS-XE-segment-routing:ipv4']['prefixes'][0]['index']['range-start'])
333
>>>

You need to start at the beginning of the JSON and work your way to the key you want. To do this you need to start at Cisco-IOS-XE-segment-routing:ipv4.
prefixes = r.json()["Cisco-IOS-XE-segment-routing:ipv4"]["prefixes"]
id = prefixes[0]["index"]["range-start"]
If there are multiple prefixes you can loop over them and access each range-start.

Since you are looping over elements, I would suggest this approach using a helper function:
def get_id(element):
prefixes = r.json()["Cisco-IOS-XE-segment-routing:ipv4"]["prefixes"]
id = prefixes[0]["index"]["range-start"]
return id
Then you can do, as in your question:
for element in r:
id = get_id(element)
print(id)

Retrieving data from JSON in python, and if object name matches, then store the key of that object

I am making REST calls on a server. The first REST call gets all the projects and from that I store the project's IDs in an array.
Below is the JSON.
For e.g. it would return something like this:
[
{
"expand": "description,lead,url,projectKeys",
"self": "http://localhost:8080/rest/api/2/project/10101",
"id": "10101",
"key": "GR1",
"name": "Group1Project",
"avatarUrls": {
"48x48": "http://localhost:8080/secure/projectavatar?avatarId=10324",
"24x24": "http://localhost:8080/secure/projectavatar?size=small&avatarId=10324",
"16x16": "http://localhost:8080/secure/projectavatar?size=xsmall&avatarId=10324",
"32x32": "http://localhost:8080/secure/projectavatar?size=medium&avatarId=10324"
},
"projectTypeKey": "software"
}
]
Then I'm looping through that array and making another REST call for each project id(10101).
This gives me groups/users against that project.
For example:
{
"self": "http://localhost:8080/rest/api/2/project/10000/role/10100",
"name": "Developers",
"id": 10100,
"actors": [
{
"id": 10207,
"displayName": "group2",
"type": "atlassian-group-role-actor",
"name": "group2",
"avatarUrl": "http://localhost:8080/secure/useravatar?size=xsmall&avatarId=10123"
}
]
}
I want to get all the project IDs where name == group2.
Following is my Python code for all of this but it's not working.
import requests
ids = []
response = requests.get('http://localhost:8080/rest/api/2/project',
auth=('*', '*'))
data = response.json()
for line in data:
ids.append(line["id"])
print(ids)
# Check if group exists in Project roles.
# If it does, then save the project name in the list of arrays.
projectNames = []
for id in ids:
url = 'http://localhost:8080/rest/api/2/project/'+id+'/role/10100'
response = requests.get(url,
auth = ('*', '*'))
data = response.json()
if data.displayName == 'group2':
projectNames.append(["id"])
Could you please help me out how to do this?
Thank you.

Tayyab,
You need to do this. It will work.
for actor in data['actors']:
if actor['displayName']=='group2':
projectNames.append(id)

projectNames = []
for id in ids:
url = 'http://localhost:8080/rest/api/2/project/'+id+'/role/10100'
response = requests.get(url,
auth = ('*', '*'))
data = response.json()
for actor in data["actors"]:
if actor["displayName"] and actor["displayName"] == "group2":
projectNames.append(actor["id"])

projectNames = set()
for id in ids:
url = 'http://localhost:8080/rest/api/2/project/'+id+'/role/10100'
response = requests.get(url,
auth = ('*', '*'))
data = response.json()
group2_actors = [actor['id'] for actor in data['actors']
if actor['displayName'] == 'group2']
if len(group2_actors) > 0:
projectNames.update(group2_actors)
projectNames is a set of unique actor ids with displayName == group2.

some_json = {}
result = [actor['id'] for actor in some_json['actors'] if actor['name']=='group2']
so result for that second json will be [10207]

How to use appendCells to append a list of columns in Google Sheets API

I'm using python 2.7 but my understanding is this syntax works with any language.
I'd like to append data to the bottom of my Google Sheets. The function below works, but I'd like to know if there is a better way to:
specify userEnteredValue on the column level instead of cell level
not have to repeat the values list for each column.
An example with my current method:
def insertOneRecord(sheetId, value):
data = {
"requests": [
{
"appendCells": {
"sheetId": sheetId,
"rows": [ {"values": [
{"userEnteredValue": {"stringValue": value[0]}},
{"userEnteredValue": {"stringValue": value[1]}}
]}],
"fields" : "userEnteredValue"
}
}
]
}
res = SHEETS.spreadsheets().batchUpdate(spreadsheetId = SPREADSHEET_ID,body = data).execute()
values = ['O','P']
insertOneRecord(sheetId, values)

This is an extraction of a code I'm working on. Maybe it could be helpful:
appendResult = spreadsheet.append(
rangeName='%s!A3:D3' % title,
values=values,
valueInputOption="USER_ENTERED",
insertDataOption="INSERT_ROWS",
)
where title is the sheet Name, values is a tuple of tuples and spreadsheet is an instance of a custom class and the method append is this:
def append(self, rangeName, values, valueInputOption='RAW', insertDataOption='INSERT_ROWS'):
self._log.debug("Inserting values starting from %s/%s" % (self.spreadsheetId, rangeName))
return self._service.spreadsheets().values().append(
spreadsheetId=self.spreadsheetId,
range=rangeName,
valueInputOption=valueInputOption,
insertDataOption=insertDataOption,
body={
'values': values
}
).execute()
self._service is the same service you can see in Step 3 of Google Sheets API - Python Quick Start, as the result of discovery.build function.

make sure you follow all the OAuth consent.
https://developers.google.com/sheets/api/quickstart/js
The documentation regarding is here.
https://developers.google.com/sheets/api/samples/rowcolumn#append_empty_rows_or_columns
function InsertNewCol() {
var requests = [];
requests.push({
"appendDimension": {
"range": {
"dimension": "COLUMNS",
"length": 3
}
}
});
gapi.client.sheets.spreadsheets.batchUpdate({
spreadsheetId: 'YourSheetID',
requests: requests
}).then(function(response) {
console.log(response);
});}