Develop Reference

get request payload in python

my code is about send get request using query parameters which depends on a page number
After that o have to do for loop to get some ids from the response and also getting the next page number of the same response
and send a new get request with the new next page number that I got from the first response, and I need to get the ids also from the new response
My code works fine , but I’m using two loop which it’s not the right way I think? I couldn’t do it with one loop any ideas?
def get():
response = requests.get(url, headers=header)
data = response.text
data = json.loads(data)
check_if_theres_next_page = data['pagination']['hasMorePages']
check_for_next_page_number = data['pagination']['nextPage']
last_page_number = data['pagination']['lastPage']
orders = data['orders']
list_of_ids = []
for manufacturingOrderId in orders:
ids = manufacturingOrderId['manufacturingOrderId']
list_of_ids.append(ids)
if check_for_next_page_number == 4:
check_for_next_page_number = last_page_number
if check_if_theres_next_page:
url_ = url + '&page_number=' + str(check_for_next_page_number)
response = requests.get(url_, headers=header)
data = response.text
data = json.loads(data)
orders = data['orders']
for manufacturingOrderId_ in orders:
ids = manufacturingOrderId_['manufacturingOrderId']
list_of_ids.append(ids)
if "nextPage" in data['pagination']:
check_for_next_page_number = data['pagination']['nextPage']
else:
check_if_theres_next_page = False
return list_of_ids

Api call using python and token_auth

"""
#Collects basic metrics from Matomo installation and returns a pandas dataframe
"""
token = os.getenv("token")
# Build url string
base_url = 'https://matomo.___.com/index.php?module=API'
site_num = '&idSite=1'
return_format = '&format=json'
period = '&period=day'
date_range = '&date=last30'
method = '&method=VisitsSummary.get'
token_string = "&token_auth=" + token
my_url = base_url + site_num + return_format + period + date_range + method + token_string
# send request for report
r = requests.get(my_url)
# parse and tidy collected data
data = pd.DataFrame(r.json()).T
data = data.reset_index()
data.columns = [
"date",
"uniq_visitors",
"users",
"visits",
"actions",
"visits_converted",
"bounces",
"sum_visit_length",
"max_actions",
"bounce_rate",
"actions_per_visit",
"avg_time_on_site",
]
return data
I am trying to get data from the matomo API using an auth_token and parameters by using above code but i am not able to access it and my url is not taking token code any one has idea how i can solve this

Given that you are using the request library, passing parameters and headers can be done using the following params in your get call:
r = requests.get(my_url, params=payload)
In the same way, an auth token is usually passed within headers:
r = requests.get(my_url, params=payload, headers=headers)
Using this format you can simply create a headers object which contains your token_auth and directly pass your parameters in a payload object:
headers = {'token_auth': token}
payload = {'module':'API', 'idSite':1, 'format':'json', 'period':'day', 'date':'last30', 'method':'VisitsSummary.get'}
Since you are now passing your parameters in you get request, there is no need to add them to the end of your url. Thus, your url should stay as https://matomo.___.com/index.php. These can then be used within your params and headers respectively. Please note that this assumes that the matomo API places the token_auth in its headers such as most APIs do. If this is not the case you could pass it directly within the params payload.
Here is a global overview:
token = os.getenv("token")
# Get url, headers and params
my_url = 'https://matomo.___.com/index.php'
payload = {'module':'API', 'idSite':1, 'format':'json', 'period':'day', 'date':'last30', 'method':'VisitsSummary.get'}
headers = {'token_auth': token}
# send request for report
r = requests.get(my_url, params=payload, headers=headers)
Note this answers your question specifically regarding the API call and not the processing after.

Requests lib changing characters to URL-encoded representation

I am trying to send a get request to an api that includes timestamps. The URL is getting changed and instead of : %253A is inserted and I get a 500 error code.
url = 'https://www.fleetTrackingSimplicity.com/REST6/api/vehiclehistory/2'
start = '2020-12-01T00:00:00Z'
end = '2020-12-02T00:00:00Z'
param = {'startdate': start, 'enddate': end, 'count':'500'}
r_auth = str(a_json['TransactionId'])
headers2 = dict (Authorization = r_auth, Accept = 'application/json')
r = requests.get(url, headers = headers2, params=param)
When I print r.url I get https://www.fleettrackingsimplicity.com/REST6/api/vehiclehistory/2?startdate=2020-12-01T00%253A00%253A00Z&enddate=2020-12-02T00%253A00%253A00Z&count=500

Python combine multiple similar functions

I am working with an API to pull back data using python. My functions work fine but I feel like I am repeating myself over and over again and there is probably something I should be doing to make this more efficient.
What each one does is gets the number of results then hits the api back up to bring back the exact number of records.
First function:
def get_categories():
headers = {"Authorization": "Bearer " + access_token} # auth plus token
response = requests.get("https://api.destination.com/categories", headers=headers) # response
data = json.loads(response.text) # load the json data
records = str(data['totalResults']) # get number of results for next call
response = requests.get("https://api.destination.com/categories?$skip=0&$top="+records, headers=headers)
all_data = json.loads(response.text) # load the json data
list_of_dict = all_data['resources'] # get rid of all but lists of dictionaries
df = pd.DataFrame.from_records(list_of_dict) # create dataframe
df['links'] = df['links'].str[0].str['href'] # just grab the links(key) items
return df # return the final dataframe
Second function:
def get_groups():
headers = {"Authorization": "Bearer " + access_token} # auth plus token
response = requests.get("https://api.destination.com/groups", headers=headers) # response
data = json.loads(response.text) # load the json data
records = str(data['totalResults']) # get number of results
response = requests.get("https://api.destination.com/groups?$skip=0&$top="+records, headers=headers)
all_data = json.loads(response.text) # load the json data
list_of_dict = all_data['resources'] # get rid of all but lists of dictionaries
df = pd.DataFrame.from_records(list_of_dict) # create dataframe
df['links'] = df['links'].str[0].str['href'] # just grab the links(key) items
return df # return the final dataframe
And 3 more functions like users that do the same thing. The only difference between them as you can see is the getlike https://api.destination.com/categories vs https://api.destination.com/groups and the number of records returned for each will be different. Is there a way to combine these and call it a certain way?

Looks like you already know how to make functions, just extend it one step further to abstract away everything that is common amongst the functions.
BASE_URL = "https://api.destination.com/{}"
def make_headers():
headers = {"Authorization": "Bearer " + access_token}
return headers
def make_params(recs):
params = {'$skip': 0, '$top': recs}
return params
def make_df(data):
list_of_dict = data['resources']
df = pd.DataFrame.from_records(list_of_dict)
df['links'] = df['links'].str[0].str['href']
return df
def process(process):
headers = make_headers()
url = BASE_URL.format(process)
resp = requests.get(url, headers=headers)
data = resp.json()
records = data['totalResults']
params = make_params(records)
resp = requests.get(url, headers=headers, params=params)
all_data = resp.json()
return make_df(all_data)
Then you can call it like the following:
process('groups')
process('categories')
You can break it up further, but you get the idea.

You can just add a parameter to this function.
As an example:
def get_categories():
headers = {"Authorization": "Bearer " + access_token} # auth plus token
response = requests.get("https://api.destination.com/categories", headers=headers) # response
data = json.loads(response.text) # load the json data
records = str(data['totalResults']) # get number of results for next call
response = requests.get("https://api.destination.com/categories?$skip=0&$top="+records, headers=headers)
all_data = json.loads(response.text) # load the json data
list_of_dict = all_data['resources'] # get rid of all but lists of dictionaries
df = pd.DataFrame.from_records(list_of_dict) # create dataframe
df['links'] = df['links'].str[0].str['href'] # just grab the links(key) items
return df # return the final dataframe
You can just refactor to:
def get_elements(element):
if element is None:
return 'not found' #defaults to 404 error.
headers = {"Authorization": "Bearer " + access_token} # auth plus token
response = requests.get("https://api.destination.com/{}".format(element), headers=headers) # response
data = json.loads(response.text) # load the json data
records = str(data['totalResults']) # get number of results for next call
response = requests.get("https://api.destination.com/{}?$skip=0&$top={}".format(element,records), headers=headers)
all_data = json.loads(response.text) # load the json data
list_of_dict = all_data['resources'] # get rid of all but lists of dictionaries
df = pd.DataFrame.from_records(list_of_dict) # create dataframe
df['links'] = df['links'].str[0].str['href'] # just grab the links(key) items
return df # return the final dataframe

How to read the next page on API using python?

I need help on how to do a loop so each time I make a GET request, it will always be the new page from the API.
I start with getting the first response. It includes a parameter to the next page next_key
{
"result": [
{
...,
...
}
],
"next_key": 123
}
Below is my current attempt
import requests
import json
url = "https://flespi.io/gw/channels/all/messages"
headers = {"Authorization": "FlespiToken 23ggh45"}
def getFirst():
data = {"limit_count":100, "limit_size":10000}
params = {"data":json.dumps(data, separators=(",", ":"))}
reqFirst = requests.get(url, params=params, headers=headers).json()
return reqFirst["next_key"] ## this returns "123"
def getDataNext():
data = {"limit_count":100, "limit_size":10000, "curr_key":getFirst()}
params = {"data":json.dumps(data, separators=(",", ":"))}
reqNext = requests.get(url, params=params, headers=headers)
jsonData = reqNext.json()
while True:
if "next_key" in jsonData:
data = {"limit_count":100, "limit_size":10000,"curr_key":jsonData["next_key"]}
params = {"data":json.dumps(data, separators=(",", ":"))}
req = requests.get(url, params=params, headers=headers).json() ## this should do GET request for the third page and so on...
print req["next_key"] # this returns "3321" which is the value for "next_key" in second page
else:
pass
getDataNext()
The full url including limit count, limit size and curr key is as follows https://flespi.io/gw/channels/all/messages?data=%7B%22curr_key%22%123%2C%22limit_count%22%3A100%2C%22limit_size%22%3A10000%7D
As you can see this only returns the second page that is jsonData["next_key"]. What I want to do is that for each GET request, the program will read the next_key and put it on the next GET request.
I am thinking to use increment on the curr_key but the key is random and also I do not know how many page there is.
I believe there must be just a simple solution for this but apparently I could not think about it. Thank you for your help and suggestion.

try this
has_next_key = False
nextKey = ""
if "next_key" in jsonData:
has_next_key = True
nextKey = jsonData["next_key"]
while has_next_key:
data = {"limit_count":100, "limit_size":10000,"curr_key":nextKey}
params = {"data":json.dumps(data, separators=(",", ":"))}
req = requests.get(url, params=params, headers=headers).json() ## this should do GET request for the third page and so on...
if "next_key" in req:
nextKey = req["next_key"]
print nextKey # this returns "3321" which is the value for "next_key" in second page
else:
has_next_key = False
# no next_key, stop the loop