Develop Reference

Python Requests Post within a nested Json - retrieve data with a specific value

I already look in stackoverflow and I could not find an answer to my problem.
I'm accessing an API from the German Government that has a output limit of 10.000 entries. I want all data from a specific city, and since there is more than 10.000 entries in the original database, I need to "do the query" while doing the requests.post.
Here is one entry of Json result, when I simply do request.post to this API:
{
"results":[
{
"_id":"CXPTYYFY807",
"CREATED_AT":"2019-12-17T14:48:17.130Z",
"UPDATED_AT":"2019-12-17T14:48:17.130Z",
"result":{
"id":"CXPTYYFY807",
"title":"Bundesstadt Bonn, SGB-315114, Ortsteilzentrum Brüser Berg, Fliesenarbeiten",
"description":["SGB-315114","Ortsteilzentrum Brüser Berg, Fliesenarbeiten"],
"procedure_type":"Ex ante Veröffentlichung (§ 19 Abs. 5)",
"order_type":"VOB",
"publication_date":"",
"cpv_codes":["45431000-7","45431100-8"],
"buyer":{
"name":"Bundesstadt Bonn, Referat Vergabedienste",
"address":"Berliner Platz 2",
"town":"Bonn",
"postal_code":"53111"},
"seller":{
"name":"",
"town":"",
"country":""
},
"geo":{
"lon":7.0944,
"lat":50.73657
},
"value":"",
"CREATED_AT":"2019-12-17T14:48:17.130Z",
"UPDATED_AT":"2019-12-17T14:48:17.130Z"}
}
],
"aggregations":{},
"pagination":{
"total":47389,
"start":0,
"end":0 }}
What I want is all the data which was bought in "town" : "Bonn"
What I already tryed:
import requests
url = 'https://daten.vergabe.nrw.de/rest/evergabe/aggregation_search'
headers = {'Accept': 'application/json', 'Content-Type': 'application/json'}
data = {"results": [{"result": {"buyer": {"town":"Bonn"}}}]}
#need to put the size limit, otherwise he delivers me less:
params = {'size': 10000}
req = requests.post(url, params=params, headers=headers, json=data)
This returns me the post, but not "filtered" by city.
I also tryed req = requests.post(url, params=params, headers=headers, data=data) , which returns me ERROR 400 .
Another way is to grab all the data with the pagination parameters on the end of the json code within a loop, but again I'm not being able to writwe down the json path to the pagination, for example : start: 0 , end:500
Can anyone help me solving it?

Try:
url = 'https://daten.vergabe.nrw.de/rest/evergabe/aggregation_search'
headers = {'Accept': 'application/json', 'Content-Type': 'application/json'}
query1 = {
"query": {
"match": {
"buyer.town": "Bonn"
}
}
}
req = requests.post(url, headers=headers, json=query1)
# Check the output
req.text
Edit:
This won't work if the filter matches with more than 10.000 results, but it may be a quick workaround to the problem you are facing.

import json
import requests
import math
url = "https://daten.vergabe.nrw.de/rest/vmp_rheinland"
size = 5000
payload = '{"sort":[{"_id":"asc"}],"query":{"match_all":{}},"size":'+str(size)+'}'
headers = {
'accept': "application/json",
'content-type': "application/json"
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers)
tenders_array = []
query_data = json.loads(response.text)
tenders_array.extend(query_data['results'])
total_hits = query_data['pagination']['total']
result_size = len(query_data['results'])
last_id = query_data['results'][-1]["_id"]
number_of_loops = ((total_hits - size) // size )
last_loop_size = ((total_hits - size) % size)
for i in range(number_of_loops+1):
if i == number_of_loops:
size=last_loop_size
payload = '{"sort":[{"_id":"asc"}],"query":{"match_all":{}},"size":'+str(size)+',"search_after":["'+last_id+'"]}'
response = requests.request("POST", url, data=payload, headers=headers)
query_data = json.loads(response.text)
result_size = len(query_data['results'])
if result_size > 0:
tenders_array.extend(query_data['results'])
last_id = query_data['results'][-1]["_id"]
else:
break
https://gist.github.com/thiagoalencar/34401e204358499ea3b9aa043a18395f
code in the gist.
Some code to paginate through elasticsearch API. This is an API over the elasticsearch API, and the docs where not so clear. Tried scroll, no sucess. This solutions uses search_after parameter without point in time, because the endpoint is not available. Some times the servers refuses the request and it is necessary to verify with response.status_code==502.
The code is messy and need refactoring. But it works. The final tenders_array contains all objects.

Trying to retrieve data from the Anbima API

I'm trying to automate a process in which i have to download some brazilian fund quotes from Anbima (Brazil regulator). I have been able to work around the first steps to retrieve the access token but i don't know how to use the token in order to make requests. Here is the tutorial website https://developers.anbima.com.br/en/como-acessar-nossas-apis/.
I have tried a lot of thing but all i get from the request is 'Could not find a required APP in the request, identified by HEADER client_id.'
If someone could share some light. Thank you in advance.
import requests
import base64
import json
requests.get("https://api.anbima.com.br/feed/fundos/v1/fundos")
ClientID = '2Xy1ey11****'
ClientSecret = 'faStF1Hc****'
codeString = ClientID + ":" + ClientSecret
codeStringBytes = codeString.encode('ascii')
base64CodeBytes = base64.b64encode(codeStringBytes)
base64CodeString = base64CodeBytes.decode('ascii')
url = "https://api.anbima.com.br/oauth/access-token"
headers = {
'content-type': 'application/json'
,'authorization': f'Basic {base64CodeString}'
}
body = {
"grant_type": "client_credentials"
}
r = requests.post(url=url, data=json.dumps(body), headers=headers, allow_redirects=True)
jsonDict = r.json()
##################
urlFundos = "https://api-sandbox.anbima.com.br/feed/precos-indices/v1/titulos-publicos/mercado-secundario-TPF"
token = jsonDict['access_token']
headers2 = {
'content-type': 'application/json'
,'authorization': f'Bearer {token}'
}
r2 = requests.get(url=urlFundos, headers=headers2)
r2.status_code
r2.text

I was having the same problem, but today I could advance. I believe you need to adjust some parameters in the header.
Follows the piece of code I developed.
from bs4 import BeautifulSoup
import requests
PRODUCTION_URL = 'https://api.anbima.com.br'
SANDBOX_URL = 'https://api-sandbox.anbima.com.br'
API_URL = '/feed/fundos/v1/fundos/'
CODIGO_FUNDO = '594733'
PRODUCTION = False
if PRODUCTION:
URL = PRODUCTION_URL
else:
URL = SANDBOX_URL
URL = URL + API_URL + CODIGO_FUNDO
HEADER = {'access_token': 'your token',
'client_id' : 'your client ID'}
html = requests.get(URL, headers=HEADER).content
soup = BeautifulSoup(html, 'html.parser')
print(soup.prettify())
The sandbox API will return a dummy JSON. To access the production API you will need to request access (I'm trying to do this just now).

url = 'https://api.anbima.com.br/oauth/access-token'
http = 'https://api-sandbox.anbima.com.br/feed/precos-indices/v1/titulos-publicos/pu-intradiario'
client_id = "oLRa*******"
client_secret = "6U2nefG*****"
client_credentials = "oLRa*******:6U2nefG*****"
client_credentials = client_credentials.encode('ascii')
senhabytes = base64.b64encode(client_credentials)
senha = base64.b64decode(senhabytes)
print(senhabytes, senha)
body = {
"grant_type": "client_credentials"
}
headers = {
'content-type': 'application/json',
'Authorization': 'Basic b0xSYTJFSUlOMWR*********************'
}
request = requests.post(url, headers=headers, json=body, allow_redirects=True)
informacoes = request.json()
token = informacoes['access_token']
headers2 = {
"content-type": "application/json",
"client_id": f"{client_id}",
"access_token": f"{token}"
}
titulos = requests.get(http, headers=headers2)
titulos = fundos.json()
I used your code as a model, then I've made some changes. I've printed the encode client_id:client_secret and then I've copied and pasted in the headers.
I've changed the data for json.

How to pass return as a value for argument with adding a string?

I prepared this method
def make_bearer_token(token_url):
body = {
"scope": "ingo_dev",
"client_id": "machine__dev",
"client_secret": "0c3b99",
"username": "test.user",
"password": "xxx",
"grant_type": "pass",
}
response = requests.post(token_url, data=body).json()
bearer_token = response["access_token"]
print({'authorization: ' + 'Bearer ' + str(response)})
return bearer_token
I want to pass that bearer_token to another method with string 'authorization':bearer_token
as a header parameter. The method is looking something like that.
#staticmethod
def post_request(endpoint, file, timeout=30):
response = requests.post(endpoint, files=file_dict, timeout=timeout)
return response

A simple solution would be to run your make_bearer_token() from within your post_request method.
Bearer is a header so make a new headers dict with your Authorization header in it, and pass it as a parameter to requests.post()
def post_request(endpoint, file, timeout=30):
headers = {
"Authorization": "Bearer {}".format(make_bearer_token(endpoint))
}
response = requests.post(endpoint, files=file_dict, headers=headers, timeout=timeout)
return response
Edit: Based on your comment below, you might want to make use of class variables
class Uploader:
def __init__(self, url):
self.headers = {
"Authorization": "Bearer {}".format(self.make_bearer_token(url))
}
def post_request(self, endpoint, file, timeout=30):
response = requests.post(endpoint, files=file_dict, headers=self.headers, timeout=timeout)
return response

Usually response access_token don't have prefix bearer. So you try this way.
import requests
bearer_token = response["access_token"]
hed = {'Authorization': 'Bearer ' + bearer_token}
data = {'key' : 'value'}
url = 'https://next.url.com'
response = requests.post(url, json=data, headers=hed)
print(response)
print(response.json())
Please try this. Let me know you feedback on the same.

How to update a pull request through Github API

I want to update the title of a pull request and performing the below to achieve it :- (followed this doc https://developer.github.com/v3/pulls/#update-a-pull-request)
data = {"title": "New title"}
url='https://hostname/api/v3/repos/owner/repo/pulls/80'
token = 'my-token'
headers = {'Content-type': 'application/json', 'Accept': 'application/json', 'Authorization': 'token %s' % token}
resp = requests.patch(url, data=json.dumps(data), headers=headers)
print resp.json()
What am I missing ? Please help.

The following worked for me:
import requests
token = "my-token"
url = "https://api.github.com/repos/:owner/:repo/pulls/:number"
payload = {
"title": "New title"
}
r = requests.patch(url, auth=("username", token), json=payload)
print r.json()

Using Python to Post json contact list to Qualtrics API, error with Content-Type

I'm trying to import contacts into a contact list in Qualtrics. I am using python to do this.
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
headers = {'X-API-TOKEN': Token, 'Content-Type':'application/json',}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports', headers=headers, data=data)
r.text
This code gives me the following error: '{"meta":{"httpStatus":"400 - Bad Request","error":{"errorMessage":"Invalid Content-Type. expected=multipart/form-data found=application/json","errorCode":"RP_0.1"},"requestId":null}}'
I changed the content type to multipart/form-data that it says it is expecting and received the response "413", which qualtrics explains means "The request body was too large. This can also happen in cases where a multipart/form-data request is malformed."
I have tested my json and verified that it is valid. Also, I don't know why the request body would be too large because it's only 13 contacts that I'm trying to import. Any ideas?

With the help of Qualtrics Support, I was eventually able to get the following code to work:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
url = "https://az1.qualtrics.com/API/v3/mailinglists/" + ContactsID + "/contactimports/"
headers = {
'content-type': "multipart/form-data; boundary=---BOUNDRY",
'x-api-token': "Token"
}
files = {'contacts': ('contacts', open('contacts.json', 'rb'), 'application/json')}
request = requests.post(url, headers=headers, files=files)
print(request.text)
Please note that if you want to use this code, you will need to change "az1" in the URL to your own Qualtrics datacenter ID.

You need to use files = .. for a multipart request:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
headers = {'X-API-TOKEN': Token}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports',files={"file":data}, headers=headers)
r.text
Once you do requests will take care of the rest:
In [36]: url = 'http://httpbin.org/post'
In [37]: headers = {'X-API-TOKEN': "123456789"}
In [38]: files = {'file': open('a.csv', 'rb')}
In [39]: r = requests.post(url, files=files, headers=headers)
In [40]: print r.text
{
"args": {},
"data": "",
"files": {
"file": "a,b,c\n1,2,3"
},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "152",
"Content-Type": "multipart/form-data; boundary=3830dbe5fa6141f69d3d85dee4ba6e78",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.10.0",
"X-Api-Token": "123456789"
},
"json": null,
"origin": "51.171.98.185",
"url": "http://httpbin.org/post"
}
In [41]: print(r.request.body)
--3830dbe5fa6141f69d3d85dee4ba6e78
Content-Disposition: form-data; name="file"; filename="a.csv"
a,b,c
1,2,3
--3830dbe5fa6141f69d3d85dee4ba6e78--
looking at the docs, you actually want something closer to:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
files = {'file': ('contact', data ,'application/json', {'X-API-TOKEN': Token})}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports',files=files)