I've got a puzzling requests that I do not seem able to dig myself out of, despite several attempts.
Context / Workflow:
1. Logging to a webservice
2. Make a Get request
3. Obtain JSoN
Issue:
- JSON obtained by going via browser is different (read: has more data) than the one obtained via my code
My code:
with requests.Session() as s:
s.headers = {'Accept': 'application/json; indent=4', 'Content-Type': 'application/json'}
secret = {
"client_id": "xxx",
"client_secret": "yyy"
}
datas = json.dumps(secret)
url = 'https://XXXX.yyy/sign_in'
response = s.post(url, data=json.dumps(secret), headers=s.headers, verify=False)
print (s.headers)
print(response)
This part works OK, returns me a cookie, which I save on
ck = s.cookies.get_dict()
I then need to make a Get request with the cookies saved after successful login
url = 'https://give.me/tests'
print ("Connecting to: " + url)
# here we do our get, passing the cookies
response = s.get(url, cookies=ck, verify=False)
data = response.text
# print (s.headers) # is OK, returns 200
# print(data)
with open('data.txt', 'w') as f:
f.write(data)
return 0
Expected Result:
"phase": {
"type": "phase",
"activity_level": null,
"logical_name": "phase.test_manual.obsolete",
"name": "Obsolete",
"index": 4,
"id": "phase.test_manual.obsolete"
},
Actual Result:
"phase": {
"type": "phase",
"id": "phase.test_manual.obsolete"
},
Documentation, in case I'm doing something wrong functionally:
Related
I am creating a charge and I want from it to take its 'hosted_url' in order to redirect a user from page.
I am trying to extract it from json, but I newbie and don't know how
...
url = "https://api.commerce.coinbase.com/charges"
payload = {
"local_price": {
"amount": 1,
"currency": USDT
},
"name": "Test for fun",
"description": "Project 2",
"pricing_type": "fixed_price"
}
headers = {
"accept": "application/json",
"X-CC-Version": "2018-03-22",
"X-CC-Api-Key" : "**********",
"content-type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)
This is a code to make a request to an api in order to create a charge and in JSON Response I get these fields:
{
"data":{
"code": "123DVAS",
"hosted_url": "https://commerce.coinbase.com/charges/123DVAS",
...
}
}
I want somehow to put this message '123DVAS' in variable or to make a redirect like this:
return render(request, 'https://commerce.coinbase.com/charges/123DVAS')
You can do like this:
# Suppose it's your response looks like this
response = {"hosted_url": "https://commerce.coinbase.com/charges/123DVAS"}
id_fecthed = response.get('hosted_url').split('/')[-1]
# You can redner like this
render(request, f'https://commerce.coinbase.com/charges/{id_fecthed}')
# If you sure that always you want to redirect to hosted_url
# Then you can use this
render(request, response.get('hosted_url'))
Edit:
Full code with data read from requests.
import json
response = requests.post(url, json=payload, headers=headers)
data = json.loads(response.text)
hosted_url = data.get("data",{}).get('hosted_url')
if hosted_url:
id_fecthed = hosted_url.split('/')[-1]
return redirect(f'https://commerce.coinbase.com/charges/{id_fecthed}')
else:
print('Host URL not available')
I'm trying to get url value from the api, but have an issue saying TypeError: string indices must be integers
Here is the array that I get from api:
[
{
"created_utc": 1643524062,
"title": "title",
"url": "https://i.redd.it/tmd5shz9rre81.gif",
},
{
"created_utc": 1643530657,
"title": "title",
"url": "https://i.redd.it/qqjykysxase81.gif",
}
]
And here is the code I use to get the url:
url = "https://reddit-meme.p.rapidapi.com/memes/trending"
headers = {
"X-RapidAPI-Key": "83df5aba87msh4580fa40781b33cp12157bjsnb4b412cb57da",
"X-RapidAPI-Host": "reddit-meme.p.rapidapi.com"
}
response = requests.request("GET", url, headers=headers)
print(response.text[0]["url"])
What am I doing wrong?
response.text is a string, you have to parse it first, with the json librarie, like this:
import requests
import json
url = "https://reddit-meme.p.rapidapi.com/memes/trending"
headers = {
"X-RapidAPI-Key": "83df5aba87msh4580fa40781b33cp12157bjsnb4b412cb57da",
"X-RapidAPI-Host": "reddit-meme.p.rapidapi.com"
}
response = requests.request("GET", url, headers=headers)
data = json.loads(response.text)
print(data[0]["url"])
I already look in stackoverflow and I could not find an answer to my problem.
I'm accessing an API from the German Government that has a output limit of 10.000 entries. I want all data from a specific city, and since there is more than 10.000 entries in the original database, I need to "do the query" while doing the requests.post.
Here is one entry of Json result, when I simply do request.post to this API:
{
"results":[
{
"_id":"CXPTYYFY807",
"CREATED_AT":"2019-12-17T14:48:17.130Z",
"UPDATED_AT":"2019-12-17T14:48:17.130Z",
"result":{
"id":"CXPTYYFY807",
"title":"Bundesstadt Bonn, SGB-315114, Ortsteilzentrum Brüser Berg, Fliesenarbeiten",
"description":["SGB-315114","Ortsteilzentrum Brüser Berg, Fliesenarbeiten"],
"procedure_type":"Ex ante Veröffentlichung (§ 19 Abs. 5)",
"order_type":"VOB",
"publication_date":"",
"cpv_codes":["45431000-7","45431100-8"],
"buyer":{
"name":"Bundesstadt Bonn, Referat Vergabedienste",
"address":"Berliner Platz 2",
"town":"Bonn",
"postal_code":"53111"},
"seller":{
"name":"",
"town":"",
"country":""
},
"geo":{
"lon":7.0944,
"lat":50.73657
},
"value":"",
"CREATED_AT":"2019-12-17T14:48:17.130Z",
"UPDATED_AT":"2019-12-17T14:48:17.130Z"}
}
],
"aggregations":{},
"pagination":{
"total":47389,
"start":0,
"end":0 }}
What I want is all the data which was bought in "town" : "Bonn"
What I already tryed:
import requests
url = 'https://daten.vergabe.nrw.de/rest/evergabe/aggregation_search'
headers = {'Accept': 'application/json', 'Content-Type': 'application/json'}
data = {"results": [{"result": {"buyer": {"town":"Bonn"}}}]}
#need to put the size limit, otherwise he delivers me less:
params = {'size': 10000}
req = requests.post(url, params=params, headers=headers, json=data)
This returns me the post, but not "filtered" by city.
I also tryed req = requests.post(url, params=params, headers=headers, data=data) , which returns me ERROR 400 .
Another way is to grab all the data with the pagination parameters on the end of the json code within a loop, but again I'm not being able to writwe down the json path to the pagination, for example : start: 0 , end:500
Can anyone help me solving it?
Try:
url = 'https://daten.vergabe.nrw.de/rest/evergabe/aggregation_search'
headers = {'Accept': 'application/json', 'Content-Type': 'application/json'}
query1 = {
"query": {
"match": {
"buyer.town": "Bonn"
}
}
}
req = requests.post(url, headers=headers, json=query1)
# Check the output
req.text
Edit:
This won't work if the filter matches with more than 10.000 results, but it may be a quick workaround to the problem you are facing.
import json
import requests
import math
url = "https://daten.vergabe.nrw.de/rest/vmp_rheinland"
size = 5000
payload = '{"sort":[{"_id":"asc"}],"query":{"match_all":{}},"size":'+str(size)+'}'
headers = {
'accept': "application/json",
'content-type': "application/json"
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers)
tenders_array = []
query_data = json.loads(response.text)
tenders_array.extend(query_data['results'])
total_hits = query_data['pagination']['total']
result_size = len(query_data['results'])
last_id = query_data['results'][-1]["_id"]
number_of_loops = ((total_hits - size) // size )
last_loop_size = ((total_hits - size) % size)
for i in range(number_of_loops+1):
if i == number_of_loops:
size=last_loop_size
payload = '{"sort":[{"_id":"asc"}],"query":{"match_all":{}},"size":'+str(size)+',"search_after":["'+last_id+'"]}'
response = requests.request("POST", url, data=payload, headers=headers)
query_data = json.loads(response.text)
result_size = len(query_data['results'])
if result_size > 0:
tenders_array.extend(query_data['results'])
last_id = query_data['results'][-1]["_id"]
else:
break
https://gist.github.com/thiagoalencar/34401e204358499ea3b9aa043a18395f
code in the gist.
Some code to paginate through elasticsearch API. This is an API over the elasticsearch API, and the docs where not so clear. Tried scroll, no sucess. This solutions uses search_after parameter without point in time, because the endpoint is not available. Some times the servers refuses the request and it is necessary to verify with response.status_code==502.
The code is messy and need refactoring. But it works. The final tenders_array contains all objects.
I would get access to this website, I get it from another website, https://www.betexplorer.com/soccer/algeria/ligue-1/bordj-bou-arreridj-tlemcen/tfvAHu7U/, in the Network section of Developer tools.
Code
import requests
url = 'https://www.betexplorer.com/archive-odds/4urejxv464x0xd4645/18/'
response = requests.get(url, headers={'User-Agent': 'my user agent'})
print(response)
Output <Response [404]>
Set Referer HTTP header to obtain correct response:
import json
import requests
url = "https://www.betexplorer.com/archive-odds/4urejxv464x0xd4645/18/"
headers = {
"Referer": "https://www.betexplorer.com",
}
data = requests.get(url, headers=headers).json()
print(json.dumps(data, indent=4))
Prints:
[
{
"date": "23.07. 17:36",
"odd": "1.88",
"change": "+0.08"
},
{
"date": "23.07. 17:34",
"odd": "1.80",
"change": "-0.01"
},
...
I'm trying to import contacts into a contact list in Qualtrics. I am using python to do this.
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
headers = {'X-API-TOKEN': Token, 'Content-Type':'application/json',}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports', headers=headers, data=data)
r.text
This code gives me the following error: '{"meta":{"httpStatus":"400 - Bad Request","error":{"errorMessage":"Invalid Content-Type. expected=multipart/form-data found=application/json","errorCode":"RP_0.1"},"requestId":null}}'
I changed the content type to multipart/form-data that it says it is expecting and received the response "413", which qualtrics explains means "The request body was too large. This can also happen in cases where a multipart/form-data request is malformed."
I have tested my json and verified that it is valid. Also, I don't know why the request body would be too large because it's only 13 contacts that I'm trying to import. Any ideas?
With the help of Qualtrics Support, I was eventually able to get the following code to work:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
url = "https://az1.qualtrics.com/API/v3/mailinglists/" + ContactsID + "/contactimports/"
headers = {
'content-type': "multipart/form-data; boundary=---BOUNDRY",
'x-api-token': "Token"
}
files = {'contacts': ('contacts', open('contacts.json', 'rb'), 'application/json')}
request = requests.post(url, headers=headers, files=files)
print(request.text)
Please note that if you want to use this code, you will need to change "az1" in the URL to your own Qualtrics datacenter ID.
You need to use files = .. for a multipart request:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
headers = {'X-API-TOKEN': Token}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports',files={"file":data}, headers=headers)
r.text
Once you do requests will take care of the rest:
In [36]: url = 'http://httpbin.org/post'
In [37]: headers = {'X-API-TOKEN': "123456789"}
In [38]: files = {'file': open('a.csv', 'rb')}
In [39]: r = requests.post(url, files=files, headers=headers)
In [40]: print r.text
{
"args": {},
"data": "",
"files": {
"file": "a,b,c\n1,2,3"
},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "152",
"Content-Type": "multipart/form-data; boundary=3830dbe5fa6141f69d3d85dee4ba6e78",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.10.0",
"X-Api-Token": "123456789"
},
"json": null,
"origin": "51.171.98.185",
"url": "http://httpbin.org/post"
}
In [41]: print(r.request.body)
--3830dbe5fa6141f69d3d85dee4ba6e78
Content-Disposition: form-data; name="file"; filename="a.csv"
a,b,c
1,2,3
--3830dbe5fa6141f69d3d85dee4ba6e78--
looking at the docs, you actually want something closer to:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
files = {'file': ('contact', data ,'application/json', {'X-API-TOKEN': Token})}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports',files=files)