I have been doing a lot of reading on this website regarding post requests from Python to an API. But despite all the recommendations to using the json library within Python, I still cant quite get my head around it.
My current predicament is that I need to make an API call, grab certain fields and post them to another API.
An example of the information i receive from my initial API request:
{
"metadata": {
"configurationVersions": [
3
],
"clusterVersion": "1.174.168.20190814-173650"
},
"id": "5c1547a6-61ca-4dc3-8971-ec8d2f542592",
"name": "Registration",
"enabled": false,
"dataType": "STRING",
"dataSources": [
{
"enabled": true,
"source": "POST_PARAMETER",
"valueProcessing": {
"splitAt": "",
"trim": false
},
"parameterName": "f=register",
"scope": {
"tagOfProcessGroup": "Production"
}
}
],
"normalization": "ORIGINAL",
"aggregation": "FIRST",
"confidential": true,
"skipPersonalDataMasking": true
}
After this call, I extract the data in the following way:
def ReqOutput(output):
x=""
out = ()
inReq = ["name","enabled","dataType","dataSources","normalization","aggregation","confidential","skipPersonalDataMasking"]
for i in output.items():
for item in inReq:
if item in i:
x = x + str(i)
out=out + i
return json.dumps(out)
As recommended in other threads, I used to json.dumps method to convert my python tuple to JSON. However, I feel like it is not working as intended
Pre json.dumps output:
'name', 'Registration', 'enabled', False, 'dataType', 'STRING', 'dataSources', [{'enabled': True, 'source': 'POST_PARAMETER', 'valueProcessing': {'splitAt': '', 'trim': False}, 'parameterName': 'f=register', 'scope': {'tagOfProcessGroup': 'Production'}}], 'normalization', 'ORIGINAL', 'aggregation', 'FIRST', 'confidential', True, 'skipPersonalDataMasking', True)
Post json.dumps output:
["name", "Registration", "enabled", false, "dataType", "STRING", "dataSources", [{"enabled": true, "source": "POST_PARAMETER", "valueProcessing": {"splitAt": "", "trim": false}, "parameterName": "f=register", "scope": {"tagOfProcessGroup": "Production"}}], "normalization", "ORIGINAL", "aggregation", "FIRST", "confidential", true, "skipPersonalDataMasking", true]
I then try and POST this to another API using:
def PostRequest (data):
postURL = "XXXX"
headers = {'Content-type': 'application/json'}
r = requests.post(postURL,data = data,headers = headers)
print(r.text)
Where I am finally met with the error:
{"error":{"code":400,"message":"Could not map JSON at '' near line 1 column 1"}}
Try getting rid of the for loops in favor of a dict comprehension:
def ReqOutput(output):
inReq = ["name","enabled","dataType","dataSources","normalization","aggregation","confidential","skipPersonalDataMasking"]
out = {output: val for output, val in output.items() if output in inReq}
return json.dumps(out)
This is more readable and will always give you a dict with attributes from output that are in inReq. The reason your JSON looks like that is because serializing a tuple will give you an Array-like structure. If what you want is an Object structure, you should serialize a dict-like object.
Related
I am currently writing a script to scrape data from an API into a Python dictionary and then export the result into a JSON file. I am trying to get the file extension from a response by splitting using .rsplit('.', 1)[-1] The only problem is that some keys have 'None" as their value and this throws the AttributeError: 'NoneType' object has no attribute 'rsplit'. Here is my code snippet:
d = requests.get(dataset_url)
output = d.json()
output_dict = {
'data_files': {
# Format to get only extension
'format': output.get('connectionParameters', {}).get('url').rsplit('.', 1)[-1],
'url': output.get('connectionParameters', {}).get('url'),
},
}
An example of JSON response with the required key is as follows:
"connectionParameters": {
"csv_escape_char": "\\",
"protocol": "DwC",
"automation": false,
"strip": false,
"csv_eol": "\\n",
"csv_text_enclosure": "\"",
"csv_delimiter": "\\t",
"incremental": false,
"url": "https://registry.nbnatlas.org/upload/1564481725489/London_churchyards_dwc.txt",
"termsForUniqueKey": [
"occurrenceID"
]
},
Any way to tackle this?
Try:
url = output.get("connectionParameters", {}).get("url") or "-"
f = url.rsplit(".", 1)[-1]
output_dict = {
"data_files": {
"format": f,
"url": url,
},
}
This will print:
{'data_files': {'format': '-', 'url': '-'}}
If the "url" parameter is None.
Newbie here. Thanks in advance for your patience!
I'm using requests in Pandas to get info from an API. Here are the API results:
{
"data": [
{
"all_awardings": [],
"allow_live_comments": false,
"author": "qu1et1",
"author_flair_css_class": null,
"author_flair_richtext": [],
"author_flair_text": null,
"author_flair_type": "text",
"author_fullname": "t2_will3i",
"author_patreon_flair": false,
"author_premium": false,
"awarders": [],
"can_mod_post": false,
"contest_mode": false,
"created_utc": 1577837075,
"domain": "open.spotify.com",
"full_link": "https://www.reddit.com/r/SpotifyPlaylists/comments/eib232/instant_addiction_jan_2020_update_your_selective/",
"gildings": {},
"id": "eib232",
"is_crosspostable": true,
"is_meta": false,
"is_original_content": false,
"is_reddit_media_domain": false,
"is_robot_indexable": true,
"is_self": false,
"is_video": false,
"link_flair_background_color": "",
"link_flair_richtext": [],
"link_flair_template_id": "933b36a2-d08e-11e8-883e-0e5f4502fa8a",
"link_flair_text": "Various",
"link_flair_text_color": "dark",
"link_flair_type": "richtext",
"locked": false,
"media": {
"oembed": {
"description": "We and our partners use cookies to personalize your experience, to show you ads based on your interests, and for measurement and analytics purposes. By using our website and our services, you agree to our use of cookies as described in our Cookie Policy.",
"height": 380,
I'm trying to get the contents of the "description" key. I've tried:
data = getPushshiftData(after,last_epoch)
for post in data:
description = post[media][oembed]["description"]
Since media and oembed are dictionaries, I'm not putting them in quotes.
I'm getting an unhashable type: 'dict' error. Could someone help me as to where I'm going wrong?
Edit: I've tried putting media and oembed in quotes, and it's giving me a key error for 'media'.
while int(after) < last_epoch:
data = getPushshiftData(after,last_epoch)
for post in data:
tmp_time = post['created_utc']
timestamps.append(tmp_time)
tmp_id = post['id']
post_ids.append(tmp_id)
tmp_full_link = post['full_link']
full_link.append(tmp_full_link)
tmp_description = (post['media']['oembed']['description']) #error
You just need to access it with quotes, I saved your output JSON in a file and read it into a dict using json.load:
with open('1.json', 'r+') as f:
data = json.load(f)
for rec in data['data']:
print(rec['media']['oembed']['description'])
Output:
We and our partners use cookies to personalize your experience, to show you ads based on your interests, and for measurement and analytics purposes. By using our website and our services, you agree to our use of cookies as described in our Cookie Policy.
I'm trying to do a basic Bitly shortening URL call. However, I cannot seem to either push the json correctly, or deal with the json response correctly... I omitted some obvious variables for brevity and obfuscated some real values for security purposes.
import requests
import json
bitly_header = {'Authorization':'Bearer
some_long_secret_character_string_here', 'Content-Type':'application/json'}
bitly_data = {
"long_url": ""+long_url+"",
"group_guid": ""+bitly_guid+""
}
short_link_resp =requests.post(bitly_endpoint,data=bitly_data,headers=bitly_header)
short_link_json = short_link_resp.json()
short_link = short_link_json["link"]
Errors is "Key error: 'link'
The json I get from Postman is:
{
"created_at": "1970-01-01T00:00:00+0000",
"id": "bit.ly/2MjdrrG",
"link": "bit.ly/2MjdrrG",
"custom_bitlinks": [],
"long_url": "google.com/",
"archived": false,
"tags": [],
"deeplinks": [],
"references": {
"group": "https://api-ssl.bitly.com/v4/groups/Bi7i8IbM1x9"
}
}
try replace data with json:
short_link_resp =requests.post(bitly_endpoint, json=bitly_data, headers=bitly_header)
see the doc ref.
I'm trying to send a json in POST method using REST, but i got error:
"Could not parse JSON data: Expecting value: line 1 column 1
(char 0)", "status": 500, "type": "ValueError",
"request": {"client_addr": "127.0.0.1",
"user_agent": "python-requests/2.3.0 CPython/3.4.2 Linux/3.16.0-41-generic",
"method": "POST", "path": "/api/adley/doc"}}
)
I'm trying to fix, use a json.dumps or json.loads, but nothing seems to work.
I need to send one key and two values. Here is the base:
{
metadata: {
idx_exp: false,
idx_exp_time: 0,
file_ext: false,
password: "",
description: "base de teste",
idx_exp_url: "",
model: {
item: "Text"
},
dt_base: "29/06/2015 14:47:10",
name: "adley",
id_base: 5,
file_ext_time: 0,
},
content: [
{
field: {
required: false,
multivalued: false,
alias: "Texto",
datatype: "Text",
name: "item",
indices: [
"Ordenado"
],
description: "placeholder"
}
}
]
}
My to send a post script:
import requests, json
url = "http://127.0.0.1/api/adley/doc"
json_data = {'value':{'item':'test'}}
response = requests.post(url, params=json_data)
print(response.text)
I can't see what's wrong with my script
try passing the data parameter instead of the params parameter to the post method. requests wants to url encode params where you want the json data. So something like this:
requests.post(url, data=json.dumps(json_data))
I'm tyring to use Mandrill on Google App Engine, with Python, and for that I'm using the urlfetch module, provided by Google.
The adaptation of the code I'm using for it is the following:
#The jMessage below was split with the formatting below to make it easier to follow
#the real code doesn't have such indentation issues
jMessage =
{
"key": mailchimp_key,
"message":
{
"track_clicks": False,
"track_opens": True,
"tags": [],
"google_analytics_domains": [],
"html": message_html,
"subject": "Reset your password",
"from_name": None,
"merge_var_map": {},
"auto_text": False,
"from_email": from_email,
"headers": {},
"google_analytics_campaign": [],
"merge_vars": [],
"text": None,
"global_merge_vars": [],
"url_strip_qs":False,
"to":
[
{
"email": to_email,
"name": None
}
]
}
}
headers = {'content-type': 'application/json'}
result = urlfetch.fetch(url='https://mandrillapp.com/api/1.0/messages/send.json',\
headers=headers, payload=json.dumps(jMessage), method=urlfetch.POST)
print result.content
The output for that is the following:
{"status":"error","code":-2,"name":"ValidationError","message":"Validation error{\"message\":\"Please enter an array\"}"}
What exactly is that array it wants? I was unsure on what to do, so I wrapped the "message" key, from the jMessagewith [], making it a list. Indeed, it stopped complaining about the array, however the output was simply [], and I didn't receive any email.