Develop Reference

How to parse JSON with a list of lists?

I am trying to parse a "complicated" JSON string that is returned to me by an API.
It looks like this:
{
"data":[
["Distance to last strike","23.0","miles"],
["Time of last strike","1/14/2022 9:23:42 AM",""],
["Number of strikes today","1",""]
]
}
While the end goal will be to extract the distance, date/time, as well as count, for right now I am just trying to successfully get the distance.
My python script is:
import requests
import json
response_API = requests.get('http://localhost:8998/api/extra/lightning.json')
data = response_API.text
parse_json = json.loads(data)
value = parse_json['Distance to last strike']
print(value)
This does not work. If I change the value line to
value = parse_json['data']
then the entire string I listed above is returned.
I am hoping it's just a simple formatting issue. Suggestions?

You have an object with a list of lists. If you fetch
value = parse_json['data']
Then you will have a list containing three lists. So:
print(value[0][1])
will print "23.0".

How can I append values from a JSON dictionary to a new list?

I have a .json file of all of my AWS target groups. This was created using aws elbv2 describe-target-groups. I want to extract every TargetGroupArn from this file and store it into a Python list.
With my current code, I get no output. I can confirm that the dictionary has data in it, but nothing is being appended to the list that I'm trying to create.
import json
from pprint import pprint
with open('target_groups.json') as f:
data = json.load(f)
items = data['TargetGroups']
arn_list = []
for key, val in data.items():
if key == 'TargetGroupArn':
arn_list.append(val)
print(arn_list)
Expected results would be for arn_list to print out looking like this:
[arn:aws:elb:xxxxxxx:targetgroup1, arn:aws:elb:xxxxxxx:targetgroup2, arn:aws:elb:xxxxxxx:targetgroup3]

Change your code to this:
import json
from pprint import pprint
with open('target_groups.json') as f:
data = json.load(f)
arn_list = []
if 'TargetGroups' in data:
items = data['TargetGroups']
for item in items:
if 'TargetGroupArn' in item:
arn_list.append(item['TargetGroupArn'])
print(arn_list)
else:
print('No data')
There are many ways to make this python code more concise. However, I prefer a more wordy style that easier to read.
Also note that this code checks that keys exist so that the code will not stackdump for missing data.

it would be better if you could post the file you are trying to get data from, but this part:
for key, val in data.items():
if key == 'TargetGroupArn':
arn_list.append(val)
need to be changed to:
for key, val in items.items():
if key == 'TargetGroupArn':
arn_list.append(val)
you get data from 'data' and add it to items, but you never actually used it.
give it a shot.

Converting Python Dictionary to JSON Array

I currently have a Python Dictionary that looks something like this:
OrderedDict([('2017-07-24', 149.7619), ('2017-07-25', 150.4019), ('2017-07-26', 151.1109), ...
that I am converting to JSON like so:
one_yr = json.dumps(priceDict)
Currently I am adding values to the dictionary from an SQL query by looping through it like so:
for i in query:
date = i[0]
close = i[1]
priceDict[date] = close
The problem is that this returns a JSON object, that i then have to convert to a JSON array.
I am wondering if I can just convert my Python Dictionary to a JSON array directly? Thanks.

json.dumps(list(priceDict.items()))
But why do you have an OrderedDict in first place? If you pass the same list you passed to OrderedDict to json.dumps it will generate your array:
json.dumps([('2017-07-24', 149.7619), ('2017-07-25', 150.4019),....])
No need for OrderedDict in this case

If you want to convert a Python Dictionary to JSON using the json.dumps() method.
`
import json
from decimal import Decimal
d = {}
d["date"] = "2017-07-24"
d["quantity"] = "149.7619"
print json.dumps(d, ensure_ascii=False)
`

I removed the OrderedDict but kept all the other data, I think I understand the request. See if this works for you:
import json
my_dict = ([('2017-07-24', 149.7619), ('2017-07-25', 150.4019), ('2017-07-26', 151.1109)])
print(json.dumps(my_dict, indent=4, sort_keys=True))

Need help in Python for JSON data scraping

I am using the below code for scraping data from a website. but I am facing key error: 0
Kindly tell me the problems in my code.
Original JSON response from the webpage:
https://www.demo.com/api/user_details/22
Response:
{"user_details":{"user_id":"22","username":"Test","user_email":"test#gmail.com"}}
I wanna scrape the username, user_id and user_email.
What I have tried:
import json
import requests
import datetime
#data outputs to a CSV file in the current directory
csv_output = open("test.csv", "w")
end_page = 5;
#scan through pages 1 to end_page for data, 20 results per page
for page in range(1,end_page+1):
r = requests.get('https://www.demo.com/api/user_details/' + str(page))
data = r.json()
for index in range(len(data["user_details"])):
csv_output.write("\"%s\",%s\n" % (data["user_details"][index]["user_id"].encode('ascii', 'ignore'))),
data["user_details"][index]["user_id"]
csv_output.close()

data["user_details"] is a dict and not a list and you are getting the error because you are trying to access the values using an index:
data["user_details"][index] ....
You can get the entries by accessing specific keys from the dict:
user_id = data["user_details"]['user_id']
username = data["user_details"]['username']
user_email = data["user_details"]['user_email']

Exactly what AKS asnwered, but I really recommend you to use a framework called Scrapy tocreate crawlers. Much easier. :)

{"user_details":{"user_id":"22","username":"Test","user_email":"test#gmail.com"}}
User details is a dictionary here. On the other hand, index is an integer coming from the range call. The first value would be 0. Your code tries to load data["user_details"][0]. But there is no key 0 in that dictionary.
To iterate over a dictionary, you can call the items method which would give you a tuple with (key, value) pair.
d = {"user_id":"22","username":"Test","user_email":"test#gmail.com"}
for k,v in d.items():
print("Key: {}, Value: {}".format(k,v))

Urlencode dictionary using Python - naming key and value in the url

I am attempting to generate a URL link in the following format using urllib and urlencode.
<img src=page.psp?KEY=%28SpecA%2CSpecB%29&VALUE=1&KEY=%28SpecA%2C%28SpecB%2CSpecC%29%29&VALUE=2>
I'm trying to use data from my dictionary to input into the urllib.urlencode() function however, I need to get it into a format where the keys and values have a variable name, like below. So the keys from my dictionary will = NODE and values will = VALUE.
wanted = urllib.urlencode( [("KEY",v1),("VALUE",v2)] )
req.write( "<a href=page.psp?%s>" % (s) );
The problem I am having is that I want the URL as above and instead I am getting what is below, rather than KEY=(SpecA,SpecB) NODE=1, KEY=(SpecA,SpecB,SpecC) NODE=2 which is what I want.
KEY=%28SpecA%2CSpecB%29%2C%28%28SpecA%2CSpecB%29%2CSpecC%29&VALUE=1%2C2
So far I have extracted keys and values from the dictionary, extracted into tuples, lists, strings and also tried dict.items() but it hasn't helped much as I still can't get it to go into the format I want. Also I am doing this using Python server pages which is why I keep having to print things as a string due to constant string errors. This is part of what I have so far:
k = (str(dict))
ver1 = dict.keys()
ver2 = dict.values()
new = urllib.urlencode(function)
f = urllib.urlopen("page.psp?%s" % new)
I am wondering what I need to change in terms of extracting values from the dictionary/converting them to different formats in order to get the output I want? Any help would be appreciated and I can add more of my code (as messy as it has become) if need be. Thanks.

This should give you the format you want:
data = {
'(SpecA,SpecB)': 1,
'(SpecA,SpecB,SpecC)': 2,
}
params = []
for k,v in data.iteritems():
params.append(('KEY', k))
params.append(('VALUE', v))
new = urllib.urlencode(params)
Note that the KEY/VALUE pairings may not be the order you want, given that dicts are unordered.