I am writing a python lambda function that reads in a json file from s3 and then will take one of the nodes and send it to another lambda function. Here is my code:
The json snippet I want
"jobstreams": [
{
"jobname": "team-summary",
"bucket": "aaa-bbb",
"key": "team-summary.json"
}
step 1 – convert JSON to python objects for processing
note: these I got from another Stack Overflow guru - thanks!!
def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values())
def json2obj(data): return json.loads(data, object_hook=_json_object_hook)
routes = json2obj(jsonText)
step 2 - I then traverse the python objects and find the json I need and dump it
for jobstream in jobstreams:
x = json.dumps(jobstream, ensure_ascii=False)
Howeever, when I print it out, I only have the values not the attributes. Why is that?
print(json.dumps(jobstream, ensure_ascii=False))
yields
["team-summary", "aaa-bbb", "team-summary.json"]
I'm assuming your full json file looks somewhat like what I have in my example
import json
js = {"jobstreams": [
{
"jobname": "team-summary",
"bucket": "aaa-bbb",
"key": "team-summary.json"
},
{
"jobname": "team-2222",
"bucket": "aaa-2222",
"key": "team-222.json"
}
]}
def extract_by_jobname(jobname):
for d in js['jobstreams']:
if d['jobname'] == jobname:
return d
json.dumps(extract_by_jobname("team-summary"))
# '{"jobname": "team-summary", "bucket": "aaa-bbb", "key": "team-summary.json"}'
I ended up creating a new Dictionary from the list that the json.dumps gave me.
["team-summary", "aaa-bbb", "team-summary.json"]
once i had the new dictionary (that is flat), then i converted that to json.... probably not the most efficient approach but i have other fish to fry. THANKS to all for your help!
Related
Here I set a json object inside a key in a redis. Later I want to perform search on the json file stored in the redis. My search key will always be a json string like in the example below and i want to match this inside the stored json file.
Currently here i am doing this by iterating and comparing but instead i want to do it with redis. How can I do it ?
rd = redis.StrictRedis(host="localhost",port=6379, db=0)
if not rd.get("mykey"):
with open(os.path.join(BASE_DIR, "my_file.josn")) as fl:
data = json.load(fl)
rd.set("mykey", json.dumps(data))
else:
key_values = json.loads(rd.get("mykey"))
search_json_key = {
"key":"value",
"key2": {
"key": "val"
}
}
# here i am searching by iterating and comparing instead i want to do it with redis
for i in key_values['all_data']:
if json.dumps(i) == json.dumps(search_json_key):
# return
# mykey format looks like this:
{
"all_data": [
{
"key":"value",
"key2": {
"key": "val"
}
},
{
"key":"value",
"key2": {
"key": "val"
}
},
{
"key":"value",
"key2": {
"key": "val"
}
},
]
}
To do search with Redis and JSON you have two options - you can use the FT CREATE command to create an index that you can then use FT SEARCH over, (while both of these web pages show the CLI syntax you can do
rd.ft().create() / search() in your python script)
OR you can check out the python OM client that will take care of that to some extent for you.
Either way you'll have to do a bit of a rework to fully take advantage of Redis' search capabilities.
I am trying to modify a JSON with Python but I can not do it correctly.
I have tried the module that comes with Python by default to deal with JSON but I can not do the next step.
The JSON to modify is this:
{
"uuid":"789ce6ed-ec0f-418b-8fad-6ba64cb8bd70",
"assetTemplate":[
{
"id":14,
"name":"Template-conectividad"
},
{
"id":54,
"name":"Template-discos-agata"
},
{
"id":17,
"name":"Template-servidor-linux"
}
],
"info":null
}
And it should look like this:
{
"uuid":"789ce6ed-ec0f-418b-8fad-6ba64cb8bd70",
"assetTemplate":[
{
"id":54,
"name":"Template-discos-agata"
},
{
"id":17,
"name":"Template-servidor-linux"
},
{
"id":85,
"name":"Template-conectividad-test"
}
],
"info":null
}
This is what I tried to remove the part I do not want but I have the part to insert the new data:
#!/usr/bin/python
import json
# We load JSON to modify
x = '{"uuid":"789ce6ed-ec0f-418b-8fad-6ba64cb8bd70","assetTemplate":[{"id":14,"name":"Template-conectividad"},{"id":54,"name":"Template-discos-agata"},{"id":17,"name":"Template-servidor-linux"}],"info":null}'
y = json.loads(x)
obj = y["assetTemplate"]
# We remove the object that we dont want
for i in range(len(obj)):
if obj[i]['id'] == 14:
del obj[i]
break
print(obj)
# We make output of what has been achieved
x = json.dumps(y)
print(x)
When you load a json, its contents are loaded as dictionaries ({} with contents as key:value) and lists ([]).
That means obj is a normal list - which you already kinda know because you iterate over it.
Because it's a normal list, you can just .append what you want as a dictionary, so:
d={"id":85,"name":"Template-conectividad-test"}
obj.append(d)
I have A json data like this:
json_data = '{"data":"[{"Date":"3/17/2017","Steam Total":60},{"Date":"3/18/2017","Steam Total":15},{"Date":"3/19/2017","Steam Total":1578},{"Date":"3/20/2017","Steam Total":1604}]", "data_details": "{"data_key":"Steam Total", "given_high":"1500", "given_low":"1000", "running_info": []}"}'
json_input_data = json_data["data"]
json_input_additional_info = json_data["data_details"]
I am getting an error:
TypeError: string indices must be integers, not str
I think there is an error in the json data. Can someone Help me on this?
In you code has some issues.
The code: json_input_data = json_data["data"], the variable json_data is not a Json Object, is a String Object and you try get a string position by string key, for get a Json object from string json use json api: json
You Json string isn't valid, this is a valid version:
{"data":[{"Date":"3/17/2017","Steam Total":60},{"Date":"3/18/2017","Steam Total":15},{"Date":"3/19/2017","Steam Total":1578},{"Date":"3/20/2017","Steam Total":1604}], "data_details": {"data_key":"Steam Total", "given_high":"1500", "given_low":"1000", "running_info": []}}
Now, your code works fine.
Try parsing your json_data to JSON format (with JSON.parse(json_data)). Currently it's type is string - which is exactly what your error says.
As Pongpira Upra pointed out, your json is not well formed and should be something like this.
{
"data":[
{
"Date":"3/17/2017",
"Steam Total":60
},
{
"Date":"3/18/2017",
"Steam Total":15
},
{
"Date":"3/19/2017",
"Steam Total":1578
},
{
"Date":"3/20/2017",
"Steam Total":1604
}
],
"data_details":{
"data_key":"Steam Total",
"given_high":"1500",
"given_low":"1000",
"running_info":[]
}
}
In order to retrieve information you should write
json_data[0]["Date"]
This would print "3/17/2017"
You declare a string called json_data and, well, then it acts like a string. That is what the exception tells you. Like others here tried to say - you do also have an error in your data, but the exception you supplied is due to accessing the string as if it was a dictionary. You need to add a missing call to e.g. json.loads(...).
You were right. Your JSON is indeed wrong.
Can you try using this json?
{
"data":[
{
"Date":"3/17/2017",
"Steam Total":60
},
{
"Date":"3/18/2017",
"Steam Total":15
},
{
"Date":"3/19/2017",
"Steam Total":1578
},
{
"Date":"3/20/2017",
"Steam Total":1604
}
],
"data_details":{
"data_key":"Steam Total",
"given_high":"1500",
"given_low":"1000",
"running_info":[]
}
}
I'm pretty new to Python, so just working my way through understanding the data sets.
I'm having a little trouble producing the JSON output that is required for the API I am working with.
I am using
import json
json.load(data_file)
Working with Python dictionary and then doing
json.dump(dict, json_data)
My data needs to look like the following when it is output.
{
"event":{
"id":10006,
"event_name":"My Event Name",
},
"sub event":[
],
"attendees":[
{
"id":11201,
"first_name":"Jeff",
"last_name":"Smith",
},
{
"id":10002,
"first_name":"Victoria",
"last_name":"Baker",
},
]
}
I have been able to create the arrays in python and dump to json, but I am having difficulty creating the event "object" in the dictionary. I am using the below:
attendees = ['attendees']
attendeesdict = {}
attendeesdict['first_name'] = "Jeff"
attendees.append(attendeesdict.copy())
Can anyone help me add the "event" object properly?
In general, going from JSON to dictionary is almost no work because the two are very similar, if not identical:
attendees = [
{
"first_name": "Jeff"
# Add other fields which you need here
},
{
"first_name": "Victoria"
}
]
In this instance, attendees is a list of dictionaries. For the event:
event = {
"id": 10006,
"event_name": "My Event Name"
}
Putting it all together:
data = {
"event": event,
"sub event": [],
"attendees": attendees
}
Now, you can convert it to a JSON object, ready to send to your API:
json_object = json.dumps(data)
Assuming you have built all the values elsewhere and now you're just putting them together:
result = {'event':event_dict, 'sub event':subevent_list, 'attendees':attendees_list}
If you want just to statically create a nested dict, you can use a single literal. If you paste the JSON above into python code, you would get a valid dict literal.
Construct your dicts and add like below
{
"event":"add your dict"
"sub event":["add your dict"],
"attendees":["add your dict"]
}
I have the same json data in two forms - one is a one-liner, the second one is just formatted output.
JSON A:
{"id":1, "name":"BoxH", "readOnly":true, "children":[{ "id":100, "name":"Box1", "readOnly":true, "children":[ { "id":1003, "name":"Box2", "children":[ { "id":1019, "name":"BoxDet", "Ids":[ "ABC", "ABC2", "DEF2", "DEFHD", "LKK" ]}]}]}]}
and JSON B:
{
"id":1,
"name":"BoxH",
"readOnly":true,
"children":[
{
"id":100,
"name":"Box1",
"readOnly":true,
"children":[
{
"id":1003,
"name":"Box2",
"children":[
{
"id":1019,
"name":"BoxDet",
"Ids":[
"ABC",
"ABC2",
"DEF2",
"DEFHD",
"LKK"
]
}
]
}
]
}
]
}
Why is it, that the code:
import json
if open('input_file.json'):
output_json = json.load('input_file.json')
in case A throws
ValueError: No JSON object could be decoded
and the case B works correctly. I'm just wondering why is it so? I thought that the JSON A and JSON B are the same for json.load. What should I do to get the both cases working?
json.load accept a file object (not file path). And you should keep the file reference. Try following:
import json
with open('input_file.json') as f:
output_json = json.load(f)
Alternatively you can use json.loads which accept serialized json string:
import json
with open('input_file.json') as f:
output_json = json.loads(f.read())
Acutally in my case there was problem with coding. As soon as I've converted the one-liner-file to UTF-8 without BOM, it started to working without any problems. The coding before was ANSI. So.. lesson learned: check the file coding.