How can I use pyjq to convert incoming String into integer?
As of now, pyjq returns a string for x attribute, and I understand why it's doing that, but isn't there any function that I can add to the schema in order to convert it into integer?
Code:
# apply incoming data to template
def apply(template_name: str, data):
result = pyjq.one(
import_template(template_name),
data
)
Incoming event data:
"dynamodb": {
"NewImage": {
"x": {
"N": "3"
},
Schema template:
"event": {
"data": {
"x": .dynamodb.NewImage.x.N,
}
}
Output:
"event": {
"data": {
"x": "3",
}
}
Looking for a solution to get output like below:
Output:
"event": {
"data": {
"x": 3,
}
}
jq can convert a string to an number using the tonumber filter.
Change the template to:
"event": {
"data": {
"x": .dynamodb.NewImage.x.N | tonumber
}
}
Related
I have two nested JSON objects with different order of elements and would like to generate hash value for both. Now, I'm comparing these two hash values and it needs to be same. How can I achieve this mechanism?
First JSON Object
{
"X":{
"Y":[
{
"A":"1",
"B":{
"b1":"2",
"b2":"2"
}
},
{
"C":"4",
"D":{
"d1":"5",
"d2":"6"
}
},
],
"Z":[
{
"E":{
"e1":"7",
"e2":"8"
},
"F":"9"
}
]
}
}
Second JSON Object
{
"X":{
"Y":[
{
"C":"4",
"D":{
"d1":"5",
"d2":"6"
}
},
{
"A":"1",
"B":{
"b1":"2",
"b2":"2"
}
},
],
"Z":[
{
"E":{
"e1":"7",
"e2":"8"
},
"F":"9"
}
]
}
}
So, here goal is I want to generate same hash value for both JSON object. How can I achieve this in Python or Golang?
The JSON object is unsorted as well the map[string]interface{}, so you should sort the maps. However, sorting could be complicated and time consuming.
Instead of using the objects I would like to use the JSON as string, this way the JSON can be Unmarshaled and the string can be sorted to create the SHA value.
var json1 = `
{
"X": {
"Y": [
{
"A": "1",
"B": {
"b1": "2",
"b2": "2"
}
},
{
"C": "4",
"D": {
"d1": "5",
"d2": "6"
}
}
],
"Z": [
{
"E": {
"e1": "7",
"e2": "8"
},
"F": "9"
}
]
}
}
`
var json2 = `
{
"X": {
"Y": [
{
"C": "4",
"D": {
"d1": "5",
"d2": "6"
}
},
{
"A": "1",
"B": {
"b1": "2",
"b2": "2"
}
}
],
"Z": [
{
"E": {
"e1": "7",
"e2": "8"
},
"F": "9"
}
]
}
}
`
type JSON struct {
X struct {
Y []struct {
A string `json:"A"`
B struct {
B1 string `json:"b1"`
B2 string `json:"b2"`
} `json:"B"`
C string `json:"C"`
D struct {
D1 string `json:"d1"`
D2 string `json:"d2"`
} `json:"D"`
} `json:"Y"`
Z []struct {
E struct {
E1 string `json:"e1"`
E2 string `json:"e2"`
} `json:"E"`
F string `json:"F"`
} `json:"Z"`
} `json:"X"`
}
func SortString(w string) string {
s := strings.Split(w, "")
sort.Strings(s)
return strings.Join(s, "")
}
func main() {
var v1, v2 interface{}
json.Unmarshal([]byte(json1), &v1)
json.Unmarshal([]byte(json2), &v2)
fmt.Println(reflect.DeepEqual(v1, v2))
var m1, m2 JSON
json.Unmarshal([]byte(json1), &m1)
json.Unmarshal([]byte(json2), &m2)
fmt.Println(reflect.DeepEqual(m1, m2))
json1 = SortString(json1)
json2 = SortString(json2)
fmt.Println(reflect.DeepEqual(json1, json2))
}
Keep in mind that the objects are unsorted, so you should evaluate if creating a JSONSort function is important (considering that JSON could be different each time).
go run json.go
false
false
true
I have a csv file with a DF with structure as follows:
my dataframe:
I want to enter the data to the following JSON format using python. I looked to couple of links (but I got lost in the nested part). The links I checked:
How to convert pandas dataframe to uniquely structured nested json
convert dataframe to nested json
"PHI": 2,
"firstname": "john",
"medicalHistory": {
"allergies": "egg",
"event": {
"inPatient":{
"hospitalized": {
"visit" : "7-20-20",
"noofdays": "5",
"test": {
"modality": "xray"
}
"vitalSign": {
"temperature": "32",
"heartRate": "80"
},
"patientcondition": {
"headache": "1",
"cough": "0"
}
},
"icu": {
"visit" : "",
"noofdays": "",
},
},
"outpatient": {
"visit":"5-20-20",
"vitalSign": {
"temperature": "32",
"heartRate": "80"
},
"patientcondition": {
"headache": "1",
"cough": "1"
},
"test": {
"modality": "blood"
}
}
}
}
If anyone can help me with the nested array, that will be really helpful.
You need one or more helper functions to unpack the data in the table like this. Write main helper function to accept two arguments: 1. df and 2. schema. The schema will be used to unpack the df into a nested structure for each row in the df. The schema below is an example of how to achieve this for a subset of the logic you describe. Although not exactly what you specified in example, should be enough of hint for you to complete the rest of the task on your own.
from operator import itemgetter
groupby_idx = ['PHI', 'firstName']
groups = df.groupby(groupby_idx, as_index=False, drop=False)
schema = {
"event": {
"eventType": itemgetter('event'),
"visit": itemgetter('visit'),
"noOfDays": itemgetter('noofdays'),
"test": {
"modality": itemgetter('test')
},
"vitalSign": {
"temperature": itemgetter('temperature'),
"heartRate": itemgetter('heartRate')
},
"patientCondition": {
"headache": itemgetter('headache'),
"cough": itemgetter('cough')
}
}
}
def unpack(obj, schema):
tmp = {}
for k, v in schema.items():
if isinstance(v, (dict,)):
tmp[k] = unpack(obj, v)
if callable(v):
tmp[k] = v(obj)
return tmp
def apply_unpack(groups, schema):
results = {}
for gidx, df in groups:
events = []
for ridx, obj in df.iterrows():
d = unpack(obj, schema)
events.append(d)
results[gidx] = events
return results
unpacked = apply_unpack(groups, schema)
I have the following dictionary in python which I'm saving into a file:
d2 = {
"CHARACTER": {
"IDENTITY": {
"FORM": {
"id": "BK1",
"type": "MAGE",
"role": "DARK"
}
},
"USER": {
"owner": {
"id": "SABBATH13"
},
"level": "16"
}
}
}
jsonfile = open('d2.json', 'w')
jsonfile.write(simplejson.dumps(d2, indent=4))
jsonfile.close()
However, I'm told this is a JSON object, which I need to turn into a JSON array of the form:
[{
"CHARACTER": {
"IDENTITY": {
"FORM": {
"id": "BK1",
"type": "MAGE",
"role": "DARK"
}
},
"USER": {
"owner": {
"id": "SABBATH13"
},
"level": "16"
}
}
}]
Which is essentially adding square brackets at the beginning and end.
What is the proper way to do this? Should I convert to string and add brackets, then convert back? Sorry, total JSON newbie here.
You're thinking at the wrong level of abstraction. It's not about the brackets, it's about that you have a data structure which is an object, when what you apparently need is a list/array of objects (even if there's just one object in the list). So:
d2 = [d2]
Now dumps this and you get what you need.
I would like to make the following JSON syntax output with python:
data={
"timestamp": "1462868427",
"sites": [
{
"name": "SiteA",
"zone": 1
},
{
"name": "SiteB",
"zone": 7
}
]
}
But I cannot manage to get the 'outer' data key there.
So far I got this output without the data key:
{
"timestamp": "1462868427",
"sites": [
{
"name": "SiteA",
"zone": 1
},
{
"name": "SiteB",
"zone": 7
}
]
}
I have tried with this python code:
sites = [
{
"name":"nameA",
"zone":123
},
{
"name":"nameB",
"zone":324
}
]
data = {
"timestamp": 123456567,
"sites": sites
}
print(json.dumps(data, indent = 4))
But how do I manage to get the outer 'data' key there?
Once you have your data ready, you can simply do this :
data = {'data': data}
JSON doesn't have =, it's all key:value.
What you're looking for is
data = {
"data": {
"timestamp": 123456567,
"sites": sites
}
}
json.dumps(data)
json.dumps() doesn't care for the name you give to the data object in python. You have to specify it manually inside the object, as a string.
I have the following JSON string (from wikipedia http://en.wikipedia.org/wiki/JSON)
{
"name":"Product",
"properties":
{
"id":
{
"type":"number",
"description":"Product identifier",
"required":true
},
"name":
{
"type":"string",
"description":"Name of the product",
"required":true
},
"price":
{
"type":"number",
"minimum":0,
"required":true
},
"tags":
{
"type":"array",
"items":
{
"type":"string"
}
},
"stock":
{
"type":"object",
"properties":
{
"warehouse":
{
"type":"number"
},
"retail":
{
"type":"number"
}
}
}
}
}
I am trying to decode this string using Python json library. I would like to access the node
properties - > stock - > properties - > warehouse.
I understand that json.loads() function stores the json string as a dictionary. But in this case properties is my key and everything under that are values. How do I access the above node.
import json
jsonText=""
file = open("c:/dir/jsondec.json")
for line in file.xreadlines():
jsonText+=line
data = json.loads(jsonText)
for k,v in data.items():
print k // shows name and properties
file.close();
Thanks
You can load json straight from the file like this:
f = open("c:/dir/jsondec.json")
data = json.load(f)
Based on your input string, data is now a dictionary that contains other dictionaries. You can just navigate up the dictionaries like so:
node = data['properties']['stock']['properties']['warehouse']
print str(node)