Python - How to flatten JSON message in pandas/JSON - python

I have a JSON message as below
JSON_MSG = {
'Header': {
'Timestamp': '2020-10-25T02:49:25.489Z',
'ID': '0422',
'msgName': 'Order',
'Source': 'OrderSys'
},
'CustomerOrderLine': [
{
**'Parameter'**: [
{'ParameterID': 'ACTIVATION_DATE', 'ParameterValue': '2020-10-25'},
{'ParameterID': 'CYCLES', 'ParameterValue': '1'},
{'ParameterID': 'EXPIRY_PERIOD', 'ParameterValue': '30'},
{'ParameterID': 'MAX_NUMBER', 'ParameterValue': '1'}
],
'Subscription': {
'Sub': '3020611',
'LoanAcc': '',
'CustomerAcc': '2020002',
'SubscriptionCreatedDate': '2020-06-23T14:42:30Z',
'BillingAcc': '40010101',
'SubscriptionContractTerm': '12',
'ServiceAcc': '11111',
'SubscriptionStatus': 'Active'
},
'PaymentOpt': 'Upfront',
'OneTimeAmt': '8.0',
'RecurringAmt': '0.0'
'BeneficiaryID': '',
'CustomerOrderID': '111',
'OrderLineCreatedDate': '2020-10-25T02:47:18Z',
'ProductOfferingPriceId': 'PP_6GB_Booster',
'ParentCustomerOrderLineID': '',
'OrderLineRequestedDate': '2020-10-25T00:00:00.000Z',
'ProductCategoryId': 'PRODUCT_OFFER',
'OrderLinePurposeName': 'ADD',
'OrderQuantity': '1.0',
'CustomerOrderLineID': '11111',
'OrderLineDeliveryAddress': {
'OrderLineDeliveryPostCode': '',
'OrderLineDeliveryTown': '',
'OrderLineDeliveryCounty': '',
'OrderLineDeliveryCountryName': ''
},
'ProductInstanceID': '95',
'ProductOfferingId': 'OFF_6GBBOOST_MONTHLY'
}
]
}
I need to flatten the JSON message and convert it into rows and capture the row count/record count
(or)
I need to find out how many elements are present under the nested array Parameter
as this would give me same result as that of flattened JSON(because Parameter is the innermost array)
So far i have tried the below code
data = json.loads(JSON_MSG)
list1 = data['CustomerOrderLine']
rec_count = len(list1)
but this code gives the outer list's result only i.e. 1
as CustomerOrderLine contains only one structs
I need the record/row count as 4 (Parameter array has 4 structs)

Not the prettiest one, but you could try something like:
list1 = JSON_MSG['CustomerOrderLine'][0]['Parameter']

To get 'Parameter' sizes of all elements, you can use list comprehension:
data = json.loads(JSON_MSG)
sizes = [len(order.get('Parameter')) for order in data.get('CustomerOrderLine', [])]

Related

How do I replace single quotes with double quotes in a python array without replacing the single quotes in the json values?

I have an array called managable:
r = requests.get("https://discord.com/api/v8/users/#me/guilds", headers = {
"Authorization": f"Bearer {access_token}"
})
guilds = r.json()
managable = []
for guild in guilds:
if int(guild["permissions"]) & 32 != 0:
managable.append(guild)
where I replace some boolean values in it:
strmanagable = str(managable).replace("True", '"true"').replace("False", '"false"').replace("None", '"none"')
and it returns an array like this:
[{'id': '0', 'name': '\'something\''}, {'id': '1', 'name': '\'two\''}]
I would like to replace the single quotes with double quotes in the array above, without replacing the single quotes in the json values.
I tried using the replace function (strmanagable.replace("'", "\"")), but it replaces single quotes in the json values too, which I don't want.
#snakecharmerb solved my question, I just had to convert managable to json
Are you look for this function json.dumps()? It converts the list into a string literal, then True becomes 'true' automatically
import json
lis1 = [{'id': '0', 'name': True}, {'id': '1', 'name': False}, {'id': '2', 'name': 'two'}]
lis1 = json.dumps(lis1)
Output
'[{"id": "0", "name": true}, {"id": "1", "name": false}, {"id": "2", "name": "two"}]'
Then if you need to convert it back, do this
lis2 = json.loads(lis1)
print(lis2)
[{'id': '0', 'name': True},
{'id': '1', 'name': False},
{'id': '2', 'name': 'two'}]

How to iterate over a JSON array and get values for a key which itself is a JSON object

I have been trying to do something simple yet something hard for me to solve it!
I have a json object that looks like:
jsonObject = {
'attributes': {
'192': { <--- This can be changed times to times meaning different number
'id': '192',
'code': 'hello',
'label': 'world',
'options': [
{
'id': '211',
'label': '5'
},
{
'id': '1202',
'label': '8.5'
},
{
'id': '54',
'label': '9'
},
{
'id': '1203',
'label': '9.5'
},
{
'id': '58',
'label': '10'
}
]
}
},
'template': '12345',
'basePrice': '51233',
'oldPrice': '51212',
'productId': 'hello',
}
and what I want to do is to get the values from options (To have both id and label saved into a list)
For now I only managed to do:
for att, value in jsonObject.items():
print(f"{att} - {value}"
How can I get the label and id?
You can try the following code:
attr = jsonObject['attributes']
temp = list(attr.values())[0] # It is same as "temp = attr['192']", but you said '192' can be changed.
options = temp['options']
for option in options:
print(f"id: {option['id']}, label: {option['label']}")

inserting the new json data in the already existing json list

am trying to insert the json data into another json data list data using update function
datasets:
data = {'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}
maindata = [{'school': {'govt':{ 'value':'public1', 'item':'zphs1'}, 'convent':{'value':'private1','item':'HPS1'}}},
{'school': {'govt':{ 'value':'public2', 'item':'zphs2'}, 'convent':{'value':'private2','item':'HPS2'}}}]
Trying to fit in the data into main data after the tag, 'school'
expected output:
maindata = [{'school': {'govt':{ 'value':'public1', 'item':'zphs1'}, 'convent':{'value':'private1','item':'HPS1'}, 'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}},
{'school': {'govt':{ 'value':'public2', 'item':'zphs2'}, 'convent':{'value':'private2','item':'HPS2'},'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}}]
code used:
for i in maindata:
i['school'].update(data)
which returns None
The dictionary update method is not expected to return anything. It performs an in-place update that mutates the target dictionary. To see what this returns, try looking at the content of maindata after the update has run.
Your code is correct: the maindata dictionary is being updated. After making the following update:
for i in maindata:
i['school'].update(data)
check the value of maindata, it should be exactly what you're expecting it to be. You can also verify this by doing the following:
desired_output = [
{'school': {'govt':{ 'value':'public1', 'item':'zphs1'}, 'convent':{'value':'private1','item':'HPS1'}, 'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}},
{'school': {'govt':{ 'value':'public2', 'item':'zphs2'}, 'convent' {'value':'private2','item':'HPS2'},'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}}
]
for i in maindata:
i['school'].update(data)
maindata == desired_output
That works for me.

How to compare json file with expected result in Python 3?

I need to prepare test which will be comparing content of .json file with expected result (we want to check if values in .json are correctly generated by our dev tool).
For test I will use robot framework or unittests but I don't know yet how to parse correctly json file.
Json example:
{
"Customer": [{
"Information": [{
"Country": "",
"Form": ""
}
],
"Id": "110",
"Res": "",
"Role": "Test",
"Limit": ["100"]
}]
}
So after I execute this:
with open('test_json.json') as f:
hd = json.load(f)
I get dict 'hd' where key is:
dict_keys(['Customer'])
and values:
dict_values([[{'Information': [{'Form': '', 'Country': ''}], 'Role': 'Test', 'Id': '110', 'Res': '', 'Limit': ['100']}]])
My problem is that I don't know how to get to only one value from Dict(e.g: Role: Test), because I can get only extract whole value. I can prepare a long string to compare with but it is not best solution for tests.
Any ideas how I can get to only one row from .json file?
Your JSON has single key 'Customer' and it has a value of list type. So when you ppass dict_keys(['Customer']) you are getting list value.
>>> hd['Customer']
[{'Id': '110', 'Role': 'Test', 'Res': '', 'Information': [{'Form': '', 'Country': ''}], 'Limit': ['100']}]
First element in list:
>>> hd['Customer'][0]
{'Id': '110', 'Role': 'Test', 'Res': '', 'Information': [{'Form': '', 'Country': ''}], 'Limit': ['100']}
Now access inside dict structure using:
>>> hd['Customer'][0]['Role']
'Test'
You can compare the dict that you loaded (say hd) to the expected results dict (say expected_dict) by running
hd.items() == expected_dict.items()

How to retrieve values from list: python

Summary: I have a list
list = [
{
'id': '1',
'elements': 'A',
'table': 'maps/partials/a.html',
'chart': 'maps/partials/charts/a.html',
},
{
'id': '2',
'elements': 'B',
'table': 'maps/partials/census/b.html',
'chart': 'maps/partials/charts/b.html',
},
{
'id': '3',
'elements': ('C','D','E','F'), //i believe it is wrong
'table': 'maps/partials/census/common.html',
'chart': 'maps/partials/charts/common.html',
},]
some_arr = ['E','2011','English','Total']
I want to compare elements with items in some_arr.
and i am doing this to get list and compare it with some_arr.
for data in list:
for i in range(len(some_arr)):
if data['elements'] == some_arr[i]:
print(data['id'])
As you can see in 'id':3 section_title has 4 values. How do i compare elements here with some_arr.
If you want to find any matches:
lst = [
{
'id': '1',
'elements': 'A',
'table': 'maps/partials/a.html',
'chart': 'maps/partials/charts/a.html',
},
{
'id': '2',
'elements': 'B',
'table': 'maps/partials/census/b.html',
'chart': 'maps/partials/charts/b.html',
},
{
'id': '3',
'elements': ('C','D','E','F'),
'table': 'maps/partials/census/common.html',
'chart': 'maps/partials/charts/common.html',
}]
some_arr = ['E','2011','English','Total']
st = set(some_arr)
from collections import Iterable
for d in lst:
val = d["elements"]
if isinstance(val, Iterable) and not isinstance(val, str):
if any(ele in st for ele in val):
print(d["id"])
else:
if val in st:
print(d["id"])
Making a set of all the elements in some_arr will give you O(1) lookups, using isinstance(val, Iterable) and not isinstance(val, str) will catch any iterable value like a list, tuple etc.. and avoid iterating over a string which could give you false positives as "F" is in "Foo".
any will short circuit on the first match so if you actually want to print id for every match then use a for loop. Lastly if you are using python2 use basestring instead of str.
You can use set.intersection to get the intersection between them but before you need to check that if it's a tuple then use intersection.
Also in the first case you can use in to check the membership to check if data['elememnt'] in in some_arr then print data['id']:
for data in list:
d=data['elements']
if isinstance(d,tuple):
if set(d).intersection(some_arr):
print(data['id'])
if d in some_arr:
print(data['id'])

Categories

Resources