Related
So, I have JSON data that I am getting from an HTML form. I have to format this data to have a specific JSON structure as shown below.
[
{
"Header": "Stats",
"Line_01": "Line 01",
"Line_02": "Line 02",
"Line_03": "Line 03",
"Line_04": "Line 04",
"Line_05": "Line 05",
"Line_06": "Line 06",
"Line_07": "Line 07"
},
{
"Header": "JUV",
"Line_01": "89",
"Line_02": "34",
"Line_03": "765",
"Line_04": "123",
"Line_05": "1",
"Line_06": "4",
"Line_07": "455"
},
{
"Header": "SMP",
"Line_01": "12",
"Line_02": "89",
"Line_03": "124",
"Line_04": "678",
"Line_05": "92",
"Line_06": "120",
"Line_07": "5"
}
]
JSON I have from the HTML form is:
[
{
"name": "00",
"value": "JUV"
},
{
"name": "00",
"value": "STATS"
},
{
"name": "00",
"value": "SMP"
},
{
"name": "00",
"value": "89"
},
{
"name": "01",
"value": "LINE 01"
},
{
"name": "02",
"value": "12"
},
{
"name": "03",
"value": "34"
},
{
"name": "04",
"value": "LINE 02"
},
{
"name": "05",
"value": "89"
},
{
"name": "06",
"value": "765"
},
{
"name": "07",
"value": "LINE 03"
},
{
"name": "08",
"value": "124"
},
{
"name": "09",
"value": "123"
},
{
"name": "10",
"value": "LINE 04"
},
{
"name": "11",
"value": "678"
},
{
"name": "12",
"value": "1"
},
{
"name": "13",
"value": "LINE 05"
},
{
"name": "14",
"value": "92"
},
{
"name": "15",
"value": "4"
},
{
"name": "16",
"value": "LINE 06"
},
{
"name": "17",
"value": "120"
},
{
"name": "18",
"value": "455"
},
{
"name": "19",
"value": "LINE 07"
},
{
"name": "20",
"value": "5"
}
]
The form looks like this: HTML form - Image on Pasteboard
The Python code I am trying so far is:
import json
jarr = []
final_file = {}
json_template = {
"Header": "",
"Line_01": "",
"Line_02": "",
"Line_03": "",
"Line_04": "",
"Line_05": "",
"Line_06": "",
"Line_07": ""
}
with open("testo101.json",) as f:
jdata = json.load(f)
k = 0
for i in range(8):
a =[]
for j in range(3):
a.append(jdata[k]['value'])
k+=1
jarr.append(a)
for i, x in enumerate(json_template):
json_template[x]=jarr[i][1]
final_file.update(json_template)
for i, x in enumerate(json_template):
json_template[x]=jarr[i][0]
final_file.update(json_template)
for i, x in enumerate(json_template):
json_template[x]=jarr[i][2]
final_file.update(json_template)
print(final_file)
So what I have done so far is:
Import the JSON file I got from HTML form.
Covert it into a 3x8 matrix... so I can get values of each column
easily.
I can fill values from the JSON in the json_template dictionary with
the help of 2D array I just created.
The Problem:
I can't figure out how to merge the 3 dictionaries that I am generating from each column of the 2D array into a single dictionary final_file so I can dump it as a JSON file and that's what I want. How can I do this... or if there is some other better way to do this then please help.
Thanks.
These code should do the job:
import json
jarr = []
final_file = [] # CHANGE #1
json_template = {
"Header": "",
"Line_01": "",
"Line_02": "",
"Line_03": "",
"Line_04": "",
"Line_05": "",
"Line_06": "",
"Line_07": ""
}
with open("testo101.json",) as f:
jdata = json.load(f)
k = 0
for i in range(8):
a =[]
for j in range(3):
a.append(jdata[k]['value'])
k+=1
jarr.append(a)
for i, x in enumerate(json_template):
json_template[x]=jarr[i][1]
final_file.append(json_template.copy()) # CHANGE #2
for i, x in enumerate(json_template):
json_template[x]=jarr[i][0]
final_file.append(json_template.copy()) # CHANGE #2
for i, x in enumerate(json_template):
json_template[x]=jarr[i][2]
final_file.append(json_template.copy()) # CHANGE #2
with open('yourjson.json', 'w') as jsonfile:
json.dump(final_file, jsonfile, indent=4)
I made two changes to your code:
You need a list of dictionaries, not just one dictionary to dump three dictionaries, so I changed final_file to a list.
After you make each json_template, I attached a copy of the template to the list. (.copy() is important, otherwise later changes will be reflected in the previous entries and you end up getting three of the same item).
I wrote the dumping code and attached it to the end. You can open yourjson.json to see the result.
I wrote a script to pull data from Verizon's connectivity management API with the following. Below is the section that requests the line information based on the search item, in this case, the SIM or iccid. I did not include the previous parts because they are just to connect to the API and get credentials.
header = {
'accept': 'application/json',
'VZ-M2M-Token': session_token,
'Authorization': 'Bearer' + bearer_token,
'Content-Type': 'application/json',
}
data = '{ "deviceId": { "id": ' + SIM +', "kind": "ICCID" }}'
response = requests.post('https://thingspace.verizon.com/api/m2m/v1/devices/actions/list', headers=header, data=data)
And the response I get is a JSON Array which looks like
{
"hasMoreData": false,
"devices": [
{
"accountName": "123456789-00001",
"billingCycleEndDate": "2020-10-31T20:00:00-04:00",
"carrierInformations": [
{
"carrierName": "Verizon Wireless",
"servicePlan": "3rrrrx48wwwwrjgjtyjtyjtyjtyj",
"state": "active"
}
],
"connected": true,
"createdAt": "2016-11-04T11:06:28-04:00",
"deviceIds": [
{
"id": "5256694405",
"kind": "mdn"
},
{
"id": "3114949302094150",
"kind": "imsi"
},
{
"id": "35922505468230",
"kind": "imei"
},
{
"id": "891480000054957290575",
"kind": "iccId"
},
{
"id": "15256694405",
"kind": "msisdn"
},
{
"id": "5256694405",
"kind": "min"
}
],
"extendedAttributes": [
{
"key": "PrimaryPlaceOfUseTitle"
},
{
"key": "PrimaryPlaceOfUseFirstName",
"value": "5256694405",
},
{
"key": "PrimaryPlaceOfUseMiddleName"
},
{
"key": "PrimaryPlaceOfUseLastName",
"value": "ESN"
},
{
"key": "PrimaryPlaceOfUseSuffix"
},
{
"key": "PrimaryPlaceOfUseAddressLine1"
},
{
"key": "PrimaryPlaceOfUseAddressLine2"
},
{
"key": "PrimaryPlaceOfUseCity"
},
{
"key": "PrimaryPlaceOfUseState"
},
{
"key": "PrimaryPlaceOfUseCountry"
},
{
"key": "PrimaryPlaceOfUseZipCode"
},
{
"key": "PrimaryPlaceOfUseZipCode4"
},
{
"key": "PrimaryPlaceOfUseCBRPhone"
},
{
"key": "PrimaryPlaceOfUseCBRPhoneType"
},
{
"key": "PrimaryPlaceOfUseEmailAddress"
},
{
"key": "AccountNumber",
"value": "5256694405-00001"
},
{
"key": "SmsrOid"
},
{
"key": "ProfileStatus"
},
{
"key": "PromoCodes",
"value": ""
},
{
"key": "PromotionStartDate",
"value": ""
},
{
"key": "PromotionScheduledEndDate",
"value": ""
},
{
"key": "LeadId",
"value": ""
},
{
"key": "CustomerName",
"value": ""
},
{
"key": "CustomerAddressLine1",
"value": ""
},
{
"key": "CustomerAddressLine2",
"value": ""
},
{
"key": "CustomerAddressCity",
"value": ""
},
{
"key": "CustomerAddressState",
"value": ""
},
{
"key": "CustomerAddressZipCode",
"value": ""
},
{
"key": "ServiceZipCode",
"value": ""
},
{
"key": "SkuNumber",
"value": "VZW080000460053"
},
{
"key": "CostCenterCode"
},
{
"key": "PreIMEI",
"value": "3592254564568445"
},
{
"key": "PreSKU",
"value": "VZW080000100037"
},
{
"key": "SIMOTADate",
"value": "4/30/2020 1:22:18 PM"
},
{
"key": "RoamingStatus",
"value": "NotRoaming"
},
{
"key": "LastRoamingStatusUpdate",
"value": "9/24/2020 5:40:26 PM"
}
],
"groupNames": [
"Default: 0220433754-00001"
],
"ipAddress": "100.100.100.100",
"lastActivationBy": "User Verizon",
"lastActivationDate": "2016-11-04T11:06:28-04:00",
"lastConnectionDate": "2020-09-24T13:40:26-04:00"
}
]
}
I added a part to my script to pull the mdn, iccid and the imei from the array with the code that is below.
def puller(line_json):
line_data = json.loads(line_json)
mdn = (line_data['devices'][0]['deviceIds'][0]['id'])
iccid = (line_data['devices'][0]['deviceIds'][3]['id'])
imei = (line_data['devices'][0]['deviceIds'][2]['id'])
print('phone = ' ,mdn)
print('SIM = ' , iccid)
print('IMEI = ' , imei)
I tested this code and it works the way it should with one test ID. I then proceeded to test with another test ID and I learned that the array structure is not always the same. That second JSON array is below. I am wondering is there a better way to find the specific values that I want, but in the way that I am not specifically telling the script where in the structure the item will be as I did above.
{
"hasMoreData": false,
"devices": [
{
"accountName": "02234234234-00001",
"billingCycleEndDate": "2020-10-31T20:00:00-04:00",
"carrierInformations": [
{
"carrierName": "Verizon Wireless",
"servicePlan": "37776xdsfewsfwe576193",
"state": "active"
}
],
"connected": true,
"createdAt": "2016-05-24T15:55:06-04:00",
"deviceIds": [
{
"id": "0945437676404",
"kind": "esn"
},
{
"id": "1234565799",
"kind": "mdn"
},
{
"id": "31148454545458767",
"kind": "imsi"
},
{
"id": "01426786678211",
"kind": "imei"
},
{
"id": "89148000006456456454",
"kind": "iccId"
},
{
"id": "1234565799",
"kind": "min"
}
],
"extendedAttributes": [
{
"key": "PrimaryPlaceOfUseTitle"
},
{
"key": "PrimaryPlaceOfUseFirstName",
"value": "096114564506772"
},
{
"key": "PrimaryPlaceOfUseMiddleName"
},
{
"key": "PrimaryPlaceOfUseLastName",
"value": "096546454806772"
},
{
"key": "PrimaryPlaceOfUseSuffix"
},
{
"key": "PrimaryPlaceOfUseAddressLine1"
},
{
"key": "PrimaryPlaceOfUseAddressLine2"
},
{
"key": "PrimaryPlaceOfUseCity"
},
{
"key": "PrimaryPlaceOfUseState"
},
{
"key": "PrimaryPlaceOfUseCountry"
},
{
"key": "PrimaryPlaceOfUseZipCode"
},
{
"key": "PrimaryPlaceOfUseZipCode4"
},
{
"key": "PrimaryPlaceOfUseCBRPhone"
},
{
"key": "PrimaryPlaceOfUseCBRPhoneType"
},
{
"key": "PrimaryPlaceOfUseEmailAddress"
},
{
"key": "AccountNumber",
"value": "02242342354-00001"
},
{
"key": "SmsrOid"
},
{
"key": "ProfileStatus"
},
{
"key": "PromoCodes",
"value": ""
},
{
"key": "PromotionStartDate",
"value": ""
},
{
"key": "PromotionScheduledEndDate",
"value": ""
},
{
"key": "LeadId",
"value": ""
},
{
"key": "CustomerName",
"value": ""
},
{
"key": "CustomerAddressLine1",
"value": ""
},
{
"key": "CustomerAddressLine2",
"value": ""
},
{
"key": "CustomerAddressCity",
"value": ""
},
{
"key": "CustomerAddressState",
"value": ""
},
{
"key": "CustomerAddressZipCode",
"value": ""
},
{
"key": "ServiceZipCode",
"value": ""
},
{
"key": "SkuNumber",
"value": "VZW12000364343005"
},
{
"key": "CostCenterCode"
},
{
"key": "PreIMEI"
},
{
"key": "PreSKU",
"value": "VZW12000334340005"
},
{
"key": "SIMOTADate",
"value": "3/13/2020 10:52:07 AM"
},
{
"key": "RoamingStatus",
"value": "NotRoaming"
},
{
"key": "LastRoamingStatusUpdate",
"value": "10/20/2020 6:14:20 PM"
}
],
"groupNames": [
"Default: 02342343754-00001"
],
"ipAddress": "101.101.101.101",
"lastActivationBy": "User Verizon",
"lastActivationDate": "2016-05-24T15:55:16-04:00",
"lastConnectionDate": "2020-10-20T14:14:20-04:00"
}
]
}
I tried to use this block of code from some research I did to find the value that I was looking for; in this case, the mdn. Problem I have is that the response returns a blank set of brackets with no information, so I know there is something I probably did wrong.
def json_extract(obj, kind):
"""Recursively fetch values from nested JSON."""
arr = []
def extract(obj, arr, kind):
"""Recursively search for values of key in JSON tree."""
if isinstance(obj, dict):
for k, v in obj.items():
if isinstance(v, (dict, list)):
extract(v, arr, kind)
elif k == kind:
arr.append(v)
elif isinstance(obj, list):
for item in obj:
extract(item, arr, kind)
return arr
values = extract(obj, arr, kind)
return values
names = json_extract(response , 'mdn')
print(names)
I understood that you are trying to find, mdn, iccid and imei'id from the json object above,so, instead of recursion and the complicated coding that you have done there, it is easier to use python's inbuilt libraries to help you out:
You can use the next function for your purpose:
# load your json data
line_data = json.loads(data)
# narrow your focus on the array in question
device_ids = line_data['devices'][0]['deviceIds']
# This gets the first item's id attribute from the list that matches the condition, and returns None if no item matches.
mdn = next((x['id'] for x in device_ids if x['kind'] == "mdn"), None)
iccid = next((x['id'] for x in device_ids if x['kind'] == "iccid"), None)
imei = next((x['id'] for x in device_ids if x['kind'] == "imei"), None)
You will need to handle the None if it was unable to find such element in the array.
Reference : Find object in list that has attribute equal to some value (that meets any condition)
I have started learning Python not long ago, and I'm trying to do recursive function for my list. This is the function:
def get_employee(j, field, value):
res = j
for x in res:
if x['Name'] == field and x['Value'] == value:
return res
elif "Properties" not in x:
if x is not None:
continue
elif "Properties" in x:
return get_employee(x['Properties'], field, value)
And this is my JSON:
[{
"Value": "Sales",
"Name": "Department",
"Properties": [{
"Value": "US",
"Name": "Country",
"Properties": [{
"Value": "Employee",
"Name": "EmployeeType",
"Properties": [{
"Value": "Manya Bishter",
"Name": "EmployeeFullName",
"Properties": [{
"Value": 1111,
"Name": "EmployeeID"
},
{
"Value": "Manya",
"Name": "EmployeeFirstName"
},
{
"Value": "Bishter",
"Name": "EmployeeLastName"
}
]
},
{
"Value": "Michael Ort",
"Name": "EmployeeFullName",
"Properties": [{
"Value": 1112,
"Name": "EmployeeID"
},
{
"Value": "Michael",
"Name": "EmployeeFirstName"
},
{
"Value": "Ort",
"Name": "EmployeeLastName"
}
]
}
]
},
{
"Value": "Manager",
"Name": "EmployeeType",
"Properties": [{
"Value": "Nick Fair",
"Name": "EmployeeFullName",
"Properties": [{
"Value": 1113,
"Name": "EmployeeID"
},
{
"Value": "Nick",
"Name": "EmployeeFirstName"
},
{
"Value": "Fair",
"Name": "EmployeeLastName"
}
]
}]
}
]
}]
},
{
"Value": "Marketing",
"Name": "Department",
"Properties": [{
"Value": "US",
"Name": "Country",
"Properties": [{
"Value": "Employee",
"Name": "EmployeeType",
"Properties": [{
"Value": "Tamta Hiresh",
"Name": "EmployeeFullName",
"Properties": [{
"Value": 1121,
"Name": "EmployeeID"
},
{
"Value": "Tamta",
"Name": "EmployeeFirstName"
},
{
"Value": "Hiresh",
"Name": "EmployeeLastName"
}
]
}]
}]
}]
}
]
The function work only on Manya, but nowhere else.
For example, if I do this:
print(get_employee(myjson, "EmployeeFirstName", "Nick"))
It will print:
[{'Value': 1111, 'Name': 'EmployeeID'}, {'Value': 'Manya', 'Name': 'EmployeeFirstName'}, {'Value': 'Bishter', 'Name': 'EmployeeLastName'}]
But for others (like Nick), it will return None.
Can you please help?
Thanks!
Here is the code:
def get_employee(data, field, value):
result = []
def recursor(j, field, value):
res = j
for x in res:
if x['Name'] == field and x['Value'] == value:
result.append(res)
elif "Properties" not in x:
if x is not None:
continue
elif "Properties" in x:
recursor(x['Properties'], field, value)
recursor(data,field,value)
return result
The problem with your recursive function is that, once it hits return get_employee(x['Properties'], field, value); it will stop the outer for loop for x in res: . Thus it will never run on the next item in your list.
{
"type": "Data",
"version": "1.0",
"box": {
"identifier": "abcdef",
"serial": "12345678"
},
"payload": {
"Type": "EL",
"Version": "1",
"Result": "Successful",
"Reference": null,
"Box": {
"Identifier": "abcdef",
"Serial": "12345678"
},
"Configuration": {
"EL": "1"
},
"vent": [
{
"ventType": "Arm",
"Timestamp": "2020-03-18T12:17:04+10:00",
"Parameters": [
{
"Name": "Arm",
"Value": "LT"
},
{
"Name": "Status",
"Value": "LD"
}
]
},
{
"ventType": "Arm",
"Timestamp": "2020-03-18T12:17:24+10:00",
"Parameters": [
{
"Name": "Arm",
"Value": "LT"
},
{
"Name": "Status",
"Value": "LD"
}
]
},
{
"EventType": "TimeUpdateCompleted",
"Timestamp": "2020-03-18T02:23:21.2979668Z",
"Parameters": [
{
"Name": "ActualAdjustment",
"Value": "PT0S"
},
{
"Name": "CorrectionOffset",
"Value": "PT0S"
},
{
"Name": "Latency",
"Value": "PT0.2423996S"
}
]
}
]
}
}
If you're looking to transfer information from a JSON file to a CSV, then you can use the following code to read in a JSON file into a dictionary in Python:
import json
with open('data.txt') as json_file:
data_dict = json.load(json_file)
You could then convert this dictionary into a list with either data_dict.items() or data_dict.values().
Then you just need to write this list to a CSV file which you can easily do by just looping through the list.
I am try to use python to extract value, but I found a weird result.
Following is part of my json variable
"weatherElement": [
{
"elementName": "ELEV",
"elementValue": {
"value": "20.0"
}
},
{
"elementName": "TEMP",
"elementValue": {
"value": "25.0"
}
},
{
"elementName": "D_TNT",
"elementValue": {
"value": "2019-11-22T02:10:00+08:00"
}
}
],
and following code is correct for getting value 25.0 which is temperature
for unit in data['records']['location']:
# print(type(unit)) <---- output <dict>
if unit['stationId'] == 'C0V490':
for wea_unit in unit['weatherElement']: # unit['weatherElement'] is list
if wea_unit['elementName'] == 'TEMP':
print(type(wea_unit['elementValue'])) # is str
return wea_unit['elementValue']
My question is why type(wea_unit['elementValue']) is str?
I think it should be dict and I should use wea_unit['elementValue']['value'] to get '25.0', but it is wrong. Are there anyone know what mistake I made? Thanks!
edit:
following is example code which can run directly
import json
def parse_json_to_dataframe(data):
for unit in data['records']['location']:
# print(type(unit))
if unit['stationId'] == 'C0A560':
for wea_unit in unit['weatherElement']: # unit['weatherElement'] is list
if wea_unit['elementName'] == 'TEMP':
print(type(wea_unit['elementValue']))
return wea_unit['elementValue']
v = {"success":"true",
"result": {"resource_id": "O-A0001-001",
"fields": [{"id": "lat", "type": "Double"},
{"id": "lon", "type": "Double"},
{"id": "locationName", "type": "String"},
{"id": "stationId", "type": "String"},
{"id": "description", "type": "String"},
{"id": "elementName", "type": "String"},
{"id": "elementValue", "type": "Double"},
{"id": "parameterName", "type": "String"},
{"id": "parameterValue", "type": "String"}]}, # result end
"records": {"location": [{"lat": "24.778333",
"lon": "121.494583",
"locationName": "福山",
"stationId": "C0A560",
"time": {"obsTime": "2019-11-22 22:00:00"},
"weatherElement": [
{"elementName": "ELEV", "elementValue": "405.0"},
{"elementName": "WDIR", "elementValue": "0"},
{"elementName": "WDSD", "elementValue": "0.0"},
{"elementName": "TEMP", "elementValue": "19.6"}],
"parameter": [
{"parameterName": "CITY_SN", "parameterValue": "06"},
{"parameterName": "TOWN_SN", "parameterValue": "061"}]}]}}
temp = parse_json_to_dataframe(v)
print(temp)