Extract certain information from json into csv - python

I have a json file that look like this:
{
"issueInfo" : [ {
"cid" : 494960,
"occurrences" : [ {
"file" : "/components/applications/diag/_common/src/diag_il.c",
"function" : "diag_il_u8StopLoopbackMicIn",
"mainEventLineNumber" : 6018,
"mainEventDescription" : "Assigning value \"10\" to \"u8ResData\" here, but that stored value is overwritten before it can be used.",
} ],
"triage" : {
"classification" : "Unclassified"
},
}
I want to extract out the information like cid, firstDetectedDateTime, file, function, mainEventLineNumber, mainEventDescription and classification. All of these information needed will be put into a csv file. The following is my coding:
import csv
import json
with open ("a.log","r") as file:
data=json.load(file)
f=csv.writer(open("test.csv", "w", newline=''))
f.writerow(["cid", "firstDetectedDateTime", "file", "function",
"mainEventLineNumber", "mainEventDescription", "classification"])
for data in file:
f.writerow(data["issueInfo"]["cid"],
data["issueInfo"]["firstDetectedDateTime"],
data["issueInfo"]["occurrences"]["file"],
data["issueInfo"]["occurrences"]["function"],
data["issueInfo"]["occurrences"]["mainEventLineNumber"],
data["issueInfo"]["occurrences"]["mainEventDescription"],
data["issueInfo"]["triage"]["classification"])
The error shown after I run the command is :
TypeError: string indices must be integers
Anyone can help me to solve this problem? Thanks

Check the type of data (It must be a dictionary). Also, there is an invalid key error firstDetectedDateTime.
Try this,
import csv
import json
with open ("a.log","r") as file:
data=json.load(file)
f=csv.writer(open("test.csv", "w", newline=''))
f.writerow(["cid", "firstDetectedDateTime", "file", "function","mainEventLineNumber","mainEventDescription", "classification"])
f.writerow([data["issueInfo"][0]["cid"],
"",
data["issueInfo"][0]["occurrences"][0]["file"],
data["issueInfo"][0]["occurrences"][0]["function"],
data["issueInfo"][0]["occurrences"][0]["mainEventLineNumber"],
data["issueInfo"][0]["occurrences"][0]["mainEventDescription"],
data["issueInfo"][0]["triage"]["classification"]])
Output CSV looks like,
cid,firstDetectedDateTime,file,function,mainEventLineNumber,mainEventDescription,classification
494960,,/components/applications/diag/_common/src/diag_il.c,diag_il_u8StopLoopbackMicIn,6018,"Assigning value ""10"" to ""u8ResData"" here, but that stored value is overwritten before it can be used.",Unclassified
If the page contains many JSON sets eg:data_sets here, Keep the headers fixed only change the portion below that.
for data in data_sets:
f.writerow([data["issueInfo"][0]["cid"],
"",
data["issueInfo"][0]["occurrences"][0]["file"],
data["issueInfo"][0]["occurrences"][0]["function"],
data["issueInfo"][0]["occurrences"][0]["mainEventLineNumber"],
data["issueInfo"][0]["occurrences"][0]["mainEventDescription"],
data["issueInfo"][0]["triage"]["classification"]])

The json library in python can parse JSON from strings or files. The library parses JSON into a Python dictionary or list
json.loads() function parses the json string data and it can be used as a normal dictionary in python. And we can access the values using keys.
import json
import csv
employee_data = '{"employee_details":[{"employee_name": "James", "email": "james#gmail.com", "job_profile": "Sr. Developer"},{"employee_name": "Smith", "email": "Smith#gmail.com", "job_profile": "Project Lead"}]}'
employee_parsed = json.loads(employee_data)
emp_data = employee_parsed['employee_details']
# open a file for writing
employ_data = open('..../EmployData.csv', 'w')
# create the csv writer object
csvwriter = csv.writer(employ_data)
count = 0
for emp in emp_data:
if count == 0:
header = emp.keys()
csvwriter.writerow(header)
count += 1
csvwriter.writerow(emp.values())
employ_data.close()

Related

How to append a dictionary to a json file in Python?

I want to read a json file in Python and add content to the file respecting the json format, but I can't manage to do it.
The json file "paises.json" :
{
"España": [
{
"Superficie": 505944,
"Población": 47450795
}
],
"Francia": [
{
"Superficie": 675417,
"Población": 67407241
}
]
}
I want to read that file in Python and append new content to it. The code is:
import json
with open("paises.json", "r", encoding="utf-8") as paises:
datos = json.load(paises)
print(datos.keys()) # Show dictionary
nombre_pais = input("\nIndique el nombre del país\n")
nueva_superficie = float(input("\nIndique la superficie\n"))
nueva_poblacion = int(input("\nIndique la Población\n"))
nuevo_contenido = {nombre_pais: {"Superficie": nueva_superficie, "Población": nueva_poblacion}}
datos.update(nuevo_contenido)
print(datos) # Show dictionary with new content
with open("paises.json", "a", encoding="utf-8") as paises:
json.dump(nuevo_contenido, paises)
The result is not what I have expected:
{
"España": [
{
"Superficie": 505944,
"Población": 47450795
}
],
"Francia": [
{
"Superficie": 675417,
"Población": 67407241
}
]
}
{"Portugal": {"Superficie": 92090, "Poblaci\u00f3n": 10295909}}
The formatting is not correct, a comma and a square bracket are missing and the coding of the accents is not ok.
What can I do to correct it?
Thank you
You can't append new data directly to JSON file because it creates incorrect JSON.
(Exception can be when you want to create multi-JSON file)
With normal JSON you have to:
read all data from file to memory,
append new values in memory,
write back all data (datos) from memory to file in "write mode", not "append mode".
with open("paises.json", "w", encoding="utf-8") as paises:
json.dump(datos, paises)

How to extract a single Key value from a JSON file?

This is my output.json file:
{
"ParsedResults": [
{
"TextOverlay": {
"Lines": [],
"HasOverlay": false,
"Message": "Text overlay is not provided as it is not requested"
},
"TextOrientation": "0",
"FileParseExitCode": 1,
"ParsedText": "180 Grade IV\r\n\u0103\u021ar: Class VIII Pass\r\nwww.facebook.com, Since 2012\r\n",
"ErrorMessage": "",
"ErrorDetails": ""
}
],
"OCRExitCode": 1,
"IsErroredOnProcessing": false,
"ProcessingTimeInMilliseconds": "343",
"SearchablePDFURL": "Searchable PDF not generated as it was not requested."
}
I am trying to get the ParsedText value from this JSON file.
This is my the code I am using:
import json
f = open('output.json',)
data = json.load(f)
print(data['ParsedResults']['TextOverlay']['ParsedText'])
f.close()
Facing this error:
TypeError: list indices must be integers or slices, not str
How to read that particular value from ParsedText, please guide. Thanks in Advance
ParsedResults is not an object, it's a list
try this:
import json
f = open('output.json',)
data = json.load(f)
print(data['ParsedResults'][0]['ParsedText'])
f.close()
data['ParsedResults'] is a list, you need to use the index to parse, So you are getting TypeError: list indices must be integers or slices, not str
use data['ParsedResults'][0].
use the following,
import json
f = open('output.json',)
data = json.load(f)
print(data['ParsedResults'][0]['TextOverlay']['ParsedText'])
f.close()

How to print from updated JSON file?

I want to print out some values from my json file and here is part of the code:
{
"records" : [
{
"dateRep" : "02/05/2021",
"day" : "02",
"month" : "05",
"year" : "2021",
"cases" : 1655,
"deaths" : 16,
"countriesAndTerritories" : "Austria",
"geoId" : "AT",
"countryterritoryCode" : "AUT",
"popData2020" : "8901064",
"continentExp" : "Europe"
},
How can i print dateRep ,cases, deaths or any value i want and put it in a variable to use it?
I am using this code to load my json file:
f = open('DailyUpdatedReport.json',)
data = json.load(f)
print(data)
My other issue : My json file need to be up-to dated I don't know how to do it. Here is URL for the json file I am using
https://opendata.ecdc.europa.eu/covid19/nationalcasedeath_eueea_daily_ei/json/
You would do this just how you would get data from any other dictionary in python. I'll put an example below, and you should be able to generalize from there. In response to your second question, I don't understand what you mean by needing to update your data. Couldn't you just copy and paste the data from the link you posted into a json?
import json
f = open('DailyUpdatedReport.json',)
data = json.load(f)
#['records']: enter records entry in data
#[0]: go into the first record
#['dateRep']: read a value from that record (in this case dateRep)
dateRep = data['records'][0]['dateRep']
print(dateRep)

How can I use jsonpath in python to change an element value in the json object

I have the following json object (Say car_details.json):
{
"name":"John",
"age":30,
"cars":
[
{
"car_model": "Mustang",
"car_brand": "Ford"
},
{
"car_model": "cx-5",
"car_brand": "Mazda"
}
}
I want to change the value of car_model from cx-5 to cx-9 through python code.
I am providing the json path to this element, through an external file. The json-path expression is basically represented as a string. Something like this:
'cars[2].car_model'
And the new value is also provided through an external file as a string:
'cx-9'
Now how do I parse through car_details.json using the jsonpath expression, and change its value to the one provided as string, and finally return the modified json object
P.S I want to do this through python code
This is an approach without using json module. Load your data in variable. Then iterate over cars key/values. If you find the key that is the value you are looking for set it to new value.
Also note: you need to close your array block, otherwise your above json is not valid. Generally I use an online json parser to check if my data is valid etc. (may be helpful in future).
data = {
"name":"John",
"age":30,
"cars":
[
{
"car_model": "Mustang",
"car_brand": "Ford"
},
{
"car_model": "cx-5",
"car_brand": "Mazda"
}
]
}
for cars in data['cars']:
for key, value in cars.items():
if key == "car_model" and value == "cx-5":
cars[key] = "cx-9"
print(data)
If you want to load your json object from a file, let's assume it is called "data.json" and is in the same directory as the python script you are going to run:
import json
with open('data.json') as json_data:
data = json.load(json_data)
for cars in data['cars']:
for key, value in cars.items():
if key == "car_model" and value == "cx-5":
cars[key] = "cx-9"
print(data)
Now if you'd like to write the content to the original file or new file, in this case I am writing to a file called "newdata.json":
import json
import re
with open('data.json') as json_data:
data = json.load(json_data)
print(data)
with open('external.txt') as f:
content = f.read()
print(content)
for cars in data['cars']:
for key, value in cars.items():
if key == "car_model" and value == "cx-5":
cars[key] = content
with open('newdata.json', 'w') as outfile:
json.dump(data, outfile)

not able to convert complete json into proper csv using python

I am using python to convert complex json into csv
I want to convert complex json into csv using python
sample json :
{
"data":[
{
"filter":"tags",
"cost":1234,
"values":[
{
"count":13,
"subvalues":[
{
"count":1,
"subvalue":"goat",
"cost":227.576
},
"storage":[
{
"resource_id":"1234343413",
"cost":25.047,
"running_hours %":61.52777777777778,
"created_in":"asd ",
"account_name":null,
"service":"csdsae",
etcc...
I have written the code as below:
my python code :
import json
import csv
f = open('file.json')
data = json.load(f)
s=csv.writer(open('test6.csv','w'))
count = 0
for item in data["breakdown"]:
s.writerow([item])
if count == 0:
header = item.keys()
s.writerow(header)
count += 1
s.writerow(item.values())
But i am not getting proper output in csv file getting only one line but the complete json is not coming

Categories

Resources