How to iterate through dictionary shape .txt file to create DataFrame - python

I have .txt file with a dictionary inside, like (several rows):
{
"data": [
{
"title": "Greatest chess game",
"created_time": "2020-02-17T16:51:44+0000"
}
]
}
I need to open this file and create df look alike:
title created_time
0 Greatest chess game 2020-02-17T16:51:44+0000
....
When I open txt file:
output_file=open('data\\data_new.txt', 'w')
with open(output_file, 'r') as reader:
print(reader.readline(5))
It treats as a:
TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper
How could I convert it ?

Not sure why you are opening the file twice, once in read mode and once in write, not sure that's actually possible.
Anyway, as mentioned what you appear to have is JSON so use the json library to read the file.
import json
import pandas as pd
with open('data_new.txt', 'r', encoding='utf-8') as reader:
json = json.load(reader )
df = pd.DataFrame(json['data'])
print(df)

I think you could use json lib to open your text file as json file, like:
import json
with open('data.txt') as json_file:
data = json.load(json_file)
#here you can make you dataframe

Related

How do I read in a txt file as a dictionary in Python?

I have a txt file that is of the following format (dictionary format):
{'EX1':'No Information Stored',
'EX2':'Foundation',
'EX3':'Foundation',
'EX4':'No Information Stored'}
Does anyone know how I would go about reading this into python to be able to use it like the dictionary that it is?
import json
with open('file.txt', 'r') as w:
data = w.read()
data_as_dict = json.loads(data)
Text file with this structure are JSON, so you can use the json module.
import json
def load_file(filename):
with open(filename) as f:
data = json.load(f)
return data
This is a custom function that return the dictionary you want.
Using the ast.literal_eval().
It can be used for conversion of other data types as well
# importing the module
import ast
# reading the data from the file
with open('dictionary.txt') as f:
data = f.read()
# reconstructing the data as a dictionary
d = ast.literal_eval(data)
print(d)
There are two methods for this,
1. Method using json.load():
.load() use to get from directly form file
import json
with open('data.json') as f:
json.load(f)
2. Method using json.loads():
.loads() use to get from string. So we need to read the file first to get string.
import json
with open('data.json') as f:
json.loads(f.read())

Is there a way to iterate over a JSON file and write specific keys and values to a new JSON file?

I'm trying to iterate over specific keys and values from a JSON file and write them to a new JSON file. I'm not getting any errors and the second JSON file is being created successfully, though the data passed onto the second JSON file is a whole object of the first JSON file.
def get_keys_values():
json_file = open("get_failed_archives.json")
json_data = json.load(json_file)
for archive_data in json_data["data"]:
archive_data["latestEvent"]["objectName"]
archive_data["latestEvent"]["time"]
archive_data["latestEvent"]["eventType"]
archive_data["latestEvent"]["eventStatus"]
with open("new_json_file.json", "w") as file:
json.dump(archive_data, file)
Is there a way to write specific keys and values from one JSON file to another JSON file?
as stated by josh, the statement archive_data["latestEvent"]["objectName"] functionally does nothing.
if you want to create a new JSON object, in your case this would functionally be a list of dict items (I think), the approach below would match what (I think) you want it to do.
I've added a pprint statement to print out what the result is.
import json
from pprint import pprint
def get_keys_values():
with open("get_failed_archives.json") as json_file:
json_data = json.load(json_file)
new_data = []
for archive_data in json_data["data"]:
new_data.append({
"objectName": archive_data["latestEvent"]["objectName"],
"time": archive_data["latestEvent"]["time"],
"eventType": archive_data["latestEvent"]["eventType"],
"eventStatus": archive_data["latestEvent"]["eventStatus"]
})
pprint(new_data, indent=4)
with open("new_json_file.json", "w") as file:
json.dump(new_data, file)

Add one more attribute to JSON file

So I have a generated JSON which one looks like ( there is a lot of it just with unique ID )
{
"id": 1,
"name": "name",
"dep": "dep",
"Title": "Title',
"email": "email"
}
I'm trying to do "append" a new field but I get an error
with open('data.json', 'w') as file:
json.dump(write_list, file)
file.close()
with open('data.json', 'w') as json_file:
entry = {'parentId': random.randrange(0, 487, 2)}
json_file.append(entry, json_file)
json_file.close()
It there is some way to add one more "key: value" to it after generating ?
There are two issues:
your are using json.dump to generate a list, but you're not using json.load to re-create the Python data structure.
You're opening the file with the w mode in the second open call, which truncates it
Try breaking each step out into its own and separating mutating data and writing to disk.
with open('data.json', 'w') as file:
json.dump(write_list, file)
#file.close() # manually closing files is unnecessary,
# when using context managers
with open('data.json', 'r') as json_file:
write_list = json.load(json_file)
entry = {'parentId': random.randrange(0, 487, 2)}
write_list.append(entry)
with open('data.json', 'w') as json_file:
json.dump(write_list, file)
The steps to do what you want are as files:
Parse the entire JSON file into a python dictionary.
Add the entry to the dictionary.
Serialize the string back to JSON.
Write the JSON file back to the file.
Also after I done this feature with Tim McNamara advice, I found a more pretty way to add new line to every JSON dict I have in file
for randomID in write_list:
randomID['parentId'] = random.randrange(0, 487, 2)
with open('data.json', 'w') as file:
json.dump(write_list, file)

How to add a key-value to JSON data retrieved from a file?

I am new to Python and I am playing with JSON data. I would like to retrieve the JSON data from a file and add to that data a JSON key-value "on the fly".
That is, my json_file contains JSON data as-like the following:
{"key1": {"key1A": ["value1", "value2"], "key1B": {"key1B1": "value3"}}}
I would like to add the "ADDED_KEY": "ADDED_VALUE" key-value part to the above data so to use the following JSON in my script:
{"ADDED_KEY": "ADDED_VALUE", "key1": {"key1A": ["value1", "value2"], "key1B": {"key1B1": "value3"}}}
I am trying to write something as-like the following in order to accomplish the above:
import json
json_data = open(json_file)
json_decoded = json.load(json_data)
# What I have to make here?!
json_data.close()
Your json_decoded object is a Python dictionary; you can simply add your key to that, then re-encode and rewrite the file:
import json
with open(json_file) as json_file:
json_decoded = json.load(json_file)
json_decoded['ADDED_KEY'] = 'ADDED_VALUE'
with open(json_file, 'w') as json_file:
json.dump(json_decoded, json_file)
I used the open file objects as context managers here (with the with statement) so Python automatically closes the file when done.
Json returned from json.loads() behave just like native python lists/dictionaries:
import json
with open("your_json_file.txt", 'r') as f:
data = json.loads(f.read()) #data becomes a dictionary
#do things with data here
data['ADDED_KEY'] = 'ADDED_VALUE'
#and then just write the data back on the file
with open("your_json_file.txt", 'w') as f:
f.write(json.dumps(data, sort_keys=True, indent=4, separators=(',', ': ')))
#I added some options for pretty printing, play around with them!
For more info check out the official doc
You can do
json_decoded['ADDED_KEY'] = 'ADDED_VALUE'
OR
json_decoded.update({"ADDED_KEY":"ADDED_VALUE"})
which works nicely if you want to add more than one key/value pair.
Of course, you may want to check for the existence of ADDED_KEY first - depends on your needs.
AND I assume you want might want to save that data back to the file
json.dump(json_decoded, open(json_file,'w'))

How to parse values from a JSON file in Python

I'm trying to get the values from the json file and the error that I'm getting is TypeError: expected string or buffer. I'm parsing the file correctly and moreover I guess my json file format is also correct. Where I'm going wrong?
Both the files are in the same directory.
Main_file.py
import json
json_data = open('meters_parameters.json')
data = json.loads(json_data) // TypeError: expected string or buffer
print data
json_data.close()
meters_parameters.json
{
"cilantro" : [{
"cem_093":[{
"kwh":{"function":"3","address":"286","length":"2"},
"serial_number":{"function":"3","address":"298","length":"2"},
"slave_id":{"function":"3","address":"15","length":"2"}
}]
}],
"elmeasure" : [{
"lg1119_d":[{
"kwh":{"function":"3","address":"286","length":"2"},
"serial_number":{"function":"3","address":"298","length":"2"},
"slave_id":{"function":"3","address":"15","length":"2"}
}]
}]
}
loads expects a string not a file handle. You need json.load:
import json
with open('meters_parameters.json') as f:
data = json.load(f)
print data
You're trying to load the file object, when you want to load everything in the file. Do:
data = json.loads(json_data.read())
.read() gets everything from the file and returns it as a string.
A with statement is much more pythonic here as well:
with open('meters_parameters.json') as myfile:
data = json.loads(myfile.read())

Categories

Resources