Can you give some idea how to do this collection. The problem is this: I get the JSON to assume the following
[{
"pk": 1,
"model": "store.book",
"fields": {
"name": "Mostly Harmless",
"author": ["Douglas", "Adams"]
}
}]
then unzip a file I save the data and close the file, the next time (this is a cycle) again receive again like JSON, for example, the following
[{
"pk": 2,
"model": "store.book",
"fields": {
"name": "Henry",
"author": ["Hans"]
}
}]
the second JSON must go into the same file in which it is located and the first. Here comes the problem how to do. At this stage, I do it in the following way, delete the brackets and put commas.Is there any smarter and a better way for this job?
Creating JSON-Serializing Django objects of a use. I would be very grateful if you share their ideas.
PS: It is important to use minimal memory. Assume that the file is around 50-60 GB and in memory to hold about 1 GB maximum
You would have to convert your data into JSON and store it into a file. Then read from the file again and append your new data to the object and again save it into the file. Here is some code that might be useful for you:
Use JSON. Documentation available at - http://docs.python.org/2/library/json.html
The first time you write into the file, you can use something like:
>>> import json
>>> fileW = open("filename.txt","w")
>>> json1 = [{
... "pk": 1,
... "model": "store.book",
... "fields": {
... "name": "Mostly Harmless",
... "author": ["Douglas", "Adams"]
... }
... }]
>>> json.dump(json1, fileW)
>>> fileW.close()
The following code could be used in a loop to read from the file and add data to it.
>>> fileLoop = open("filename.txt","r+")
>>> jsonFromFile = json.load(fileLoop)
>>> jsonFromFile
[{u'pk': 1, u'model': u'store.book', u'fields': {u'name': u'Mostly Harmless', u'author': [u'Douglas', u'Adams']}}]
>>> newJson = [{
... "pk": 2,
... "model": "store.book",
... "fields": {
... "name": "Henry",
... "author": ["Hans"]
... }
... }]
>>> jsonFromFile.append(newJson[0])
>>> jsonFromFile
[{u'pk': 1, u'model': u'store.book', u'fields': {u'name': u'Mostly Harmless', u'author': [u'Douglas', u'Adams']}}, {'pk': 2, 'model': 'store.book', 'fields': {'name': 'Henry', 'author': ['Hans']}}]
>>> json.dump(jsonFromFile, fileLoop)
>>> fileLoop.close()
You do not need to parse the JSON because you are only storing it. The following (a) creates a file and (b) appends text to the file in each cycle.
from os.path import getsize
def init(filename):
"""
Creates a new file and sets its content to "[]".
"""
with open(filename, 'w') as f:
f.write("[]")
f.close()
def append(filename, text):
"""
Appends a JSON to a file that has been initialised with `init`.
"""
length = getsize(filename) #Figure out the number of characters in the file
with open(filename, 'r+') as f:
f.seek(length - 1) #Go to the end of the file
if length > 2: #Insert a delimiter if this is not the first JSON
f.write(",\n")
f.write(text[1:-1]) #Append the JSON
f.write("]") #Write a closing bracket
f.close()
filename = "temp.txt"
init(filename)
while mycondition:
append(filename, getjson())
If you did not have to save the JSON after each cycle, you could do the following
jsons = []
while mycondition:
jsons.append(getjson()[1:-1])
with open("temp.txt", "w") as f:
f.write("[")
f.write(",".join(jsons))
f.write("]")
f.close()
To avoid creating multigigabytes objects, you could store each object on a separate line. It requires you to dump each object without newlines used for formatting (json strings themselves may use \n (two chars) as usual):
import json
with open('output.txt', 'a') as file: # open the file in the append mode
json.dump(obj, file,
separators=',:') # the most compact representation by default
file.write("\n")
Related
I have a .json file with many entries looking like this:
{
"name": "abc",
"time": "20220607T190731.442",
"id": "123",
"relatedIds": [
{
"id": "456",
"source": "sourceA"
},
{
"id": "789",
"source": "sourceB"
}
],
}
I am saving each entry in a python object, however, I only need the related ID from source A. Problem is, the related ID from source A is not always first place in that nested list.
So data['relatedIds'][0]['id'] is not reliable to yield the right Id.
Currently I am solving the issue like this:
import json
with open("filepath", 'r') as file:
data = json.load(file)
for value in data['relatedIds']:
if(value['source'] == 'sourceA'):
id_from_a = value['id']
entry = Entry(data['name'], data['time'], data['id'], id_from_a)
I don't think this approach is the optimal solution though, especially if relatedIds list gets longer and more entries appended to the JSON file.
Is there a more sophisticated way of singling out this 'id' value from a specified source without looping through all entries in that nested list?
For a cleaner solution, you could try using python's filter() function with a simple lambda:
import json
with open("filepath", 'r') as file:
data = json.load(file)
filtered_data = filter(lambda a : a["source"] == "sourceA", data["relatedIds"])
id_from_a = next(filtered_data)['id']
entry = Entry(data['name'], data['time'], data['id'], id_from_a)
Correct me if I misunderstand how your json file looks, but it seems to work for me.
One step at a time, in order to get to all entries:
>>> data["relatedIds"]
[{'id': '789', 'source': 'sourceB'}, {'id': '456', 'source': 'sourceA'}]
Next, in order to get only those entries with source=sourceA:
>>> [e for e in data["relatedIds"] if e["source"] == "sourceA"]
[{'id': '456', 'source': 'sourceA'}]
Now, since you don't want the whole entry, but just the ID, we can go a little further:
>>> [e["id"] for e in data["relatedIds"] if e["source"] == "sourceA"]
['456']
From there, just grab the first ID:
>>> [e["id"] for e in data["relatedIds"] if e["source"] == "sourceA"][0]
'456'
Can you get whatever generates your .json file to produce the relatedIds as an object rather than a list?
{
"name": "abc",
"time": "20220607T190731.442",
"id": "123",
"relatedIds": {
"sourceA": "456",
"sourceB": "789"
}
}
If not, I'd say you're stuck looping through the list until you find what you're looking for.
I have a JSON file with 10000 data entries like below in a file.
{
"1":{
"name":"0",
"description":"",
"image":""
},
"2":{
"name":"1",
"description":"",
"image":""
},
...
}
I need to write each entry in this object into its own file.
For example, the output of each file looks like this:
1.json
{
"name": "",
"description": "",
"image": ""
}
I have the following code, but I'm not sure how to proceed from here. Can anyone help with this?
import json
with open('sample.json', 'r') as openfile:
# Reading from json file
json_object = json.load(openfile)
You can use a for loop to iterate over all the fields in the outer object, and then create a new file for each inner object:
import json
with open('sample.json', 'r') as input_file:
json_object = json.load(input_file)
for key, value in json_object.items():
with open(f'{key}.json', 'w') as output_file:
json.dump(value, output_file)
I have large file (about 3GB) which contains what looks like a JSON file but isn't because it lacks commas (,) between "observations" or JSON objects (I have about 2 million of these "objects" in my data file).
For example, this is what I have:
{
"_id": {
"$id": "fh37fc3huc3"
},
"messageid": "4757724838492485088139042828",
"attachments": [],
"usernameid": "47284592942",
"username": "Alex",
"server": "475774810304151552",
"text": "Must watch",
"type": "462050823720009729",
"datetime": "2018-08-05T21:20:20.486000+00:00",
"type": {
"$numberLong": "0"
}
}
{
"_id": {
"$id": "23453532dwq"
},
"messageid": "232534",
"attachments": [],
"usernameid": "273342",
"usernameid": "Alice",
"server": "475774810304151552",
"text": "https://www.youtube.com/",
"type": "4620508237200097wd29",
"datetime": "2018-08-05T21:20:11.803000+00:00",
"type": {
"$numberLong": "0"
}
And this is what I want (the comma between "observations"):
{
"_id": {
"$id": "fh37fc3huc3"
},
"messageid": "4757724838492485088139042828",
"attachments": [],
"username": "Alex",
"server": "475774810304151552",
"type": {
"$numberLong": "0"
}
},
{
"_id": {
"$id": "23453532dwq"
},
"messageid": "232534",
"attachments": [],
"usernameid": "Alice",
"server": "475774810304151552",
"type": {
"$numberLong": "0"
}
This is what I tried but it doesn't give me a comma where I need it:
import re
with open('dataframe.txt', 'r') as input, open('out.txt', 'w') as output:
output.write("[")
for line in input:
line = re.sub('', '},{', line)
output.write(' '+line)
output.write("]")
What can I do so that I can add a comma between each JSON object in my datafile?
This solution presupposes that none of the fields in JSON contains neither { nor }.
If we assume that there is at least one blank line between JSON dictionaries, an idea: let's maintain unclosed curly brackets count ({) as unclosed_count; and if we meet an empty line, we add the coma once.
Like this:
with open('test.json', 'r') as input_f, open('out.json', 'w') as output_f:
output_f.write("[")
unclosed_count = 0
comma_after_zero_added = True
for line in input_f:
unclosed_count_change = line.count('{') - line.count('}')
unclosed_count += unclosed_count_change
if unclosed_count_change != 0:
comma_after_zero_added = False
if line.strip() == '' and unclosed_count == 0 and not comma_after_zero_added:
output_f.write(",\n")
comma_after_zero_added = True
else:
output_f.write(line)
output_f.write("]")
Assuming sufficient memory, you can parse such a stream one object at a time using json.JSONDecoder.raw_decode directly, instead of using json.loads.
>>> x = '{"a": 1}\n{"b": 2}\n' # Hypothetical output of open("dataframe.txt").read()
>>> decoder = json.JSONDecoder()
>>> x = '{"a": 1}\n{"b":2}\n'
>>> decoder.raw_decode(x)
({'a': 1}, 8)
>>> decoder.raw_decode(x, 9)
({'b': 2}, 16)
The output of raw_decode is a tuple containing the first JSON value decoded and the position in the string where the remaining data starts. (Note that json.loads just creates an instance of JSONDecoder, and calls the decode method, which just calls raw_decode and artificially raises an exception if the entire input isn't consumed by the first decoded value.)
A little extra work is involved; note that you can't start decoding with whitespace, so you'll have to use the returned index to detect where the next value starts, following any additional whitespace at the returned index.
Another way to view your data is that you have multiple json records separated by whitespace. You can use the stdlib JSONDecoder to read each record, then strip whitespace and repeat til done. The decoder reads a record from a string and tells you how far it got. Apply that iteratively to the data until all is consumed. This is far less risky than making a bunch of assumptions about what data is contained in the json itself.
import json
def json_record_reader(filename):
with open(filename, encoding="utf-8") as f:
txt = f.read().lstrip()
decoder = json.JSONDecoder()
result = []
while txt:
data, pos = decoder.raw_decode(txt)
result.append(data)
txt = txt[pos:].lstrip()
return result
print(json_record_reader("data.json"))
Considering the size of your file, a memory mapped text file may be the better option.
If you're sure that the only place you will find a blank line is between two dicts, then you can go ahead with your current idea, after you fix its execution. For every line, check if it's empty. If it isn't, write it as-is. If it is, write a comma instead
with open('dataframe.txt', 'r') as input_file, open('out.txt', 'w') as output_file:
output_file.write("[")
for line in input_file:
if line.strip():
output_file.write(line)
else:
output_file.write(",")
output_file.write("]")
If you cannot guarantee that any blank line must be replaced by a comma, you need a different approach.
You want to replace a close-bracket, followed by an empty line (or multiple whitespace), followed by an open-bracket, with },{.
You can keep track of the previous two lines in addition to the current line, and if these are "}", "", and "{" in that order, then write a comma before writing the "{".
from collections import deque
with open('dataframe.txt', 'r') as input_file, open('out.txt', 'w') as output_file:
last_two_lines = deque(maxlen=2)
output_file.write("[")
for line in input_file:
line_s = line.strip()
if line_s == "{" and list(last_two_lines) == ["}", ""]:
output_file.write("," + line)
else:
output_file.write(line)
last_two_lines.append(line_s)
Alternatively, if you want to stick with regex, then you could do
with open('dataframe.txt') as input_file:
file_contents = input_file.read()
repl_contents = re.sub(r'\}(\s+)\{', r'},\1{', file_contents)
with open('out.txt', 'w') as output_file:
output_file.write(repl_contents)
Here, the regex r"\}(\s+)\{" matches the pattern we're looking for (\s+ matches multiple whitespace characters, and captures them in group 1, which we then use in the replacement string as \1.
Note that you will need to read and run re.sub on the entire file, which will be slow.
So I'm pulling data from an API and want to save only specific dicts and list from the JSON response. The problem is that when I dump the data inside the loop it creates very weird looking data in the file which isn't actually JSON.
r=requests.get(url,headers=header)
result=r.json()
with open ('myfile.json','a+') as file:
for log in result['logs']:
hello=json.dump(log['log']['driver']['username'], file)
hello=json.dump(log['log']['driver']['first_name'],file)
hello=json.dump(log['log']['driver']['last_name'],file)
for event in log['log']['events']:
hello=json.dump(event['event']['id'],file)
hello=json.dump(event['event']['start_time'],file)
hello=json.dump(event['event']['type'],file)
hello=json.dump(event['event']['location'],file)
The end goal here is to convert this data into a CSV. The only reason I'm saving it to a JSON file is so that I can load it and save it into a CSV then. The API endpoint I'm targeting is Logs:
https://developer.keeptruckin.com/reference#get-logs
I think #GBrandt has the right idea as far as creating valid JSON output goes, but as I said in a comment, I don't think that JSON-to-JSON conversion step is really necessary — since you could just create the CSV file from the JSON you already have:
(Modified to also split start_time into two separate fields as per you follow-on question.)
result = r.json()
with open('myfile.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
for log in result['logs']:
username = log['log']['driver']['username']
first_name = log['log']['driver']['first_name']
last_name = log['log']['driver']['last_name']
for event in log['log']['events']:
id = event['event']['id']
start_time = event['event']['start_time']
date, time = start_time.split('T') # Split time into two fields.
_type = event['event']['type'] # Avoid using name of built-in.
location = event['event']['location']
if not location:
location = "N/A"
writer.writerow(
(username, first_name, last_name, id, date, time, _type, location))
It looks like you're just dumping individual JSON strings into the file in an unstructured way.
json.dump will not magically create a JSON dict-like object and save it into the file. See:
json.dump(log['log']['driver']['username'], file)
What it actually does there is just stringifying the driver's username and dumping it right into the file, so the file will have only a string, not a JSON object (which I'm guessing is what you want). It is JSON, just not really useful.
What you're looking for is this:
r=requests.get(url,headers=header)
result=r.json()
with open ('myfile.json','w+') as file:
logs = []
for log in result['logs']:
logs.append({
'username': log['log']['driver']['username'],
'first_name': log['log']['driver']['first_name'],
'last_name': log['log']['driver']['last_name'],
# ...
'events': [
({
'id': event['event']['id'],
'start_time': event['event']['start_time'],
# ...
}) for event in log['log']['events']
]
})
json.dump(logs, file)
Also, I would recommend not using append mode on JSON files, a .json is expected to hold a single JSON object (as far as I'm concerned).
How about the code below (A sample json is loaded from a file instead of via HTTP call in order to get data to work with).
Sample JSON taken from https://developer.keeptruckin.com/reference#get-logs
import json
with open('input.json', 'r') as f_in:
data = json.load(f_in)
data_to_collect = []
logs = data['logs']
with open('output.json', 'w') as f_out:
for log in logs:
_log = log['log']
data_to_collect.append({key: _log['driver'].get(key) for key in ['username', 'first_name', 'last_name']})
data_to_collect[-1]['events'] = []
for event in _log['events']:
data_to_collect[-1]['events'].append(
{key: event['event'].get(key) for key in ['id', 'start_time', 'type', 'location']})
json.dump(data_to_collect, f_out)
Output file
[
{
"username": "demo_driver",
"first_name": "Demo",
"last_name": "Driver",
"events": [
{
"start_time": "2016-10-16T07:00:00Z",
"type": "driving",
"id": 221,
"location": "Mobile, AL"
},
{
"start_time": "2016-10-16T09:00:00Z",
"type": "sleeper",
"id": 474,
"location": null
},
{
"start_time": "2016-10-16T11:00:00Z",
"type": "driving",
"id": 475,
"location": null
}
]
}
]
I have a dict like this:
sample = {'ObjectInterpolator': 1629, 'PointInterpolator': 1675, 'RectangleInterpolator': 2042}
I can't figure out how to dump the dict to a JSON file as showed below:
{
"name": "interpolator",
"children": [
{"name": "ObjectInterpolator", "size": 1629},
{"name": "PointInterpolator", "size": 1675},
{"name": "RectangleInterpolator", "size": 2042}
]
}
Is there a pythonic way to do this?
You may guess that I want to generate a d3 treemap.
import json
with open('result.json', 'w') as fp:
json.dump(sample, fp)
This is an easier way to do it.
In the second line of code the file result.json gets created and opened as the variable fp.
In the third line your dict sample gets written into the result.json!
Combine the answer of #mgilson and #gnibbler, I found what I need was this:
d = {
"name": "interpolator",
"children": [{
'name': key,
"size": value
} for key, value in sample.items()]
}
j = json.dumps(d, indent=4)
with open('sample.json', 'w') as f:
print >> f, j
It this way, I got a pretty-print json file.
The tricks print >> f, j is found from here:
http://www.anthonydebarros.com/2012/03/11/generate-json-from-sql-using-python/
d = {"name":"interpolator",
"children":[{'name':key,"size":value} for key,value in sample.items()]}
json_string = json.dumps(d)
Since python 3.7 the ordering of dicts is retained https://docs.python.org/3.8/library/stdtypes.html#mapping-types-dict
Dictionaries preserve insertion order. Note that updating a key does not affect the order. Keys added after deletion are inserted at the end
Also wanted to add this (Python 3.7)
import json
with open("dict_to_json_textfile.txt", 'w') as fout:
json_dumps_str = json.dumps(a_dictionary, indent=4)
print(json_dumps_str, file=fout)
Update (11-04-2021): So the reason I added this example is because sometimes you can use the print() function to write to files, and this also shows how to use the indentation (unindented stuff is evil!!). However I have recently started learning about threading and some of my research has shown that the print() statement is not always thread-safe. So if you need threading you might want to be careful with this one.
This should give you a start
>>> import json
>>> print json.dumps([{'name': k, 'size': v} for k,v in sample.items()], indent=4)
[
{
"name": "PointInterpolator",
"size": 1675
},
{
"name": "ObjectInterpolator",
"size": 1629
},
{
"name": "RectangleInterpolator",
"size": 2042
}
]
with pretty-print format:
import json
with open(path_to_file, 'w') as file:
json_string = json.dumps(sample, default=lambda o: o.__dict__, sort_keys=True, indent=2)
file.write(json_string)
If you're using Path:
example_path = Path('/tmp/test.json')
example_dict = {'x': 24, 'y': 25}
json_str = json.dumps(example_dict, indent=4) + '\n'
example_path.write_text(json_str, encoding='utf-8')