Python can't load JSON created by json.dump - python

This seems to be a very straightforward problem where I am trying to read a JSON file and modify a few of the fields.
with open("example.json", "r+") as f:
data = json.load(f)
# perform modifications to data
f.truncate(0)
json.dump(data, f)
It worked the first time I manually created a JSON file and stored the correct JSON file, but by the second time I run the same script it gives me this error:
ValueError: No JSON object could be decoded
Why is that? It surprises me that the json module cannot parse the file created by the module itself.


Based on the code you provided, and what you are trying to do (return the file cursor back to the beginning of the file), you are not actually doing that with f.truncate. You are actually truncating your file. i.e. Clearing the file entirely.
Per the help on the truncate method:
truncate(...)
truncate([size]) -> None. Truncate the file to at most size bytes.
Size defaults to the current file position, as returned by tell().
What you are actually looking to do with returning the cursor back to the beginning of the file is using seek.
The help on seek:
seek(...)
seek(offset[, whence]) -> None. Move to new file position.
So, explicitly, to get back to the beginning of the file you want f.seek(0).
For the sake of providing an example of what is happening. Here is what happens with truncate:
File has stuff in it:
>>> with open('v.txt') as f:
... res = f.read()
...
>>> print(res)
1
2
3
4
Call truncate and see that file will now be empty:
>>> with open('v.txt', 'r+') as f:
... f.truncate(0)
...
0
>>> with open('v.txt', 'r') as f:
... res = f.read()
...
>>> print(res)
>>>
Using f.seek(0):
>>> with open('v.txt') as f:
... print(f.read())
... print(f.read())
... f.seek(0)
... print(f.read())
...
1
2
3
4
0
1
2
3
4
>>>
The long gap between the first output shows the cursor at the end of the file. Then we call the f.seek(0) (the 0 output is from the f.seek(0) call) and output our f.read().


You are missing one line, f.seek(0)
with open("example.json", "r+") as f:
data = json.load(f)
# perform modifications to data
f.seek(0);
f.truncate(0)
json.dump(data, f)

Related

Reading Two Files and Writing To One File Using Python3

I'm currently using Python 3 on Ubuntu 18.04. I'm not a programmer by any means and I'm not asking for a code review, however, I'm having an issue that I can't seem to resolve.
I have 1 text file named content.txt that I'm reading lines from.
I have 1 text file named standard.txt that I'm reading lines from.
I have 1text file named outfile.txt that I'm writing to.
content = open("content.txt", "r").readlines()
standard = open("standard.txt", "r").readlines()
outfile = "outfile.txt"
outfile_set = set()
with open(outfile, "w") as f:
for line in content:
if line not in standard:
outfile_set.add(line)
f.writelines(sorted(outfile_set))
I'm not sure where to put the following line though. My for loop nesting may all be off:
f.write("\nNo New Content")
Any code examples to make this work would be most appreciated. Thank you.

if i understand good you whant to add outfile_set if this is not empty to the outfile or add the string "\nNo New Content"
Replace the line
f.writelines(sorted(outfile_set))
to
if any(outfile_set):
f.writelines(sorted(outfile_set))
else:
f.write("\nNo New Content")

I'm assuming that you want to write "No new content" to the file if every line in content is in standard. So you might do something like:
with open(outfile, "w") as f:
for line in content:
if line not in standard:
outfile_set.add(line)
if len(outfile_set) > 0:
f.writelines(sorted(outfile_set))
else:
f.write("\nNo New Content")
Your original code was almost there!

You can reduce your runtime a lot by using set/frozenset:
with open("content.txt", "r") as f:
content = frozenset(f.readlines()) # only get distinct values from file
with open("standard.txt", "r") as f:
standard = frozenset(f.readlines()) # only get distinct values from file
# only keep whats in content but not in standard
outfile_set = sorted(content-standard) # set difference, no loops or tests needed
with open ("outfile.txt","w") as outfile:
if outfile_set:
outfile.writelines(sorted(outfile_set))
else:
outfile.write("\nNo New Content")
You can read more about it here:
set operator list (python 2 - but valid for 3 - can't find this overview in py3 doku
set difference
Demo:
# Create files
with open("content.txt", "w") as f:
for n in map(str,range(1,10)): # use range(1,10,2) for no changes
f.writelines(n+"\n")
with open("standard.txt", "w") as f:
for n in map(str,range(1,10,2)):
f.writelines(n+"\n")
# Process files:
with open("content.txt", "r") as f:
content = set(f.readlines())
with open("standard.txt", "r") as f:
standard = set(f.readlines())
# only keep whats in content but not in standard
outfile_set = sorted(content-standard)
with open ("outfile.txt","w") as outfile:
if outfile_set:
outfile.writelines(sorted(outfile_set))
else:
outfile.write("\nNo New Content")
with open ("outfile.txt") as f:
print(f.read())
Output:
2
4
6
8
or
No New Content

How to find byte at specific index from file?

I need to print byte at specific position in file that i know path. So I open default file in "rb" mode and then I need to know what byte is on 15 position. It is posible ?

Here's how you can achieve this with seek:
with open('my_file', 'rb') as f:
f.seek(15)
f.read(1)

Another way you could do this is to read the entire document and slice it:
First read the contense of the file:
file = open('test.txt', 'rb')
a = file.read()
Then take the desired value:
b = a[14]
Then don't forget to close the file afterwards:
file.close()
Or so that is closes automatically:
with open('test.txt', 'rb') as file:
a = file.read()
b = a[14]

Read and process a text file and save to csv

The files I have seem to be in a "dict" format...
file header is as follows: time,open,high,low,close,volume
next line is as follows:
{"t":[1494257340],"o":[206.7],"h":[209.3],"l":[204.50002],"c":[204.90001],"v":[49700650]}`
import csv
with open ('test_data.txt', 'rb') as f:
for line in f:
dict_file = eval(f.read())
time = (dict_file['t']) # print (time) result [1494257340]
open_price = (dict_file['o']) # print (open_price) result [206.7]
high = (dict_file['h']) # print (high) result [209.3]
low = (dict_file['l']) # print (low) result [204.50002]
close = (dict_file['c']) # print (close) result [204.90001]
volume = (dict_file['v']) # print (volume) result [49700650]
print (time, open_price, high, low, close, value)
# print result [1494257340] [206.7] [209.3] [204.50002] [204.90001] [49700650]
# I need to remove the [] from the output.
# expected result
# 1494257340, 206.7, 209.3, 204.50002, 204.90001, 49700650
the result I need is (change time ("epoch date format") to dd,mm,yy
5/8/17, 206.7, 209.3, 204.50002, 204.90001, 49700650
so I know I need the csv.writer function

I see a number of problems in the code you submitted. I recommend you to break your task into small pieces and see if you can make them work individually. So what are you trying to do is:
open a file
read the file line by line
eval each line to get a dict object
get values from that object
write those values in a (separate?) csv file
Right?
Now do each one, one small step at the time
opening a file.
You're pretty much on point there:
with open('test_data.txt', 'rb') as f:
print(f.read())
# b'{"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}\n'
You can open the file in r mode instead, it will give you strings instead of byte type objects
with open('test_data.txt', 'r') as f:
print(f.read())
# {"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}
It might cause some problems but should work since eval can handle it just fine (at least in python 3)
read the file line by line
with open('test_data.txt', 'rb') as f:
for line in f:
print(line)
# b'{"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}\n'
Here is another problem in your code, you're not using line variable and trying to f.read() instead. This will just read entire file (starting from the second line, since the first one is been read already). Try to swap one for another and see what happens
eval each line to get a dict object
Again. This works fine. but I would add some protection here. What if you get an empty line in the file or a misformatted one. Also if this file comes from an untrusted source you may become a victim of a code injection here, like if a line in your file changed to:
print("You've been hacked") or {"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}
with open('test_data.txt', 'rb') as f:
for line in f:
dict_file = eval(line)
print(dict_file)
# You've been hacked
# {'t': [1494257340], 'o': [207.75], 'h': [209.8], 'l': [205.75], 'c': [206.35], 'v': [61035956]}
I don't know your exact specifications, but you should be safer with json.loads instead.
...
Can you continue on your own from there?
get values from the object
I think dict_file['t'] doesn't give you the value you expect.
What does it give you?
Why?
How to fix it?
write those values in a csv file
Can you write some random string to a file?
What scv format looks like? Can you format your values to match it
Check the docs for csv module, can it be of help to you?
And so on and so forth...
EDIT: Solution
# you can save the print output in a file by running:
# $ python convert_to_csv.py > output.cvs
import datetime, decimal, json, os
CSV_HEADER = 'time,open,high,low,close,volume'
with open('test_data.txt', 'rb') as f:
print(CSV_HEADER)
for line in f:
data = json.loads(line, parse_float=decimal.Decimal)
data['t'][0] = datetime.datetime.fromtimestamp(data['t'][0]) \
.strftime('%#d/%#m/%y' if os.name == 'nt' else '%-d/%-m/%y')
print(','.join(str(data[k][0]) for k in 'tohlcv'))
Running:
$ cat test_data.txt
{"t":[1494257340],"o":[207.75],"h":[209.8],"l":[205.75],"c":[206.35],"v":[61035956]}
{"t":[1490123123],"o":[107.75],"h":[109.8],"l":[105.75],"c":[106.35],"v":[11035956]}
{"t":[1491234234],"o":[307.75],"h":[309.8],"l":[305.75],"c":[306.35],"v":[31035956]}
$ python convert_to_csv.py
time,open,high,low,close,volume
8/5/17,207.75,209.8,205.75,206.35,61035956
21/3/17,107.75,109.8,105.75,106.35,11035956
3/4/17,307.75,309.8,305.75,306.35,31035956

python read value from file and change it and write back to file

i am reading a value from a file and then adding up with another and then writing back to the same file.
total = 0
initial = 10
with open('file.txt', 'rb') as inp, open('file.txt', 'wb') as outp:
content = inp.read()
try:
total = int(content) + int(initial)
outp.write(str(total))
except ValueError:
print('{} is not a number!'.format(content))
it is successfully reading the value from file, but when writing, nothing is stored in the file.
what is wrong here?
update
I want to replace the old value, not append to it. Erase the old value and then put the new value instead.

you can't open your file twice simultaneously,
your code should look like this:
total = 0
initial = 10
with open('file.txt', 'rb') as inp:
content = inp.read()
total = int(content) + int(initial)
with open('file.txt', 'wb') as outp:
outp.write(str(total))
A look at this could help you:
Beginner Python: Reading and writing to the same file

I do not know which Python version you use, but both 2.7.13 and 3.6.1 versions give me the following error: b'' is not a number!. So because an error is raised, the write instruction is not interpreted.
The with statement is evaluated from left to right. So first, your file is open in read mode. Right after that, it is open in write mode and that causes the file to be truncated: there is nothing more to read.
You should proceed in two steps:
total = 0
initial = 10
# First, read the file and try to convert its content to an integer
with open('file.txt', 'r') as inp:
content = inp.read()
try:
total = int(content) + int(initial)
except ValueError:
print('Cannot convert {} to an int'.format(content))
with open('file.txt', 'w') as outp:
outp.write(str(total))

Reading a file and then overwriting it in Python

I've been trying to read a file and then overwrite it with some updated data.
I've tried doing it like this:
#Created filename.txt with some data
with open('filename.txt', 'r+') as f:
data = f.read()
new_data = process(data) # data is being changed
f.seek(0)
f.write(new_data)
For some reason, it doesn't overwrite the file and the content of it stays the same.

Truncate the file after seeking to the front. That will remove all of the existing data.
>>> open('deleteme', 'w').write('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
>>> f = open('deleteme', 'r+')
>>> f.read()
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
>>> f.seek(0)
>>> f.truncate()
>>> f.write('bbb')
>>> f.close()
>>> open('deleteme').read()
'bbb'
>>>

You should add a call to truncate after seek as tdelaney suggested.
Try reading and writing in different scopes, the code is more clear that way and the data processing is not done when the file handlers are open.
data = ''
with open('filename.txt', 'r') as f:
data = f.read()
new_data = process(data)
with open('filename.txt', 'w+') as f:
f.write(new_data)

Categories

Resources