I am trying to create a file and write data from a variable to that new file using JSON. Currently, I have a variable called open_diction which is a large dictionary in a file that contains other data. So I am trying to create a new file called open_diction_saved.json and write the data from open_diction to that new file. Currently I am getting the error TypeError: is not JSON serializable
f = open ("open_diction_saved.json","w")
json.dumps(f)
f.write(open_diction)
f.close()
Any help would be great!
The problem is that you're trying to serialise a writable file object. If your intention is to overwrite open_diction_saved.json, then the following code would be what you're looking for.
f = open("open_diction_saved.json", 'w')
f.write(json.dumps(open_diction)) #serialise open_diction obj, then write to file
f.close()
You need to place json.dumps() inside the write() method:
import json
open_diction = {'a':1, 'b':2}
with open("open_diction_saved.json", "w") as f:
f.write(json.dumps(open_diction))
Related
I basically want to take a CSV file as input, and return a dictionary as a response. Since the default fastapi UploadFile module returns a spooled temp file, which i can't use as an input to pandas.read_csv(), what i am trying to do is write the contents of the uploaded file to another file buffer, and than use read_csv() and to_dict() to convert the CSV data to a dictionary.
This is how i am trying to do it:
async def CSVToDict(file: UploadFile = File(...)):
if file.filename.endswith('.csv'):
#open a new file buffer to write the data to
with open(file.filename, 'rb+') as temp_file:
#read the contents of the uploaded file
content = await file.read()
#write the content to the new buffer
temp_file.write(content)
#convert the data to a dictionary
dict = pandas.read_csv(temp_file, header=None, index_col=0, squeeze=True).to_dict()
#delete the buffer as i don't want to store anything in the server
temp_file.flush()
#return the dictionary as data
return {"data": dict}
else:
return {"Invalid file format"}
but i am getting the following error:
pandas.errors.EmptyDataError: No columns to parse from file
I understand that this is because the file i am creating "as temp_file" is not created as a CSV file and read_csv() requires a CSV file to work. But i can't figure out a way to correct this. How do i use pandas to convert an uploaded csv file to a dictionary??
Thank you for your time...
And wishing all a happy new year!!
I resolved the issue, instead of writing data to a new file, we can indeed encode the data to UTF-8 instead of reading bytes.
Using the default CSV module, we can pass the file after decoding it:
iterator = csv.reader(codecs.iterdecode(data.file, 'utf-8'), delimiter=',')
This "iterator" can now be used as a CSV reader.
Read more about codecs for more details.
Say I ran some code that produces multiple arrays as its output. How might I save the entire output in one go as in matlab?
In matlab I'd simply say save(data) -> load('data').
Apologies if this is a basic quesiton.
How to save a python object:
To save objects to file in Python, you can use pickle:
Import the pickle module.
Get a file handle in write mode that points
to a file path.
Use pickle.dump to write the object that we want to
save to file via that file handle.
How to use it:
import pickle
object = Object()
filehandler = open(filename, 'w')
pickle.dump(object, filehandler)
How to load a python object:
Import the pickle module.
Get a file handle in read mode that points to a file path that has a file that contains the serialized form of a Python object.
Use pickle.load to read the object from the file that contains the serialized form of a Python object via the file handle.
How to use it:
import pickle
filehandler = open(filename, 'r')
object = pickle.load(filehandler)
Extras : save multiple objects at once
Obviously, using a list you can also store multiple objects at once:
object_a = "foo"
object_b = "bar"
object_c = "baz"
objects_list = [object_a , object_b , object_c ]
file_name = "my_objects.pkl"
open_file = open(file_name, "wb")
pickle.dump(objects_list, open_file)
open_file.close()
This is my first question here, I'm new to python and trying to figure some things out to set up an automatic 3D model processing chain that relies on data being stored in JSON files moving from one server to another.
The problem is that I need to store absolute paths to files that are being processed, but these absolute paths should be modified in the original JSON files upon the first time that they are processed.
Basically the JSON file comes in like this:
{
"normaldir": "D:\\Outgoing\\1621_1\\",
"projectdir": "D:\\Outgoing\\1622_2\\"
}
And I would like to rename the file paths to
{
"normaldir": "X:\\Incoming\\1621_1\\",
"projectdir": "X:\\Incoming\\1622_2\\",
}
What I've been trying to do is replace the first part of the path using this code, but it isn't working:
def processscan(scanfile):
configfile= MonitorDirectory + scanfile
with open(configfile, 'r+') as file:
content = file.read()
file.seek(0)
content.replace("D:\\Outgoing\\", "X:\\Incoming\\")
file.write(content)
However this was not working at all, so I tried interpreting the JSON file properly and replacing the key code from here:
def processscan(scanfile):
configfile= MonitorDirectory + scanfile
with open(configfile, 'r+') as settingsData:
settings = json.load(settingsData)
settings['normaldir'] = 'X:\\Incoming\\1621_1\\'
settings['projectdir'] = 'X:\\Incoming\\1622_2\\'
settingsData.seek(0) # rewind to beginning of file
settingsData.write(json.dumps(settings,indent=2,sort_keys=True)) #write the updated version
settingsData.truncate() #truncate the remainder of the data in the file
This works perfectly, however I'm replacing the whole path so it won't really work for every JSON file that I need to process. What I would really like to do is to take a JSON key corresponding to a file path, keep the last 8 characters and replace the rest of the patch with a new string, but I can't figure out how to do this using json in python, as far as I can tell I can't edit part of a key.
Does anyone have a workaround for this?
Thanks!
Your replace logic failed as you need to reassign content to the new string,str.replace is not an inplace operation, it creates a new string:
content = content.replace("D:\\Outgoing\\", "X:\\Incoming\\")
Using the json approach just do a replace too, using the current value:
settings['normaldir'] = settings['normaldir'].replace("D:\\Outgoing\\", "X:\\Incoming\\")
You also would want truncate() before you write or just reopen the file with w and dump/write the new value, if you really wanted to just keep the last 8 chars and prepend a string:
settings['normaldir'] = "X:\\Incoming\\" + settings['normaldir'][-8:]
Python come with a json library.
With this library, you can read and write JSON files (or JSON strings).
Parsed data is converted to Python objects and vice versa.
To use the json library, simply import it:
import json
Say your data is stored in input_data.json file.
input_data_path = "input_data.json"
You read the file like this:
import io
with io.open(input_data_path, mode="rb") as fd:
obj = json.load(fd)
or, alternatively:
with io.open(input_data_path, mode="rb") as fd:
content = fd.read()
obj = json.loads(content)
Your data is automatically converted into Python objects, here you get a dict:
print(repr(obj))
# {u'projectdir': u'D:\\Outgoing\\1622_2\\',
# u'normaldir': u'D:\\Outgoing\\1621_1\\'}
note: I'm using Python 2.7 so you get the unicode string prefixed by "u", like u'projectdir'.
It's now easy to change the values for normaldir and projectdir:
obj["normaldir"] = "X:\\Incoming\\1621_1\\"
obj["projectdir"] = "X:\\Incoming\\1622_2\\"
Since obj is a dict, you can also use the update method like this:
obj.update({'normaldir': "X:\\Incoming\\1621_1\\",
'projectdir': "X:\\Incoming\\1622_2\\"})
That way, you use a similar syntax like JSON.
Finally, you can write your Python object back to JSON file:
output_data_path = "output_data.json"
with io.open(output_data_path, mode="wb") as fd:
json.dump(obj, fd)
or, alternatively with indentation:
content = json.dumps(obj, indent=True)
with io.open(output_data_path, mode="wb") as fd:
fd.write(content)
Remarks: reading/writing JSON objects is faster with a buffer (the content variable).
.replace returns a new string, and don't change it. But you should not treat json-files as normal text files, so you can combine parsing json with replace:
def processscan(scanfile):
configfile= MonitorDirectory + scanfile
with open(configfile, 'rb') as settingsData:
settings = json.load(settingsData)
settings = {k: v.replace("D:\\Outgoing\\", "X:\\Incoming\\")
for k, v in settings.items()
}
with open(configfile, 'wb') as settingsData:
json.dump(settings, settingsData)
Since the Json And Pickle methods aren't working out, i've decided to save my dictionaries as strings, and that works, but they arent being read.
I.E
Dictionary
a={'name': 'joe'}
Save:
file = open("save.txt", "w")
file.write(str(a))
file.close()
And that works.
But my load method doesn't read it.
Load:
f = open("save.txt", "r")
a = f
f.close()
So, it just doesn't become f.
I really don't want to use json or pickle, is there any way I could get this method working?
First, you're not actually reading anything from the file (the file is not its contents). Second, when you fix that, you're going to get a string and need to transform that into a dictonary.
Fortunately both are straightforward to address....
from ast import literal_eval
with open("save.txt") as infile:
data = literal_eval(infile.read())
I have json file with some data, and would like to occasionally update this file.
I read the file:
with open('index.json', 'rb') as f:
idx = json.load(f)
then check for presence of a key from potentially new data, and if key is not present update the file:
with open('index.json', mode='a+') as f:
json.dump(new_data, f, indent=4)
However this procedure just creates new json object (python dict) and appends it as new object in output json file, making the file not valid json file.
Is there any simple way to append new data to json file without overwriting whole file, by updating the initial dict?
One way to do what you're after is to write one JSON object per line in the file. I'm using that approach and it works quite well.
A nice benefit is that you can read the file more efficiently (memory-wise) because you can read one line at a time. If you need all of them, there's no problem with assembling a list in Python, but if you don't you're operating much faster and you can also append.
So to initially write all your objects, you'd do something like this:
with open(json_file_path, "w") as json_file:
for data in data_iterable:
json_file.write("{}\n".format(json.dumps(data)))
Then to read efficiently (will consume little memory, no matter the file size):
with open(json_file_path, "r") as json_file:
for line in json_file:
data = json.loads(line)
process_data(data)
To update/append:
with open(json_file_path, "a") as json_file:
json_file.write("{}\n".format(json.dumps(new_data)))
Hope this helps :)