Say I ran some code that produces multiple arrays as its output. How might I save the entire output in one go as in matlab?
In matlab I'd simply say save(data) -> load('data').
Apologies if this is a basic quesiton.
How to save a python object:
To save objects to file in Python, you can use pickle:
Import the pickle module.
Get a file handle in write mode that points
to a file path.
Use pickle.dump to write the object that we want to
save to file via that file handle.
How to use it:
import pickle
object = Object()
filehandler = open(filename, 'w')
pickle.dump(object, filehandler)
How to load a python object:
Import the pickle module.
Get a file handle in read mode that points to a file path that has a file that contains the serialized form of a Python object.
Use pickle.load to read the object from the file that contains the serialized form of a Python object via the file handle.
How to use it:
import pickle
filehandler = open(filename, 'r')
object = pickle.load(filehandler)
Extras : save multiple objects at once
Obviously, using a list you can also store multiple objects at once:
object_a = "foo"
object_b = "bar"
object_c = "baz"
objects_list = [object_a , object_b , object_c ]
file_name = "my_objects.pkl"
open_file = open(file_name, "wb")
pickle.dump(objects_list, open_file)
open_file.close()
Related
Using a library such as Python's io, I can create a file e.g. csv format, in memory. However, I cannot get a UNC (Universal Naming Convention) referencing this file. How would I be able to assign such a name to an in-memory file created by Python?
A "file" created in memory using the io module is not a "file" as far as the OS is concerned. You can't open it by name or access it from outside the process. This file only exists as a variable in the program.
You can get the string contents directly, as in the example # https://docs.python.org/3/library/io.html#io.StringIO
import io
output = io.StringIO()
output.write('First line.\n')
print('Second line.', file=output)
contents = output.getvalue()
or you can rewind the file and read it back:
output.seek(0)
contents = output.read()
I have a file and want to convert it into BytesIO object so that it can be stored in database's varbinary column.
Please can anyone help me convert it using python.
Below is my code:
f = open(filepath, "rb")
print(f.read())
myBytesIO = io.BytesIO(f)
myBytesIO.seek(0)
print(type(myBytesIO))
Opening a file with open and mode read-binary already gives you a Binary I/O object.
Documentation:
The easiest way to create a binary stream is with open() with 'b' in the mode string:
f = open("myfile.jpg", "rb")
So in normal circumstances, you'd be fine just passing the file handle wherever you need to supply it. If you really want/need to get a BytesIO instance, just pass the bytes you've read from the file when creating your BytesIO instance like so:
from io import BytesIO
with open(filepath, "rb") as fh:
buf = BytesIO(fh.read())
This has the disadvantage of loading the entire file into memory, which might be avoidable if the code you're passing the instance to is smart enough to stream the file without keeping it in memory. Note that the example uses open as a context manager that will reliably close the file, even in case of errors.
I'm trying to interface with an existing library that uses the built in open() function to read a .json file using either a str or bytes object representing a path, or an object implementing the os.PathLike protocol.
My function generate a dictionary which is converted to json using json.dump(), but I'm not sure how to pass that to the existing function which expects a file path.
I was thinking something like this might work, but I'm not sure how to get a os.PathLike object of a TemporaryFile.
import tempfile
temp_file = tempfile.TemporaryFile('w')
json.dump('{"test": 1}', fp=temp_file)
file = open(temp_file.path(), 'r')
Create a NamedTemporaryFile() object instead; it has a .name attribute you can pass on to the function:
from tempfile import NamedTemporaryFile
with NamedTemporaryFile('w') as jsonfile:
json.dump('{"test": 1}', jsonfile)
jsonfile.flush() # make sure all data is flushed to disk
# pass the filename to something that expects a string
open(jsonfile.name, 'r')
Opening an already-open file object does have issues on Windows (you are not allowed to); there you'd have to close the file object first (making sure to disable delete-on-close), and delete it manually afterwards:
from tempfile import NamedTemporaryFile
import os
jsonfile = NamedTemporaryFile('w', delete=False)
try:
with jsonfile:
json.dump('{"test": 1}', jsonfile)
# pass the filename to something that expects a string
open(jsonfile.name, 'r')
finally:
os.unlink(jsonfile.name)
The with statement causes the file to be closed when the suite exits (so by the time you reach the open() call).
I am trying to create a file and write data from a variable to that new file using JSON. Currently, I have a variable called open_diction which is a large dictionary in a file that contains other data. So I am trying to create a new file called open_diction_saved.json and write the data from open_diction to that new file. Currently I am getting the error TypeError: is not JSON serializable
f = open ("open_diction_saved.json","w")
json.dumps(f)
f.write(open_diction)
f.close()
Any help would be great!
The problem is that you're trying to serialise a writable file object. If your intention is to overwrite open_diction_saved.json, then the following code would be what you're looking for.
f = open("open_diction_saved.json", 'w')
f.write(json.dumps(open_diction)) #serialise open_diction obj, then write to file
f.close()
You need to place json.dumps() inside the write() method:
import json
open_diction = {'a':1, 'b':2}
with open("open_diction_saved.json", "w") as f:
f.write(json.dumps(open_diction))
This is my first question here, I'm new to python and trying to figure some things out to set up an automatic 3D model processing chain that relies on data being stored in JSON files moving from one server to another.
The problem is that I need to store absolute paths to files that are being processed, but these absolute paths should be modified in the original JSON files upon the first time that they are processed.
Basically the JSON file comes in like this:
{
"normaldir": "D:\\Outgoing\\1621_1\\",
"projectdir": "D:\\Outgoing\\1622_2\\"
}
And I would like to rename the file paths to
{
"normaldir": "X:\\Incoming\\1621_1\\",
"projectdir": "X:\\Incoming\\1622_2\\",
}
What I've been trying to do is replace the first part of the path using this code, but it isn't working:
def processscan(scanfile):
configfile= MonitorDirectory + scanfile
with open(configfile, 'r+') as file:
content = file.read()
file.seek(0)
content.replace("D:\\Outgoing\\", "X:\\Incoming\\")
file.write(content)
However this was not working at all, so I tried interpreting the JSON file properly and replacing the key code from here:
def processscan(scanfile):
configfile= MonitorDirectory + scanfile
with open(configfile, 'r+') as settingsData:
settings = json.load(settingsData)
settings['normaldir'] = 'X:\\Incoming\\1621_1\\'
settings['projectdir'] = 'X:\\Incoming\\1622_2\\'
settingsData.seek(0) # rewind to beginning of file
settingsData.write(json.dumps(settings,indent=2,sort_keys=True)) #write the updated version
settingsData.truncate() #truncate the remainder of the data in the file
This works perfectly, however I'm replacing the whole path so it won't really work for every JSON file that I need to process. What I would really like to do is to take a JSON key corresponding to a file path, keep the last 8 characters and replace the rest of the patch with a new string, but I can't figure out how to do this using json in python, as far as I can tell I can't edit part of a key.
Does anyone have a workaround for this?
Thanks!
Your replace logic failed as you need to reassign content to the new string,str.replace is not an inplace operation, it creates a new string:
content = content.replace("D:\\Outgoing\\", "X:\\Incoming\\")
Using the json approach just do a replace too, using the current value:
settings['normaldir'] = settings['normaldir'].replace("D:\\Outgoing\\", "X:\\Incoming\\")
You also would want truncate() before you write or just reopen the file with w and dump/write the new value, if you really wanted to just keep the last 8 chars and prepend a string:
settings['normaldir'] = "X:\\Incoming\\" + settings['normaldir'][-8:]
Python come with a json library.
With this library, you can read and write JSON files (or JSON strings).
Parsed data is converted to Python objects and vice versa.
To use the json library, simply import it:
import json
Say your data is stored in input_data.json file.
input_data_path = "input_data.json"
You read the file like this:
import io
with io.open(input_data_path, mode="rb") as fd:
obj = json.load(fd)
or, alternatively:
with io.open(input_data_path, mode="rb") as fd:
content = fd.read()
obj = json.loads(content)
Your data is automatically converted into Python objects, here you get a dict:
print(repr(obj))
# {u'projectdir': u'D:\\Outgoing\\1622_2\\',
# u'normaldir': u'D:\\Outgoing\\1621_1\\'}
note: I'm using Python 2.7 so you get the unicode string prefixed by "u", like u'projectdir'.
It's now easy to change the values for normaldir and projectdir:
obj["normaldir"] = "X:\\Incoming\\1621_1\\"
obj["projectdir"] = "X:\\Incoming\\1622_2\\"
Since obj is a dict, you can also use the update method like this:
obj.update({'normaldir': "X:\\Incoming\\1621_1\\",
'projectdir': "X:\\Incoming\\1622_2\\"})
That way, you use a similar syntax like JSON.
Finally, you can write your Python object back to JSON file:
output_data_path = "output_data.json"
with io.open(output_data_path, mode="wb") as fd:
json.dump(obj, fd)
or, alternatively with indentation:
content = json.dumps(obj, indent=True)
with io.open(output_data_path, mode="wb") as fd:
fd.write(content)
Remarks: reading/writing JSON objects is faster with a buffer (the content variable).
.replace returns a new string, and don't change it. But you should not treat json-files as normal text files, so you can combine parsing json with replace:
def processscan(scanfile):
configfile= MonitorDirectory + scanfile
with open(configfile, 'rb') as settingsData:
settings = json.load(settingsData)
settings = {k: v.replace("D:\\Outgoing\\", "X:\\Incoming\\")
for k, v in settings.items()
}
with open(configfile, 'wb') as settingsData:
json.dump(settings, settingsData)