From what I understand, the only way to edit an object in a pickle file is to unpickle each object, edit the desired object, and repickle everything back into the original file.
This is what I tried doing:
pickleWrite = open(fileName, 'wb')
pickleRead = open(fileName, 'rb')
#unpickle objects and put it in dataList
dataList = list()
try:
while True:
dataList.append(pickle.load(pickleRead))
except EOFError:
pass
#change desired pickle object
dataList[0] = some change
#clear pickle file
pickleWrite.truncate(0)
#repickle each item in data list
for data in dataList:
pickle.dump(data, fileName)
For some reason, this makes the pickle file have some large number of unknown symbols at the front of the file making it unpickleable.
Error when we try to unpickle:
_pickle.UnpicklingError: invalid load key, '\x00'.
I would suggest avoiding to create multiple opened connection to the same file like this.
Instead, you can try:
# Read the contents
with open(filename, 'rb') as file:
dataList = pickle.load(file)
# something to dataList
# Overwrite the picle file
with open(filename, 'wb') as file:
pickle.dump(dataList, file)
Related
I have a python function here that takes in 2 params, old_file is the name of the file that contains the content that I want to copy over to the new file and the new_file is the name for the file I want to create. Right now my code looks like this and it would throw an error which is a type error that says "TypeError: expected str, bytes or os.PathLike object, not TextIOWrapper". Is there a more efficient way to do it?. In addition when I point at with open(file, "r+") as f2: I get this warning
Note. this works if the file is already pre-made but not when i make it in the function
def copy_In(old_file, new_file):
file = open(new_file, "w")
with open(file, "r+") as f2:
for x in range(10):
f2.readline()
pos = f2.tell()
f2_remainder = f2.read()
f2.seek(pos)
with open(old_file, "r") as f1:
for line in f1:
f2.write(line)
f2.write(f2_remainder)
just do it like:
def copy_file(old_fname, new_fname):
with open(old_fname, "rt") as old_fobj, open(new_fname, "wt") as new_fobj:
new_fobj.write(old_fobj.read())
we are opening first file in read mode and second in write mode and reading the entire content of first file and writing it in second file.
I'm working with TSV file (here below my_file) and trying to write it down to another temp file with a random ID (here below my_temp_file)and this is what I wrote:
def temp_generator():
while True:
my_string = 'tmp' + str(random.randint(1,1000000000))
if not os.path.exists(my_string):
return my_string
randomID = temp_generator()
my_temp_file = open('mytemp_'+randomID + '.tsv', 'w')
with open(my_file, 'r+') as mf:
for line in mf:
my_temp_file.write(line)
my_temp_file.close()
mf.close()
The output is something like:
mytemp_1283189.tsv
Now I'd like to work with my_temp_file.tsv in order to modify its content and rename it but if I try to open it with:
with open (my_temp_file.tsv, 'r') as mtf:
data = mtf.read()
print(data)
This is what I obtain:
TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper
What can I do?
Issue
The pattern handle = open(path)
is opening a file at path from path and returns the handle assigned to handle. You can use handle to .write, .read, or .close. But you can not open it again or use it as input to open - which expects a Path-like object, e.g. a filename.
Fixed
def temp_generator():
while True:
my_string = 'tmp' + str(random.randint(1,1000000000))
if not os.path.exists(my_string):
return my_string
randomID = temp_generator()
# copy from input (my_file) to output, a random temp file (my_temp_file)
my_temp_file = 'mytemp_' + randomID + '.tsv'
with open(my_temp_file, 'w') as mtf:
with open(my_file, 'r+') as mf: # my_file is supposed to be a Path-like object
for line in mf:
mtf.write(line)
# since with..open used, no close needed (auto-close!)
# modify output (content and rename the file)
# remember: my_temp_file is holding a Path or filename
with open(my_temp_file, 'r') as mtf: # open the file again
data = mtf.read()
print(data)
See also:
[Solved] Python TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper
Python documentation about TextIOWrapper: io — Core tools for working with streams
Your error is here, assuming you are actually using open(my_temp_file
my_temp_file = open('mytemp_'+randomID + '.tsv', 'w')
with open(my_file, 'r+') as mf:
You've already opened the file, so you shouldn't open it again using the file handle as the parameter. You should prefer only using the with way of opening files, too
For example
my_temp_file = 'mytemp_'+randomID + '.tsv'
with open(my_temp_file, 'r+') as mf:
Even then, if you're going to eventually rename the file, just make it the name you want from the start
I am trying to create a small game for fun, and I want to save and load previous run scores. I started a test file to mess around and try to figure out how pickling works. I have a pickle file with a small set of number. How do I add numbers to the pickle file and save it for the next run.
Currently I have it like this:
new_score = 9
filename = "scoreTest.pk"
outfile = open(filename,'wb')
infile = open(filename,'rb')
with infile as f:
scores = pickle.load(f)
scores.add(new_score)
pickle.dump(scores, outfile)
When I run it like this I get this error:
EOFError: Ran out of input
If someone could please tell me what is wrong and how to do it correctly that would be great. Apologies for any un-optimal code, I'm new to code.
You are trying to juggle a reader and writer on the same file at the same time. The open(filename, 'wb') of the write deletes whatever happened to be in the file so there is no data for the reader. You should only open the file when you really need to use it. And its better to write to a temporary file and rename it. If something goes wrong you haven't lost your data.
import pickle
import os
new_score = 9
filename = "scoreTest.pk"
tmp_filename = "scoreTest.tmp"
try:
with open(filename, 'rb') as infile:
scores = pickle.load(f)
except (IOError, EOFError) as e:
scores = default # whatever that is
scores.add(new_score)
with open(tmp_filename, 'wb') as outfile:
pickle.dump(scores, outfile)
os.rename(tmp_filename, filename)
Scenario is i need to convert dictionary object as json and write to a file . New Dictionary objects would be sent on every write_to_file() method call and i have to append Json to the file .Following is the code
def write_to_file(self, dict=None):
f = open("/Users/xyz/Desktop/file.json", "w+")
if json.load(f)!= None:
data = json.load(f)
data.update(dict)
f = open("/Users/xyz/Desktop/file.json", "w+")
f.write(json.dumps(data))
else:
f = open("/Users/xyz/Desktop/file.json", "w+")
f.write(json.dumps(dict)
Getting this error "No JSON object could be decoded" and Json is not written to the file. Can anyone help ?
this looks overcomplex and highly buggy. Opening the file several times, in w+ mode, and reading it twice won't get you nowhere but will create an empty file that json won't be able to read.
I would test if the file exists, if so I'm reading it (else create an empty dict).
this default None argument makes no sense. You have to pass a dictionary or the update method won't work. Well, we can skip the update if the object is "falsy".
don't use dict as a variable name
in the end, overwrite the file with a new version of your data (w+ and r+ should be reserved to fixed size/binary files, not text/json/xml files)
Like this:
def write_to_file(self, new_data=None):
# define filename to avoid copy/paste
filename = "/Users/xyz/Desktop/file.json"
data = {} # in case the file doesn't exist yet
if os.path.exists(filename):
with open(filename) as f:
data = json.load(f)
# update data with new_data if non-None/empty
if new_data:
data.update(new_data)
# write the updated dictionary, create file if
# didn't exist
with open(filename,"w") as f:
json.dump(data,f)
I am using a script to strip exif data from uploaded JPGs in Python, before writing them to disk. I'm using Flask, and the file is brought in through requests
file = request.files['file']
strip the exif data, and then save it
f = open(file)
image = f.read()
f.close()
outputimage = stripExif(image)
f = ('output.jpg', 'w')
f.write(outputimage)
f.close()
f.save(os.path.join(app.config['IMAGE_FOLDER'], filename))
Open isn't working because it only takes a string as an argument, and if I try to just set f=file, it throws an error about tuple objects not having a write attribute. How can I pass the current file into this function before it is read?
file is a FileStorage, described in http://werkzeug.pocoo.org/docs/datastructures/#werkzeug.datastructures.FileStorage
As the doc says, stream represents the stream of data for this file, usually under the form of a pointer to a temporary file, and most function are proxied.
You probably can do something like:
file = request.files['file']
image = file.read()
outputimage = stripExif(image)
f = open(os.path.join(app.config['IMAGE_FOLDER'], 'output.jpg'), 'w')
f.write(outputimage)
f.close()
Try the io package, which has a BufferedReader(), ala:
import io
f = io.BufferedReader(request.files['file'])
...
file = request.files['file']
image = stripExif(file.read())
file.close()
filename = 'whatever' # maybe you want to use request.files['file'].filename
dest_path = os.path.join(app.config['IMAGE_FOLDER'], filename)
with open(dest_path, 'wb') as f:
f.write(image)