Is there a short way to get number of objects in pickled file - shorter than writing a function that opens the file, keeps calling pickle.load method and updating num_of_objs by 1 until it catches EOFError and returns the value?
No, there isn't. The pickle format does not store that information.
If you need that type of metadata, you need to add it to the file yourself when writing:
pickle.dump(len(objects), fileobj)
for ob in objects:
pickle.dump(ob, fileobj)
Now the first record tells you how many more are to follow.
There is no direct way of finding the length of a pickle, but if you are afraid of running an endless loop you could try the following,
company_id_processed=[]
with open("responses_pickle.pickle", "rb") as f:
while True:
try:
current_id=pickle.load(f)['name']
company_id_processed.append(current_id)
except EOFError:
print 'Pickle ends'
break
The best way is to store and load data object with descriptive file name. For example, if you want to save two dataframes, you can name the pickle file as "datasets_name_2DFs.pickle". When you want to load them, you can simply get the number in the file name and use for loop equals to that number to get the pickle object. This is easier for me. For the code part, you can do what ever suits you.
Or you can use other methods like this:
with open(path, "wb") as f:
pickle.dump(len(data), f)
for value in data:
pickle.dump(value, f)
data_list = []
with open(path, "rb") as f:
for _ in range(pickle.load(f)):
data_list.append(pickle.load(f))
print data_list
Related
I want to generate a log file in which I have to print two lists for about 50 input files. So, there are approximately 100 lists reported in the log file. I tried using pickle.dump, but it adds some strange characters in the beginning of each value. Also, it writes each value in a different line and the enclosing brackets are also not shown.
Here is a sample output from a test code.
import pickle
x=[1,2,3,4]
fp=open('log.csv','w')
pickle.dump(x,fp)
fp.close()
output:
I want my log file to report:
list 1 is: [1,2,3,4]
If you want your log file to be readable, you are approaching it the wrong way by using pickle which "implements binary protocols"--i.e. it is unreadable.
To get what you want, replace the line
pickle.dump(x,fp)
with
fp.write(' list 1 is: '
fp.write(str(x))
This requires minimal change in the rest of your code. However, good practice would change your code to a better style.
pickle is for storing objects in a form which you could use to recreate the original object. If all you want to do is create a log message, the builtin __str__ method is sufficient.
x = [1, 2, 3, 4]
with open('log.csv', 'w') as fp:
print('list 1 is: {}'.format(x), file=fp)
Python's pickle is used to serialize objects, which is basically a way that an object and its hierarchy can be stored on your computer for use later.
If your goal is to write data to a csv, then read the csv file and output what you read inside of it, then read below.
Writing To A CSV File see here for a great tutorial if you need more info
import csv
list = [1,2,3,4]
myFile = open('yourFile.csv', 'w')
writer = csv.writer(myFile)
writer.writerow(list)
the function writerow() will write each element of an iterable (each element of the list in your case) to a column. You can run through each one of your lists and write it to its own row in this way. If you want to write multiple rows at once, check out the method writerows()
Your file will be automatically saved when you write.
Reading A CSV File
import csv
with open('example.csv', newline='') as File:
reader = csv.reader(File)
for row in reader:
print(row)
This will run through all the rows in your csv file and will print it to the console.
Since the Json And Pickle methods aren't working out, i've decided to save my dictionaries as strings, and that works, but they arent being read.
I.E
Dictionary
a={'name': 'joe'}
Save:
file = open("save.txt", "w")
file.write(str(a))
file.close()
And that works.
But my load method doesn't read it.
Load:
f = open("save.txt", "r")
a = f
f.close()
So, it just doesn't become f.
I really don't want to use json or pickle, is there any way I could get this method working?
First, you're not actually reading anything from the file (the file is not its contents). Second, when you fix that, you're going to get a string and need to transform that into a dictonary.
Fortunately both are straightforward to address....
from ast import literal_eval
with open("save.txt") as infile:
data = literal_eval(infile.read())
My code currently writes a dictionary which contains scores for a class to a CSV file. This part is correctly done by the program and the scores are wrote to file, however the latest dictionary written to file is not printed. For example, after the code has been ran once, it will not be printed however once the code has been ran for a second time, the first bit of data is printed however the new data isn't. Can someone tell me where I am going wrong?
SortedScores = sorted(Class10x1.items(), key = lambda t: t[0], reverse = True) #this sorts the scores in alphabetical order and by the highest score
FileWriter = csv.writer(open('10x1 Class Score.csv', 'a+'))
FileWriter.writerow(SortedScores) #the sorted scores are written to file
print "Okay here are your scores!\n"
I am guessing the problem is here somewhere however I cannot quite pinpoint what or where it is. I have tried to solve this by changing the mode of the file when it is read back in to r, r+ and rb, however all have the same consequence.
ReadFile = csv.reader(open("10x1 Class Score.csv", "r")) #this opens the file using csv.reader in read mode
for row in ReadFile:
print row
return
from Input output- python docs:
It is good practice to use the with keyword when dealing with file objects. This has the advantage that the file is properly closed after its suite finishes, even if an exception is raised on the way. It is also much shorter than writing equivalent try-finally blocks:
>>> with open('workfile', 'r') as f:
... read_data = f.read()
>>> f.closed
True
File objects have some additional methods, such as isatty() and truncate() which are less frequently used; consult the Library Reference for a complete guide to file objects.
I'm not sure why they bury that so far in the documentation since it is really useful and a very common beginner mistake:
SortedScores = sorted(Class10x1.items(), key = lambda t: t[0], reverse = True) #this sorts the scores in alphabetical order and by the highest score
with open('10x1 Class Score.csv', 'a+') as file:
FileWriter = csv.writer(file)
FileWriter.writerow(SortedScores) #the sorted scores are written to file
print "Okay here are your scores!\n"
this will close the file for you even if an error is raised which prevents many possibilities of loss of data
the reason it did not appear to write to the file is because when you do .write_row() it doesn't immediately write to the hard drive, only to a buffer which is occasionally emptied into the file on hard drive, although with only one write statement it has no need to empty.
Remember to close the file after operation, otherwise the data will not be saved properly.
Try to use with keyword so that Python will handle the closure for you:
import csv
with open('10x1 Class Score.csv', 'a+') as f:
csv_writer = csv.writer(f)
# write something into the file
...
# when the above block is done, file will be automatically closed
# so that the file is saved properly
This question already has answers here:
Write and read a list from file
(3 answers)
Closed 8 years ago.
I was wondering how I can save a list entered by the user. I was wondering how to save that to a file. When I run the program it says that I have to use a string to write to it. So, is there a way to assign a list to a file, or even better every time the program is run it automatically updates the list on the file? That would be great the file would ideally be a .txt.
stuffToDo = "Stuff To Do.txt"
WRITE = "a"
dayToDaylist = []
show = input("would you like to view the list yes or no")
if show == "yes":
print(dayToDaylist)
add = input("would you like to add anything to the list yes or no")
if add == "yes":
amount=int(input("how much stuff would you like to add"))
for number in range (amount):
stuff=input("what item would you like to add 1 at a time")
dayToDaylist.append(stuff)
remove = input("would you like to remove anything to the list yes or no")
if add == "yes":
amountRemoved=int(input("how much stuff would you like to remove"))
for numberremoved in range (amountRemoved):
stuffremoved=input("what item would you like to add 1 at a time")
dayToDaylist.remove(stuffremoved);
print(dayToDaylist)
file = open(stuffToDo,mode = WRITE)
file.write(dayToDaylist)
file.close()
You can pickle the list:
import pickle
with open(my_file, 'wb') as f:
pickle.dump(dayToDaylist, f)
To load the list from the file:
with open(my_file, 'rb') as f:
dayToDaylist = pickle.load( f)
If you want to check if you have already pickled to file:
import pickle
import os
if os.path.isfile("my_file.txt"): # if file exists we have already pickled a list
with open("my_file.txt", 'rb') as f:
dayToDaylist = pickle.load(f)
else:
dayToDaylist = []
Then at the end of your code pickle the list for the first time or else update:
with open("my_file.txt", 'wb') as f:
pickle.dump(l, f)
If you want to see the contents of the list inside the file:
import ast
import os
if os.path.isfile("my_file.txt"):
with open("my_file.txt", 'r') as f:
dayToDaylist = ast.literal_eval(f.read())
print(dayToDaylist)
with open("my_file.txt", 'w') as f:
f.write(str(l))
for item in list:
file.write(item)
You should check out this post for more info:
Writing a list to a file with Python
Padraic's answer will work, and is a great general solution to the problem of storing the state of a Python object on disk, but in this specific case Pickle is a bit overkill, not to mention the fact that you might want this file to be human-readable.
In that case, you may want to dump it to disk like such (this is from memory, so there may be syntax errors):
with open("list.txt","wt") as file:
for thestring in mylist:
print(thestring, file=file)
This will give you a file with your strings each on a separate line, just like if you printed them to the screen.
The "with" statement just makes sure the file is closed appropriately when you're done with it. The file keyword param to print() just makes the print statement sort of "pretend" that the object you gave it is sys.stdout; this works with a variety of things, such as in this case file handles.
Now, if you want to read it back in, you might do something like this:
with open("list.txt","rt") as file:
#This grabs the entire file as a string
filestr=file.read()
mylist=filestr.split("\n")
That'll give you back your original list. str.split chops up the string it's being called on so that you get a list of sub-strings of the original, splitting it every time it sees the character you pass in as a parameter.
I have a list, which may be empty or non-empty.
I want to create a new file which contains that list in a format that is human-readable and easy for my next script to parse. In the case where the list is non-empty, this works fine and my next script reads in the json file. But when the list is empty, I get "ValueError: No JSON object could be decoded." This makes sense, because when I open the file, there is indeed no content and thus no JSON object.
I'm OK with the fact that some lists are empty. So, either I want to write an empty JSON object or I want my reader script to be OK with not finding a JSON object.
Here's the relevant code:
Writer Script
favColor = [] OR favColor = ['blue'] OR favColor = ['blue', 'green']
fileName = 'favoriteColor.json'
outFile = open(fileName, 'w')
json.dump(outFile, favColor)
outFile.close()
Reader Script
fileName = 'favoriteColor.json'
inFile = open(fileName, 'r')
colors = json.load(inFile)
inFile.close()
Any help or suggestions much appreciated. If I need to provide more rationale for why I'm doing this, I can provide that as well, just thought I'd start off with minimum necessary to understand the problem.
Modify your reader script to this:
with open('favoriteColor.json') as inFile:
try:
colors = json.load(inFile)
except ValueError:
colors = []
This attempts to load the file as a json. If it fails due to a value error, we know that this is because the json is empty. Therefore we can just assign colors to an empty list. It is also preferable to use the "with" construct to load files since it closes them automatically.
I wouldn't take the approach you're trying. I would instead json.dump a dictionary, eg:
d = {'var1': '123', 'var2': [1, 2, 3]}
json.dump(d, fileout)
Then use dict.get to default it to a suitable value:
json_dict = json.load(filein)
favColor = json_dict.get('favColor', [])
Then you still have compulsory values that can except if not present using [] notation.
Puts the logic of missing values in your code instead of the json parsers...
You can also add the name of the variable in your json file:
json.dump({'favorite-color': favColor}, outFile)
outFile.close()
And handle the case of an empty list when reading the json file:
data = json.load(inFile)
if len(data['favorite-color']) == 0:
...
Note that you must provide the object that you want to save and then the file-like object to json.dump.