I need to write a 2D numpy array to a file, including its dimensions so I can read it from a C++ program and create the corresponding array.
I have written some simple code that saves the array and it can be read from C++, but if I try to write the array's size first it always gives me an error.
Here's my simple python code:
1 file = open("V.bin","wb")
2 file.write(V.shape)
3 file.write(V)
4 file.close()
The second line gives the error, I've also tried:
n1, n2 = V.shape
file.write(n1)
file.write(n2)
But it doesn't work either.
I'm adding the error it shows:
Traceback (most recent call last):
file.write(V.shape[0])
TypeError: must be string or buffer, not int
Thanks!
You can use numpy.save() which saves in binary.
You can use numpy.savetext if you want to save it as ascii.
Alternatively (since it looks like you're dealing with binary data), if you want to save the raw data stream, you could use ndarray.tostring to get a string of bytes that you can dump to the file directly.
The advantage of this approach is that you can create your own file format. The downside is that you need to create a string in order to actually write it to the file.
And since you say that you're getting an error on the second line, it's an error because f.write expects a string. You're trying to pass it a tuple or ints. You could use struct.pack to solve this problem:
f.write(struct.pack('2i',*array.shape))
Related
I'm confused as to why this isn't wanting to write to an outfile.
I extract data from a txt file using np.loadtxt() I try to write to an existing file, but I'm getting this error 'numpy.float64' object is not iterable' I'm not looping through a float value, rather looping through each element of the array then writing it to an existing file.
Here's my code
mass = np.loadtxt('C:/Users/Luis/Desktop/data.txt',usecols=0)
with open('test','w') as outfile:
for i in range(len(mass)):
outfile.writelines(mass[i])
could it be that the function with open() doesn't work with NumPy arrays?
Thanks
with open() is a context manager used to work with files (works with all files). The writelines() method takes str as input argument. So you do have to convert to writelines(str(mass[i])). But still the temp variable i is iterating over the line but the max value it takes is length of the file.(length of the file may not be equal to the length of each line in the file). I think this is what you want.
mass = np.loadtxt('/content/sample_data/st.txt',usecols=0)
with open('test.txt','w') as outfile:
for line in mass:
outfile.writelines(str(line))
In the process of tweaking scipy's wavfile reader to read only chunks of data at once, I'm encountering a few problems with numpy's tostring() method. Using the default reader to extract the signal and convert to bytes works fine. However, once I use numpy's fromstring() to read a set amount of data (length 512000), tostring() no longer works. Printing either results in a blank line or the terminal asking me if I want to install a dozen new fonts. I use the dtype of the data when extracting, like
signal = np.fromstring(fid.read(512000), dtype=dtype)
print(signal.tostring())
This usually yields a blank line. If I try to force a dtype, then it'll spit out random characters that don't exist. Using scipy.io.wavfile to read the file, then take a 512000 length segment of it will still convert correctly with tostring(), so I know it isn't a length or a type error. Are there common errors of this type occurring with the tostring() method? How do I fix them?
I understand what these functions do, but what would be the practical use of reading/writing some as a list, and what is the use of writing a list to a file, if you could just use write() and then later use readlines() to view it as a list?
The practical upside of writing data to disk is that you can drop it from your system memory and come back to it later.
.writelines() accepts any iterable that produces strings. It could be implemented as:
def writelines(self, lines):
for line in lines:
self.write(line)
If you call file.writelines(["abc", "def"]) then the file will contain: abcdef. And file.readlines() would return ["abcdef"] if you read it back. As you see the roundtrip produces different result.
If you call file.write(["abc", "def"]) then you get TypeError: must be string or buffer, not list. Again it is different.
I'm having a really difficult time writing integers out to a file. Here's my situation. I have a file, let's call it 'idlist.txt'. It has multiple columns and is fairly long (10,000 rows), but I only care about the first column of data.
I'm loading it into python using:
import numpy as np
FH = np.loadtxt('idlist.txt',delimiter=',',comments='#')
# Testing initial data type
print FH[0,0],type(FH[0,0])
>>> 85000370342.0 <type 'numpy.float64'>
# Converting to integers
F = [int(FH[i,0]) for i in range(len(FH))]
print F[0],type(F[0])
>>> 85000370342 <type 'long'>
As you can see, the data must be made into integers. What I now would like to do is to write the entries of this list out as the first column of another file (really the only column in the entire file), we can rename it 'idonly.txt'. Here is how I'm trying to do it:
with open('idonly.txt','a') as f:
for i in range(len(F)):
f.write('%d\n' % (F[i]))
This is clearly not producing the desired output - when I open the file 'idonly.txt', each entry is actually a float (i.e. - 85000370342.0). What exactly is going on here, and why is writing integers to a file such a complicated task? I found the string formatting idea from here: How to write integers to a file , but it didn't fix my issue.
Okay, well it appears that this is completely my fault. When I'm opening the file I'm using the mode 'a', which means append. It turns out that the first time I wrote this out to a file I did it incorrectly, and ever since I've been appending the correct answer onto that and simply not looking down as far as I should since it's a really long file.
For reference here are all of the modes you can use when handling files in python: http://www.tutorialspoint.com/python/python_files_io.htm. Choose carefully.
Try to use:
f.write('{}'.format(F[i]));
So, I have a .txt file that I want to read into Pylab. The problem is that, when I try to do so using the numpy.loadtxt("filename.txt"), Pylab cannot read the numbers in the array in my file as float values (it returns the error: cannot convert string to float.).
I am not sure if there is something wrong with my syntax as above; when I remove the quotation marks inside the parentheses, numpy.loadtxt(filename.txt), Pylab returns the error: filename is not defined.
Any suggestions on how to read a series of numbers saved in a .txt file into Pylab as an array of floats?
You need to provide sample lines in your filename.txt file. I guess you may need to read the doc for numpy.loadtxt here. There are some good examples on the document page.
BTW, the second command numpy.loadtxt(filename.txt) is wrong since you have not defined a variable filename.