I want to create two binary files to append numpy arrays into each one of them during a loop. I wrote the following method (I use Python 2.7):
for _ in range(5):
C = np.random.rand(1, 5)
r = np.random.rand(1, 5)
with open("C.bin", "ab") as file1, open("r.bin", "ab") as file2:
# Append to binary files
np.array(C).tofile(file1)
np.array(r).tofile(file2)
# Now printing to check if appending is successful
C = np.load("C.bin")
r = np.load("r.bin")
print (C)
print (r)
However, I keep getting this error:
Traceback (most recent call last):
File "test.py", line 15, in <module>
C = np.load("C.bin")
File "/anaconda/lib/python2.7/site-packages/numpy/lib/npyio.py", line 429, in load
"Failed to interpret file %s as a pickle" % repr(file))
IOError: Failed to interpret file 'C.bin' as a pickle
I tried to fix it but I cannot see anything more. Any help is appreciated.
NOTE: I intentionally want to use np.load because later on I will be loading the dataset from the disk into a numpy array for further processing.
You should use the save method that is built in the numpy to store the array in the files. Here what your code should look like:
for _ in range(5):
C = np.random.rand(1, 5)
r = np.random.rand(1, 5)
np.save('C', C)
np.save('r', r)
# Now printing to check if appending is successful
C = np.load("C.npy")
r = np.load("r.npy")
print (C)
print (r)
del C, r
Please refer to the documentation https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.load.html
Related
I have a list, in python, which is given by:
inputs = ['eos', 5, 10, 20, 30]
The first element is a string and the others are int. I want to save this in a file and then open it in another python notebook, in a way that i can call input[0] and get as output the string "eos" and call the others elements, for exemple, as input[1]and get as output the int 5.
I tried to save the list inputs as follows:
np.savetxt(path + '/inputs.txt', inputs, delimiter=" ", header = 'Eos N1 N2 N3 N4')
but it gave an error:
TypeError: Mismatch between array dtype ('<U32') and format specifier ('%.18e')
How can i fiz this error? It's important that i be able to call any element of the file (of the list inputs) through the form inputs[i] and that the elements remain string ("eos") or integers (5, 10, 20, 30).
I'd use pickle in this case.. It's super easy
# data can be anything...
data0 = ['eos', 5, 10, 20, 30]
# write data into file
with open('datafile.txt', 'wb') as f:
pickle.dump(mylist, f)
# read back and check
f = open ("datafile.txt", "rb")
data1 = pickle.load(f)
# [bonus] check data is same
try:
assert data0==data1
except:
print("data isn't preserved")
else:
print("data is preserved")
I am new to both Matlab and Python and I have to convert a program in Matlab to Python. I am not sure how to typecast the data after reading from the file in Python. The file used is a binary file.
Below is the Matlab code:
fid = fopen (filename, 'r');
fseek (fid, 0, -1);
meta = zeros (n, 9, 'single');
v = zeros (n, 128, 'single');
d = 0;
for i = 1:n
meta(i,:) = fread (fid, 9, 'float');
d = fread (fid, 1, 'int');
v(i,:) = fread (fid, d, 'uint8=>single');
end
I have written the below program in python:
fid = open(filename, 'r')
fid.seek(0 , 0)
meta = np.zeros((n,9),dtype = np.float32)
v = np.zeros((n,128),dtype = np.float32)
for i in range(n):
data_str = fid.read(9);
meta[1,:] = unpack('f', data_str)
For this unpack, I getting the error as
"unpack requires a string argument of length 4"
.
Please suggest someway to make it work.
I looked a little in the problem mainly because I need this in the near future, too. Turns out there is a very simple solution using numpy, assuming you have a matlab matrix stored like I do.
import numpy as np
def read_matrix(file_name):
return np.fromfile(file_name, dtype='<f') # little-endian single precision float
arr = read_matrix(file_path)
print arr[0:10] #sample data
print len(arr) # number of elements
The data type (dtype) you must find out yourself. Help on this is here. I used fwrite(fid,value,'single'); to store the data in matlab, if you have the same, the code above will work.
Note, that the returned variable is a list; you'll have to format it to match the original shape of your data, in my case len(arr) is 307200 from a matrix of the size 15360 x 20.
Hi I am having an issue using unstack in python,
fileID= open('B1b1_t100000.beam','r');
npart = 1E6;
ncoord = 7;
coords = np.reshape(struct.unpack('d'*int(ncoord*npart),fileID.read()),(npart,ncoord));
fileID.close()
And I am getting the error
Traceback (most recent call last):
File "transfer_lev_B1.py", line 30, in <module>
coords = np.reshape(struct.unpack('d'*int(ncoord*npart),fileID.read()),(npart,ncoord));
struct.error: unpack requires a string argument of length 56000000
I cant really see where the problem is. The file byte size is 56000000. In a previous attempt with np=1E4 the code worked for a different file with the same format (less total lines). But i have the problem when i go to a larger file with more lines..
ok I solved my problem,
import struct
import numpy as np
import matplotlib.pyplot as plt
if __name__ == '__main__':
fileID= open('B1b1_t100000.beam','r');
npart = 1E6;
ncoord = 7;
coords=np.fromfile('B1b1_t100000.beam',dtype=np.float64);
coords=coords[:(npart*ncoord)];
coords=np.reshape(coords,(npart,ncoord));
fileID.close()
# Beam 1
b1_x=coords[:,0];
b1_y=coords[:,2];
b1_z=coords[:,4];
b1_px=coords[:,1];
b1_py=coords[:,3];
b1_deltap =coords[:,5];
beam1=np.array([b1_x,b1_px,b1_y,b1_py,b1_z,b1_deltap,coords[:,6]],np.float64);
beam1=beam1.T;
# Map applied and new coordinates calculated.
x_mod=np.sqrt(foc)*coords[:,0];
y_mod=np.sqrt(foc)*coords[:,2];
px_mod=np.sqrt(defoc)*coords[:,1];
py_mod=np.sqrt(defoc)*coords[:,3];
beam1_mod=np.array([x_mod,px_mod,y_mod,py_mod,b1_z,b1_deltap,coords[:,6]],np.float64);
beam1_mod=beam1_mod.T;
#---------------Check shape of matrix----------------
#print coords.shape
# print (beam1_mod).shape
# print beam1.shape
# print 'beam1= \n', beam1
# print 'modified \n', beam1_mod
#----------------------------------------------------
# New coordinates printed to binary file.
fileMod=open("B1b1_t100000_mod.beam","w");
beam1_mod.tofile(fileMod);
fileMod.close()
I am attempting the following program.
from numpy import zeros,loadtxt
from pylab import plot,xlim,show
from cmath import exp,pi
def dft(y):
N = len(y)
c = zeros(N//2+1,complex)
for k in range(N//2+1):
for n in range(N):
c[k] += y[n]*exp(-2j*pi*k*n/N)
return c
y = loadtxt("pitch.txt",float)
c = dft(y)
plot(abs(c))
xlim(0,500)
show()
However, when I attempt to run the program, I receive an error code for the line 13:
y = loadtxt("pitch.txt",float)
File "C:\Python32\lib\site-packages\numpy\lib\npyio.py", line 689, in loadtxt
fh = iter(open(fname, 'U'))
IOError: [Errno 2] No such file or directory: 'pitch.txt'
I was given a file that has all the resources needed to run the program, and I uploaded them into the same folders I have my Python program saved in. The pitch.txt file is a text file with a single column of numbers. I'm wondering if there is something wrong with the written program, or do did I upload the files in the wrong.
I'm trying to convert binary files written in C to and from HDF5 files using Python.
To read the binary file Python works like this:
pos=np.fromfile(f, count=npt*3, dtype='f4').reshape((npt, 3))
To write the same thing I've tried, without success, array.tofile() and now I'm tryin to use ctypes like that (stitching together different answers found on the web):
import ctypes as c
print "Loading C libraries with ctype"
libc = c.CDLL("libc.so.6") # Linux
# fopen()
libc.fopen.restype = c.c_void_p
def errcheck(res, func, args):
if not res: raise IOError
return res
libc.fopen.errcheck = errcheck
# errcheck() could be similarly defined for `fwrite`, `fclose`
c_int_p = c.POINTER(c.c_int)
c_float_p = c.POINTER(c.c_float)
c_double_p = c.POINTER(c.c_double)
def c_write(data, f, numpy_type, c_type_p, nbyte, count):
data = data.astype(numpy_type)
data_p = data.ctypes.data_as(c_type_p)
nitems = libc.fwrite(data_p, nbyte, count, f)
if nitems != data.size: # not all data were written
print "Not all data were written, exit..."
sys.exit()
c_write(pos, f, np.int32, c_int_p, 4, npart.size)
You should probably look into the struct module, it's awesome for packing and unpacking data at the lowest byte-per-byte level.