Develop Reference

unpack_from requires a buffer of at least 784 bytes

I'm running the following function for an ML model.
def get_images(filename):
bin_file = open(filename, 'rb')
buf = bin_file.read() # all the file are put into memory
bin_file.close() # release the measure of operating system
index = 0
magic, num_images, num_rows, num_colums = struct.unpack_from(big_endian + four_bytes, buf, index)
index += struct.calcsize(big_endian + four_bytes)
images = [] # temp images as tuple
for x in range(num_images):
im = struct.unpack_from(big_endian + picture_bytes, buf, index)
index += struct.calcsize(big_endian + picture_bytes)
im = list(im)
for i in range(len(im)):
if im[i] > 1:
im[i] = 1
However, I am receiving an error at the line:
im = struct.unpack_from(big_endian + picture_bytes, buf, index)
With the error:
error: unpack_from requires a buffer of at least 784 bytes
I have noticed this error is only occurring at certain iterations. I cannot figure out why this is might be the case. The dataset is a standard MNIST dataset which is freely available online.
I have also looked through similar questions on SO (e.g. error: unpack_from requires a buffer) but they don't seem to resolve the issue.

You didn't include the struct formats in your mre so it is hard to say why you are getting the error. Either you are using a partial/corrupted file or your struct formats are wrong.
This answer uses the test file 't10k-images-idx3-ubyte.gz' and file formats found at http://yann.lecun.com/exdb/mnist/
Open the file and read it into a bytes object (gzip is used because of the file's type).
import gzip,struct
with gzip.open(r'my\path\t10k-images-idx3-ubyte.gz','rb') as f:
data = bytes(f.read())
print(len(data))
The file format spec says the header is 16 bytes (four 32 bit ints) - separate it from the pixels with a slice then unpack it
hdr,pixels = data[:16],data[16:]
magic, num_images, num_rows, num_cols = struct.unpack(">4L",hdr)
# print(len(hdr),len(pixels))
# print(magic, num_images, num_rows, num_cols)
There are a number of ways to iterate over the individual images.
img_size = num_rows * num_cols
imgfmt = "B"*img_size
for i in range(num_images):
start = i * img_size
end = start + img_size
img = pixels[start:end]
img = struct.unpack(imgfmt,img)
# do work on the img
Or...
imgfmt = "B"*img_size
for img in struct.iter_unpack(imgfmt, pixels):
img = [p if p == 0 else 1 for p in img]
The itertools grouper recipe would probably also work.

TypeError: only size-1 arrays can be converted to Python scalars - Earth Observation

I'm attempting to create a script that will take a .GTiff file as an argument input and then extract some information out of the file to create a stats.txt file that will give me the classID, fractional coverage and total number of pixels of that classID.
Thus far I believe I have everything I need but I keep running into the same error and my attempts to rectify the error haven't proven to be very fruitful.
#!/usr/bin/env python
import sys
import calendar
import os
import gdal
import numpy as np
from scipy.stats import mode
from IPython import embed
GDAL2NUMPY = { gdal.GDT_Byte : np.uint8,
gdal.GDT_UInt16 : np.uint16,
gdal.GDT_Int16 : np.int16,
gdal.GDT_UInt32 : np.uint32,
gdal.GDT_Int32 : np.int32,
gdal.GDT_Float32 : np.float32,
gdal.GDT_Float64 : np.float64,
gdal.GDT_CInt16 : np.complex64,
gdal.GDT_CInt32 : np.complex64,
gdal.GDT_CFloat32 : np.complex64,
gdal.GDT_CFloat64 : np.complex128
}
#Open the original training data .tif map file.
fname = sys.argv[1]
lc_dataset = gdal.Open(fname)
lc = lc_dataset.ReadAsArray()
lc = np.array(lc)
#Calculating total number of pixels with a valid Land Cover ID.
fill_value = 0
number_of_pixels = np.where(lc != fill_value)[0].shape[0]
#Get the number of classes and corresponding IDs.
lc_classes = np.unique(lc)
#Split each class into its contituante pixel and write result to file.
for classID in range(1, lc_classes):
lc_class_pixels = np.where(lc == classID)[0].shape[0]
FractionalCover = lc_class_pixels/number_of_pixels
f.write(classID, FractionalCoverage, lc_class_pixels)
f.close()
When I run this, it chuck up the following traceback:
Traceback (most recent call last):
File "GeneratingLCstats.py", line 45, in <module>
for classID in range(1, lc_classes):
TypeError: only size-1 arrays can be converted to Python scalars
I've attempted a few changes as I'm sure the error is related to numpy data and native python data interactions, but converting all my arrays to numpy arrays and attempting to reformat the code has proved in vain as the same error persists.
If anyone can suggest a fix that would be greatly appreciated!
Thanks.

Well, the function lc_classes = np.unique(lc) returns an array. When you try to write the for loop as
for classID in range(1, lc_classes)
Here, lc_classes is an array and trying to give it as a bound for the range causes the error. If you want to iterate over the length of the array, you can modify the code to :
for classID in range(1, len(lc_classes))

Python Read part of large binary file

I have large binary file (size ~2.5Gb). It contains header (size 336 byte) and seismic signal data (x, y and z channels) with type int32. Count of discrete is 223 200 000.
I need read part of signal. For example, I want get part of signal in interval of discrete [216 000 000, 219 599 999].
I wrote the function:
def reading(path, start_moment, end_moment):
file_data = open(path, 'rb')
if start_moment is not None:
bytes_value = start_moment * 4 * 3
file_data.seek(336 + bytes_value)
else:
file_data.seek(336)
if end_moment is None:
try:
signals = np.fromfile(file_data, dtype=np.int32)
except MemoryError:
return None
finally:
file_data.close()
else:
moment_count = end_moment - start_moment + 1
try:
signals = np.fromfile(file_data, dtype=np.int32,
count=moment_count * 3)
except MemoryError:
return None
finally:
file_data.close()
channel_count = 3
signal_count = signals.shape[0] // channel_count
signals = np.reshape(signals, newshape=(signal_count, channel_count))
return signals
If I run script with the function in PyCharm IDE I get error:
Traceback (most recent call last): File
"D:/AppsBuilding/test/testReadBaikal8.py", line 41, in
signal_2 = reading(path=path, start_moment=216000000, end_moment=219599999) File
"D:/AppsBuilding/test/testReadBaikal8.py", line 27, in reading
count=moment_count * 3) OSError: obtaining file position failed
But if I run script with parameters: start_moment=7200000, end_moment=10799999 all ok.
On my PC was installed Windows7 32bit. Memory size is 1.95Gb
Please, help me resolve this problem.

Divide the file into small segments, freeing memory after each small
piece of content is processed
def read_in_block(file_path):
BLOCK_SIZE = 1024
with open(file_path, "r") as f:
while True:
block = f.read(BLOCK_SIZE)
if block:
yield block
else:
return
print block

I don't use Numpy but I don't see anything obviously wrong with your code. However, you say the file is approximately 2.5 GB in size. A triplet index of 219,599,999 requires a file at least 2.45 GB in size:
$ calc
; 219599999 * 4 * 3
2635199988
; 2635199988 / 1024^3
~2.45422123745083808899
Are you sure your file is really that large?
I also don't use MS Windows but the following toy programs work for me. The first creates a data file that mimics the structure of yours. The second shows that it can read the final data triplet. What happens if you run these on your system?
fh = open('x', 'wb')
fh.write(b'0123456789')
for i in range(0, 1000):
s = bytes('{:03d}'.format(i), 'ascii')
fh.write(b'a' + s + b'b' + s + b'c' + s)
Read the data from file x:
fh = open('x', 'rb')
triplet = 999
fh.seek(10 + triplet * 3 * 4)
data = fh.read(3 * 4)
print(data)

Python Binary files.

Hi I am having an issue using unstack in python,
fileID= open('B1b1_t100000.beam','r');
npart = 1E6;
ncoord = 7;
coords = np.reshape(struct.unpack('d'*int(ncoord*npart),fileID.read()),(npart,ncoord));
fileID.close()
And I am getting the error
Traceback (most recent call last):
File "transfer_lev_B1.py", line 30, in <module>
coords = np.reshape(struct.unpack('d'*int(ncoord*npart),fileID.read()),(npart,ncoord));
struct.error: unpack requires a string argument of length 56000000
I cant really see where the problem is. The file byte size is 56000000. In a previous attempt with np=1E4 the code worked for a different file with the same format (less total lines). But i have the problem when i go to a larger file with more lines..

ok I solved my problem,
import struct
import numpy as np
import matplotlib.pyplot as plt
if __name__ == '__main__':
fileID= open('B1b1_t100000.beam','r');
npart = 1E6;
ncoord = 7;
coords=np.fromfile('B1b1_t100000.beam',dtype=np.float64);
coords=coords[:(npart*ncoord)];
coords=np.reshape(coords,(npart,ncoord));
fileID.close()
# Beam 1
b1_x=coords[:,0];
b1_y=coords[:,2];
b1_z=coords[:,4];
b1_px=coords[:,1];
b1_py=coords[:,3];
b1_deltap =coords[:,5];
beam1=np.array([b1_x,b1_px,b1_y,b1_py,b1_z,b1_deltap,coords[:,6]],np.float64);
beam1=beam1.T;
# Map applied and new coordinates calculated.
x_mod=np.sqrt(foc)*coords[:,0];
y_mod=np.sqrt(foc)*coords[:,2];
px_mod=np.sqrt(defoc)*coords[:,1];
py_mod=np.sqrt(defoc)*coords[:,3];
beam1_mod=np.array([x_mod,px_mod,y_mod,py_mod,b1_z,b1_deltap,coords[:,6]],np.float64);
beam1_mod=beam1_mod.T;
#---------------Check shape of matrix----------------
#print coords.shape
# print (beam1_mod).shape
# print beam1.shape
# print 'beam1= \n', beam1
# print 'modified \n', beam1_mod
#----------------------------------------------------
# New coordinates printed to binary file.
fileMod=open("B1b1_t100000_mod.beam","w");
beam1_mod.tofile(fileMod);
fileMod.close()

Importing sound files into Python as NumPy arrays (alternatives to audiolab)

I've been using Audiolab to import sound files in the past, and it worked quite well. However:
It doesn't support some formats, like mp3, because the underlying libsndfile refuses to support them
It doesn't work in Python 2.6 under Windows, and the author is not around to fix it
-
In [2]: from scikits import audiolab
--------------------------------------------------------------------
ImportError Traceback (most recent call last)
C:\Python26\Scripts\<ipython console> in <module>()
C:\Python26\lib\site-packages\scikits\audiolab\__init__.py in <module>()
23 __version__ = _version
24
---> 25 from pysndfile import formatinfo, sndfile
26 from pysndfile import supported_format, supported_endianness, \
27 supported_encoding, PyaudioException, \
C:\Python26\lib\site-packages\scikits\audiolab\pysndfile\__init__.py in <module>()
----> 1 from _sndfile import Sndfile, Format, available_file_formats, available_encodings
2 from compat import formatinfo, sndfile, PyaudioException, PyaudioIOError
3 from compat import supported_format, supported_endianness, supported_encoding
ImportError: DLL load failed: The specified module could not be found.``
So I would like to either:
Figure out why it's not working in 2.6 (something wrong with _sndfile.pyd?) and maybe find a way to extend it to work with unsupported formats
Find a complete replacement for audiolab

Audiolab is working for me on Ubuntu 9.04 with Python 2.6.2, so it might be a Windows problem. In your link to the forum, the author also suggests that it is a Windows error.
In the past, this option has worked for me, too:
from scipy.io import wavfile
fs, data = wavfile.read(filename)
Just beware that data may have int data type, so it is not scaled within [-1,1). For example, if data is int16, you must divide data by 2**15 to scale within [-1,1).

Sox http://sox.sourceforge.net/ can be your friend for this. It can read many many different formats and output them as raw in whatever datatype you prefer. In fact, I just wrote the code to read a block of data from an audio file into a numpy array.
I decided to go this route for portability (sox is very widely available) and to maximize the flexibility of input audio types I could use. Actually, it seems from initial testing that it isn't noticeably slower for what I'm using it for... which is reading short (a few seconds) of audio from very long (hours) files.
Variables you need:
SOX_EXEC # the sox / sox.exe executable filename
filename # the audio filename of course
num_channels # duh... the number of channels
out_byps # Bytes per sample you want, must be 1, 2, 4, or 8
start_samp # sample number to start reading at
len_samp # number of samples to read
The actual code is really simple. If you want to extract the whole file, you can remove the start_samp, len_samp, and 'trim' stuff.
import subprocess # need the subprocess module
import numpy as NP # I'm lazy and call numpy NP
cmd = [SOX_EXEC,
filename, # input filename
'-t','raw', # output file type raw
'-e','signed-integer', # output encode as signed ints
'-L', # output little endin
'-b',str(out_byps*8), # output bytes per sample
'-', # output to stdout
'trim',str(start_samp)+'s',str(len_samp)+'s'] # only extract requested part
data = NP.fromstring(subprocess.check_output(cmd),'<i%d'%(out_byps))
data = data.reshape(len(data)/num_channels, num_channels) # make samples x channels
PS: Here is code to read stuff from audio file headers using sox...
info = subprocess.check_output([SOX_EXEC,'--i',filename])
reading_comments_flag = False
for l in info.splitlines():
if( not l.strip() ):
continue
if( reading_comments_flag and l.strip() ):
if( comments ):
comments += '\n'
comments += l
else:
if( l.startswith('Input File') ):
input_file = l.split(':',1)[1].strip()[1:-1]
elif( l.startswith('Channels') ):
num_channels = int(l.split(':',1)[1].strip())
elif( l.startswith('Sample Rate') ):
sample_rate = int(l.split(':',1)[1].strip())
elif( l.startswith('Precision') ):
bits_per_sample = int(l.split(':',1)[1].strip()[0:-4])
elif( l.startswith('Duration') ):
tmp = l.split(':',1)[1].strip()
tmp = tmp.split('=',1)
duration_time = tmp[0]
duration_samples = int(tmp[1].split(None,1)[0])
elif( l.startswith('Sample Encoding') ):
encoding = l.split(':',1)[1].strip()
elif( l.startswith('Comments') ):
comments = ''
reading_comments_flag = True
else:
if( other ):
other += '\n'+l
else:
other = l
if( output_unhandled ):
print >>sys.stderr, "Unhandled:",l
pass

FFmpeg supports mp3s and works on Windows (http://zulko.github.io/blog/2013/10/04/read-and-write-audio-files-in-python-using-ffmpeg/).
Reading an mp3 file:
import subprocess as sp
FFMPEG_BIN = "ffmpeg.exe"
command = [ FFMPEG_BIN,
'-i', 'mySong.mp3',
'-f', 's16le',
'-acodec', 'pcm_s16le',
'-ar', '44100', # ouput will have 44100 Hz
'-ac', '2', # stereo (set to '1' for mono)
'-']
pipe = sp.Popen(command, stdout=sp.PIPE, bufsize=10**8)
Format data into numpy array:
raw_audio = pipe.proc.stdout.read(88200*4)
import numpy
audio_array = numpy.fromstring(raw_audio, dtype="int16")
audio_array = audio_array.reshape((len(audio_array)/2,2))

In case you want to do this for MP3
Here's what I'm using: It uses pydub and scipy.
Full setup (on Mac, may differ on other systems):
import tempfile
import os
import pydub
import scipy
import scipy.io.wavfile
def read_mp3(file_path, as_float = False):
"""
Read an MP3 File into numpy data.
:param file_path: String path to a file
:param as_float: Cast data to float and normalize to [-1, 1]
:return: Tuple(rate, data), where
rate is an integer indicating samples/s
data is an ndarray(n_samples, 2)[int16] if as_float = False
otherwise ndarray(n_samples, 2)[float] in range [-1, 1]
"""
path, ext = os.path.splitext(file_path)
assert ext=='.mp3'
mp3 = pydub.AudioSegment.from_mp3(file_path)
_, path = tempfile.mkstemp()
mp3.export(path, format="wav")
rate, data = scipy.io.wavfile.read(path)
os.remove(path)
if as_float:
data = data/(2**15)
return rate, data
Credit to James Thompson's blog

I've been using PySoundFile lately instead of Audiolab. It installs easily with conda.
It does not support mp3, like most things. MP3 is no longer patented, so there's no reason why it can't support it; someone just has to write support into libsndfile.