Develop Reference

Bitplane decomposition of 16 bit two's complement signed integer signal data?

I am trying to do bit plane decomposition of a 16 bit two's complement signed integer signal data(electrocardiogram) in python so i will get 16 signal data bit plane. I know how to decompose an 8 bit unsigned integer image signal, and i reimplement the code in this problem. I thought that i am supposed to get bit plane data with its values containing negative because it is originaly a 16 bit signed integer, but i got resulting 16 bit unsigned integer signal not 16 bit signed integer signal.
Here's my code:
import numpy as np
def intToTcbin16(value):
return format(value % (1 << 16), '016b')
def Tcbin16ToInt(bin):
while len(bin)<16 :
bin = '0'+bin
if bin[0] == '0':
return int(bin, 2)
else:
return -1 * (int(''.join('1' if x == '0' else '0' for x in bin), 2) + 1)
def bitplanedecomposesignal(ecgdat):
lst = []
for j in range(len(ecgdat)):
lst.append(intToTcbin16(ecgdat[j]))
sixteen = (np.array([Tcbin16ToInt(i[0]) for i in lst],dtype = np.int16)*32768)
fiveteen = (np.array([Tcbin16ToInt(i[1]) for i in lst],dtype = np.int16)*16384)
fourteen = (np.array([Tcbin16ToInt(i[2]) for i in lst],dtype = np.int16)*8192)
thirteen = (np.array([Tcbin16ToInt(i[3]) for i in lst],dtype = np.int16)*4096)
twelve = (np.array([Tcbin16ToInt(i[4]) for i in lst],dtype = np.int16)*2048)
eleven = (np.array([Tcbin16ToInt(i[5]) for i in lst],dtype = np.int16)*1024)
ten = (np.array([Tcbin16ToInt(i[6]) for i in lst],dtype = np.int16)*512)
nine = (np.array([Tcbin16ToInt(i[7]) for i in lst],dtype = np.int16)*256)
eight = (np.array([Tcbin16ToInt(i[8]) for i in lst],dtype = np.int16)*128)
seven = (np.array([Tcbin16ToInt(i[9]) for i in lst],dtype = np.int16)*64)
six = (np.array([Tcbin16ToInt(i[10]) for i in lst],dtype = np.int16)*32)
five = (np.array([Tcbin16ToInt(i[11]) for i in lst],dtype = np.int16)*16)
four = (np.array([Tcbin16ToInt(i[12]) for i in lst],dtype = np.int16)*8)
three = (np.array([Tcbin16ToInt(i[13]) for i in lst],dtype = np.int16)*4)
two = (np.array([Tcbin16ToInt(i[14]) for i in lst],dtype = np.int16)*2)
one = (np.array([Tcbin16ToInt(i[15]) for i in lst],dtype = np.int16)*1)
return sixteen,fiveteen,fourteen,thirteen,twelve,eleven,ten,nine,eight,seven,six,five,four,three,two,one
here is the signal plot before decomposition:
for example, here is the 16th bitplane signal plot after decomposition:
What did i do wrong?how to do it right?and how to recompose it back?

In the sixteen line, change the 32768 to -32768. Everything else looks right.
Like you said, the planes of the existing bitplanedecomposesignal() code reconstruct the value as if it were unsigned 16-bit data rather than signed. However, if the most significant bit is on, then the represented value is negative, and we should subtract 2^16 = 65536 from the unsigned value. So the most significant bit should contribute 32768 - 65536 = -32768 rather than +32768.
Example:
value = −32700 decimal
= 1000000001000100 binary int16
↑ ↑ ↑
−2^15 2^6 2^2
−2^15 + 2^6 + 2^2 = −32700 decimal = value
Side comment: Numpy has good efficient bitwise functions that you might find useful. I'd consider using np.bitwise_and to extract bitplanes.

Convert Bytes to Floating Point Numbers WITHOUT using STRUCT

I'm trying to write my "personal"(without using any modules or functions : struct,float....,int....,...) python version of STL binary file reader, according to WIKIPEDIA : A binary STL file contains :
a 80-character (byte) headern which is generally ignored.
a 4-byte unsigned integer indicating the number of triangular facets in the file.
Each triangle is described by twelve 32-bit floating-point numbers: three for the normal and then three for the X/Y/Z coordinate of each vertex – just as with the ASCII version of STL. After these follows a 2-byte ("short") unsigned integer that is the "attribute byte count" – in the standard format, this should be zero because most software does not understand anything else. (((3+3+3)+3)*4+2=50 bytes for each point)
--Floating-point numbers are represented as IEEE floating-point numbers and are assumed to be little-endian--
With the help of two saviors I discovered how unsigned integers are stored, I can figure out the number of triangular facets in the file with 3 methods (computed by hand ) :
def int_from_bytes(inbytes): # ShadowRanger's
res = 0
for i, b in enumerate(inbytes):
res |= b << (i * 8)
return res
or
def int_from_bytes(inbytes): # ShadowRanger's
res = 0
for b in inbytes:
res <<= 8 # Adjust bytes seen so far to make room for new byte
res |= b # Mask in new byte
return res
or
def unsigned_int(s): # Robᵩ's
result = 0
for ch in s[::-1]:
result *= 256
result += ch
return result
Now I have to convert the rest of the file (3rd item in the list):Floating-point numbers
for the first point the 50-bytes are :
b'\x9a'b'\xa3' b'\x14' b'\xbe' b'\x05' b'$' b'\x85' b'\xbe' b'N' b'b'
b't' b'?' b'\xcd' b'\xa6' b'\x04' b'\xc4' b'\xfb' b';' b'\xd4' b'\xc1'
b'\x84' b'w' b'\x81' b'A' b'\xcd' b'\xa6' b'\x04' b'\xc4' b'\xa5' b'\x15'
b'\xd3' b'\xc1' b'\xb2' b'\xc7' b'\x81' b'A' b'\xef' b'\xa6' b'\x04'
b'\xc4' b'\x81' b'\x14' b'\xd3' b'\xc1' b'Y' b'\xc7' b'\x81' b'A' b'\x00'
b'\x00'
How can I convert this by hand, What is the principle of the representation, what rules should I know to do the conversion by hand (some bytes don't start with \x ). ?
Thank you for your time.

Like this:
def int_from_bytes(inbytes):
res = 0
shft = 0
for b in inbytes:
res |= ord(b) << shft
shft += 8
return res
def float_from_bytes(inbytes):
bits = int_from_bytes(inbytes)
mantissa = ((bits&8388607)/8388608.0)
exponent = (bits>>23)&255
sign = 1.0 if bits>>31 ==0 else -1.0
if exponent != 0:
mantissa+=1.0
elif mantissa==0.0:
return sign*0.0
return sign*pow(2.0,exponent-127)*mantissa
print float_from_bytes('\x9a\xa3\x14\xbe')
print float_from_bytes('\x00\x00\x00\x40')
print float_from_bytes('\x00\x00\xC0\xbf')
output:
-0.145155340433
2.0
-1.5
The format is IEEE-754 floating point. Try this out to see what each bit means: https://www.h-schmidt.net/FloatConverter/IEEE754.html

Convert base-2 binary number string to int

I'd simply like to convert a base-2 binary number string into an int, something like this:
>>> '11111111'.fromBinaryToInt()
255
Is there a way to do this in Python?

You use the built-in int() function, and pass it the base of the input number, i.e. 2 for a binary number:
>>> int('11111111', 2)
255
Here is documentation for Python 2, and for Python 3.

Just type 0b11111111 in python interactive interface:
>>> 0b11111111
255

Another way to do this is by using the bitstring module:
>>> from bitstring import BitArray
>>> b = BitArray(bin='11111111')
>>> b.uint
255
Note that the unsigned integer (uint) is different from the signed integer (int):
>>> b.int
-1
Your question is really asking for the unsigned integer representation; this is an important distinction.
The bitstring module isn't a requirement, but it has lots of performant methods for turning input into and from bits into other forms, as well as manipulating them.

Using int with base is the right way to go. I used to do this before I found int takes base also. It is basically a reduce applied on a list comprehension of the primitive way of converting binary to decimal ( e.g. 110 = 2**0 * 0 + 2 ** 1 * 1 + 2 ** 2 * 1)
add = lambda x,y : x + y
reduce(add, [int(x) * 2 ** y for x, y in zip(list(binstr), range(len(binstr) - 1, -1, -1))])

If you wanna know what is happening behind the scene, then here you go.
class Binary():
def __init__(self, binNumber):
self._binNumber = binNumber
self._binNumber = self._binNumber[::-1]
self._binNumber = list(self._binNumber)
self._x = [1]
self._count = 1
self._change = 2
self._amount = 0
print(self._ToNumber(self._binNumber))
def _ToNumber(self, number):
self._number = number
for i in range (1, len (self._number)):
self._total = self._count * self._change
self._count = self._total
self._x.append(self._count)
self._deep = zip(self._number, self._x)
for self._k, self._v in self._deep:
if self._k == '1':
self._amount += self._v
return self._amount
mo = Binary('101111110')

Here's another concise way to do it not mentioned in any of the above answers:
>>> eval('0b' + '11111111')
255
Admittedly, it's probably not very fast, and it's a very very bad idea if the string is coming from something you don't have control over that could be malicious (such as user input), but for completeness' sake, it does work.

A recursive Python implementation:
def int2bin(n):
return int2bin(n >> 1) + [n & 1] if n > 1 else [1]

If you are using python3.6 or later you can use f-string to do the
conversion:
Binary to decimal:
>>> print(f'{0b1011010:#0}')
90
>>> bin_2_decimal = int(f'{0b1011010:#0}')
>>> bin_2_decimal
90
binary to octal hexa and etc.
>>> f'{0b1011010:#o}'
'0o132' # octal
>>> f'{0b1011010:#x}'
'0x5a' # hexadecimal
>>> f'{0b1011010:#0}'
'90' # decimal
Pay attention to 2 piece of information separated by colon.
In this way, you can convert between {binary, octal, hexadecimal, decimal} to {binary, octal, hexadecimal, decimal} by changing right side of colon[:]
:#b -> converts to binary
:#o -> converts to octal
:#x -> converts to hexadecimal
:#0 -> converts to decimal as above example
Try changing left side of colon to have octal/hexadecimal/decimal.

For large matrix (10**5 rows and up) it is better to use a vectorized matmult. Pass in all rows and cols in one shot. It is extremely fast. There is no looping in python here. I originally designed it for converting many binary columns like 0/1 for like 10 different genre columns in MovieLens into a single integer for each example row.
def BitsToIntAFast(bits):
m,n = bits.shape
a = 2**np.arange(n)[::-1] # -1 reverses array of powers of 2 of same length as bits
return bits # a

For the record to go back and forth in basic python3:
a = 10
bin(a)
# '0b1010'
int(bin(a), 2)
# 10
eval(bin(a))
# 10

Using Python How can I read the bits in a byte?

I have a file where the first byte contains encoded information. In Matlab I can read the byte bit by bit with var = fread(file, 8, 'ubit1'), and then retrieve each bit by var(1), var(2), etc.
Is there any equivalent bit reader in python?

Read the bits from a file, low bits first.
def bits(f):
bytes = (ord(b) for b in f.read())
for b in bytes:
for i in xrange(8):
yield (b >> i) & 1
for b in bits(open('binary-file.bin', 'r')):
print b

The smallest unit you'll be able to work with is a byte. To work at the bit level you need to use bitwise operators.
x = 3
#Check if the 1st bit is set:
x&1 != 0
#Returns True
#Check if the 2nd bit is set:
x&2 != 0
#Returns True
#Check if the 3rd bit is set:
x&4 != 0
#Returns False

With numpy it is easy like this:
Bytes = numpy.fromfile(filename, dtype = "uint8")
Bits = numpy.unpackbits(Bytes)
More info here:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromfile.html

You won't be able to read each bit one by one - you have to read it byte by byte. You can easily extract the bits out, though:
f = open("myfile", 'rb')
# read one byte
byte = f.read(1)
# convert the byte to an integer representation
byte = ord(byte)
# now convert to string of 1s and 0s
byte = bin(byte)[2:].rjust(8, '0')
# now byte contains a string with 0s and 1s
for bit in byte:
print bit

Joining some of the previous answers I would use:
[int(i) for i in "{0:08b}".format(byte)]
For each byte read from the file. The results for an 0x88 byte example is:
>>> [int(i) for i in "{0:08b}".format(0x88)]
[1, 0, 0, 0, 1, 0, 0, 0]
You can assign it to a variable and work as per your initial request.
The "{0.08}" is to guarantee the full byte length

To read a byte from a file: bytestring = open(filename, 'rb').read(1). Note: the file is opened in the binary mode.
To get bits, convert the bytestring into an integer: byte = bytestring[0] (Python 3) or byte = ord(bytestring[0]) (Python 2) and extract the desired bit: (byte >> i) & 1:
>>> for i in range(8): (b'a'[0] >> i) & 1
...
1
0
0
0
0
1
1
0
>>> bin(b'a'[0])
'0b1100001'

There are two possible ways to return the i-th bit of a byte. The "first bit" could refer to the high-order bit or it could refer to the lower order bit.
Here is a function that takes a string and index as parameters and returns the value of the bit at that location. As written, it treats the low-order bit as the first bit. If you want the high order bit first, just uncomment the indicated line.
def bit_from_string(string, index):
i, j = divmod(index, 8)
# Uncomment this if you want the high-order bit first
# j = 8 - j
if ord(string[i]) & (1 << j):
return 1
else:
return 0
The indexing starts at 0. If you want the indexing to start at 1, you can adjust index in the function before calling divmod.
Example usage:
>>> for i in range(8):
>>> print i, bit_from_string('\x04', i)
0 0
1 0
2 1
3 0
4 0
5 0
6 0
7 0
Now, for how it works:
A string is composed of 8-bit bytes, so first we use divmod() to break the index into to parts:
i: the index of the correct byte within the string
j: the index of the correct bit within that byte
We use the ord() function to convert the character at string[i] into an integer type. Then, (1 << j) computes the value of the j-th bit by left-shifting 1 by j. Finally, we use bitwise-and to test if that bit is set. If so return 1, otherwise return 0.

Supposing you have a file called bloom_filter.bin which contains an array of bits and you want to read the entire file and use those bits in an array.
First create the array where the bits will be stored after reading,
from bitarray import bitarray
a=bitarray(size) #same as the number of bits in the file
Open the file,
using open or with, anything is fine...I am sticking with open here,
f=open('bloom_filter.bin','rb')
Now load all the bits into the array 'a' at one shot using,
f.readinto(a)
'a' is now a bitarray containing all the bits

This is pretty fast I would think:
import itertools
data = range(10)
format = "{:0>8b}".format
newdata = (False if n == '0' else True for n in itertools.chain.from_iterable(map(format, data)))
print(newdata) # prints tons of True and False

I think this is a more pythonic way:
a = 140
binary = format(a, 'b')
The result of this block is:
'10001100'
I was to get bit planes of the image and this function helped me to write this block:
def img2bitmap(img: np.ndarray) -> list:
if img.dtype != np.uint8 or img.ndim > 2:
raise ValueError("Image is not uint8 or gray")
bit_mat = [np.zeros(img.shape, dtype=np.uint8) for _ in range(8)]
for row_number in range(img.shape[0]):
for column_number in range(img.shape[1]):
binary = format(img[row_number][column_number], 'b')
for idx, bit in enumerate("".join(reversed(binary))[:]):
bit_mat[idx][row_number, column_number] = 2 ** idx if int(bit) == 1 else 0
return bit_mat
Also by this block, I was able to make primitives image from extracted bit planes
img = cv2.imread('test.jpg', cv2.IMREAD_GRAYSCALE)
out = img2bitmap(img)
original_image = np.zeros(img.shape, dtype=np.uint8)
for i in range(original_image.shape[0]):
for j in range(original_image.shape[1]):
for data in range(8):
x = np.array([original_image[i, j]], dtype=np.uint8)
data = np.array([data], dtype=np.uint8)
flag = np.array([0 if out[data[0]][i, j] == 0 else 1], dtype=np.uint8)
mask = flag << data[0]
x[0] = (x[0] & ~mask) | ((flag[0] << data[0]) & mask)
original_image[i, j] = x[0]

Find maximum signed short integer in python

How do I get the maximum signed short integer in Python (i.e. SHRT_MAX in C's limits.h)?
I want to normalize samples from a single channel of a *.wav file, so instead of a bunch of 16-bit signed integers, I want a bunch of floats between 1 and -1. Here's what I've got (the pertinent code is in the normalized_samples() function):
def samples(clip, chan_no = 0):
# *.wav files generally come in 8-bit unsigned ints or 16-bit signed ints
# python's wave module gives sample width in bytes, so STRUCT_FMT
# basically converts the wave.samplewidth into a struct fmt string
STRUCT_FMT = { 1 : 'B',
2 : 'h' }
for i in range(clip.getnframes()):
yield struct.unpack(STRUCT_FMT[clip.getsampwidth()] * clip.getnchannels(),
clip.readframes(1))[chan_no]
def normalized_samples(clip, chan_no = 0):
for sample in samples(clip, chan_no):
yield float(sample) / float(32767) ### THIS IS WHERE I NEED HELP

GregS is right, this is not the right way to solve the problem. If your samples are known 8 or 16 bit, you don't want to be dividing them by a number that varies by platform.
You may be running into trouble because a signed 16-bit int actually ranges from -32768 to 32767. Dividing by 32767 is going to give you < -1 in the extreme negative case.
Try this:
yield float(sample + 2**15) / 2**15 - 1.0

Here is a way using cython
getlimit.py
import pyximport; pyximport.install()
import limits
print limits.shrt_max
limits.pyx
import cython
cdef extern from "limits.h":
cdef int SHRT_MAX
shrt_max = SHRT_MAX

in module sys, sys.maxint. Though I'm not sure that is the correct way to solve your problem.

I can't imagine circumstances on a modern computer (i.e. one that uses 2's complement integers) where this would fail:
assert -32768 <= signed_16_bit_integer <= 32767
To do exactly what you asked for:
if signed_16_bit_integer >= 0:
afloat = signed_16_bit_integer / 32767.0
else:
afloat = signed_16_bit_integer / -32768.0
Having read your code a bit more closely: you have sample_width_in_bytes so just divide by 255 or 256 if it's B and by 32768 if it's h

#!/usr/bin/env python2
# maximums.py
####################################333#########################
B16_MAX = (1 << 16) - 1
B15_MAX = (1 << 15) - 1
B08_MAX = (1 << 8) - 1
B07_MAX = (1 << 7) - 1
print
print "hex(B16_MAX) =",hex(B16_MAX) # 0xffff
print "hex(B15_MAX) =",hex(B15_MAX) # 0x7fff
print "hex(B08_MAX) =",hex(B08_MAX) # 0xff
print "hex(B07_MAX) =",hex(B07_MAX) # 0x7f
print
####################################333#########################
UBYTE2_MAX = B16_MAX
SBYTE2_MAX = B15_MAX
UBYTE1_MAX = B08_MAX
SBYTE1_MAX = B07_MAX
print
print "UBYTE2_MAX =",UBYTE2_MAX # 65535
print "SBYTE2_MAX =",SBYTE2_MAX # 32767
print "UBYTE1_MAX =",UBYTE1_MAX # 255
print "SBYTE1_MAX =",SBYTE1_MAX # 127
print
####################################333#########################
USHRT_MAX = UBYTE2_MAX
SHRT_MAX = SBYTE2_MAX
CHAR_MAX = UBYTE1_MAX
BYTE_MAX = SBYTE1_MAX
print
print "USHRT_MAX =",USHRT_MAX # 65535
print " SHRT_MAX =", SHRT_MAX # 32767
print " CHAR_MAX =", CHAR_MAX # 255
print " BYTE_MAX =", BYTE_MAX # 127
print
####################################333#########################