Develop Reference

Unpack IEEE 754 Floating Point Number

I am reading two 16 bit registers from a tcp client using the pymodbus module. The two registers make up a 32 bit IEEE 754 encoded floating point number. Currently I have the 32 bit binary value of the registers shown in the code below.
start_address = 0x1112
reg_count = 2
client = ModbusTcpClient(<IP_ADDRESS>)
response = client.read_input_registers(start_address,reg_count)
reg_1 = response.getRegister(0)<<(16 - (response.getRegister(0).bit_length())) #Get in 16 bit format
reg_2 = response.getRegister(1)<<(16 - (response.getRegister(1).bit_length())) #Get in 16 bit format
volts = (reg_1 << 16) | reg_2 #Get the 32 bit format
The above works fine to get the encoded value the problem is decoding it. I was going to code something like in this video but I came across the 'f' format in the struct module for IEEE 754 encoding. I tried decode the 32 bit float stored in volts in the code above using the unpack method in the struct module but ran into the following errors.
val = struct.unpack('f',volts)
>>> TypeError: a bytes-like object is required, not 'int'
Ok tried convert it to a 32 bit binary string.
temp = bin(volts)
val = struct.unpack('f',temp)
>>> TypeError: a bytes-like object is required, not 'str'
Tried to covert it to a bytes like object as in this post and format in different ways.
val = struct.unpack('f',bytes(volts))
>>> TypeError: string argument without an encoding
temp = "{0:b}".format(volts)
val = struct.unpack('f',temp)
>>> ValueError: Unknown format code 'b' for object of type 'str'
val = struct.unpack('f',volts.encode())
>>> struct.error: unpack requires a buffer of 4 bytes
Where do I add this buffer and where in the documentation does it say I need this buffer with the unpack method? It does say in the documentation
The string must contain exactly the amount of data required by the format (len(string) must equal calcsize(fmt)).
The calcsize(fmt) function returns a value in bytes but the len(string) returns a value of the length of the string, no?
Any suggestions are welcome.
EDIT
There is a solution to decoding below however a better solution to obtaining the 32 bit register value from the two 16 bit register values is shown below compared to the original in the question.
start_address = 0x1112
reg_count = 2
client = ModbusTcpClient(<IP_ADDRESS>)
response = client.read_input_registers(start_address,reg_count)
reg_1 = response.getRegister(0)
reg_2 = response.getRegister(1)
# Shift reg 1 by 16 bits
reg_1s = reg_1 << 16
# OR with the reg_2
total = reg_1s | reg_2

I found a solution to the problem using the BinaryPayloadDecoder.fromRegisters() from the pymodbus moudule instead of the struct module. Note that this solution is specific to the modbus smart meter device I am using as the byte order and word order of the registers could change in other devices. It may still work in other devices to decode registers but I would advise to read the documentation of the device first to be sure. I left in the comments in the code below but when I refer to page 24 this is just for my device.
from pymodbus.client.sync import ModbusTcpClient
from pymodbus.constants import Endian
from pymodbus.payload import BinaryPayloadDecoder
start_address = 0x1112
reg_count = 2
client = ModbusTcpClient(<IP_ADDRESS>)
response = client.read_input_registers(start_address,reg_count)
# The response will contain two registers making a 32 bit floating point number
# Use the BinaryPayloadDecoder.fromRegisters() function to decode
# The coding scheme for a 32 bit float is IEEE 754 https://en.wikipedia.org/wiki/IEEE_754
# The MS Bytes are stored in the first address and the LS bytes are stored in the second address,
# this corresponds to a big endian byte order (Second parameter in function)
# The documentation for the Modbus registers for the smart meter on page 24 says that
# the low word is the first priority, this correspond to a little endian word order (Third parameter in function)
decoder = BinaryPayloadDecoder.fromRegisters(response.registers, Endian.Big, wordorder=Endian.Little)
final_val = (decoder.decode_32bit_float())
client.close()
EDIT
Credit to juanpa-arrivillaga and chepner the problem can be solved using the struct module also with the byteorder='little'. The two functions in the code below can be used if the byteorder is little or if the byte order is big depending upon the implementation.
import struct
from pymodbus.client.sync import ModbusTcpClient
def big_endian(response):
reg_1 = response.getRegister(0)
reg_2 = response.getRegister(1)
# Shift reg 1 by 16 bits
reg_1s = reg_1 << 16
# OR with the reg_2
total = reg_1s | reg_2
return total
def little_endian(response):
reg_1 = response.getRegister(0)
reg_2 = response.getRegister(1)
# Shift reg 2 by 16 bits
reg_2s = reg_2 << 16
# OR with the reg_1
total = reg_2s | reg_1
return(total)
start_address = 0x1112
reg_count = 2
client = ModbusTcpClient(<IP_ADDRESS>)
response = client.read_input_registers(start_address,reg_count)
# Little
little = little_endian(response)
lit_byte = little.to_bytes(4,byteorder='little')
print(struct.unpack('f',lit_byte))
# Big
big = big_endian(response)
big_byte = big.to_bytes(4,byteorder='big')
print(struct.unpack('f',big_byte))

struct.pack_into requires more bytes then specified in format

Python struct.pack_into with format char 'x' requires more bytes.
I am trying to learn about python byte arrays to be able to write my own IP,TPC,UDP headers. I use the struct in python to pack and unpack binary data so the specified types given the format string.
ba2 = bytearray(2)
print(ba2, "The size: ", ba2.__len__())
struct.pack_into(">bx", ba2, 1, 1)
print(struct.unpack(">bx", ba2))
Now when I try to pack into a buffer of length 2 with ">bx" as format, according to above code, I get the error:
bytearray(b'\x00\x00') The size: 2
Traceback (most recent call last):
File "D:/User/Documents/Python/Network/Main.py", line 58, in <module>
bitoperations_bytes_bytearrays_test()
File "D:/User/Documents/Python/Network/Main.py", line 49, in bitoperations_bytes_bytearrays_test
struct.pack_into(">bx", ba2, 1, 1)
struct.error: pack_into requires a buffer of at least 2 bytes
but I have a byte array of 2 bytes.
What am I doing wrong?
And please reference to some documentation, if I have missed it (I have read the python doc, but may have missed it).
Edit:
Sorry if I was unclear. but i want to just change the second byte in the byte array. Thus the 'x' padd in the format.
And as stupid as i was it is just to exclude the 'x' in the format like thiss:
struct.pack_into(">b", ba2, 1, 1)
and the right packing will have ben made. With this output:
bytearray(b'\x00\x00') The size: 2
A pack with one byte shift: 0001
(0, 1)

You need one additional parameter for pack_into() function call. The third parameter is mandatory and it is offset in the target buffer (refer to https://docs.python.org/2/library/struct.html). Your format is also incorrect, because it just expects one byte. Following code fixes your problems:
import struct
ba2 = bytearray(2)
print(ba2, "The size: ", ba2.__len__())
struct.pack_into("bb", ba2, 0, 1, 1)
print(struct.unpack("bb", ba2))

And as stupid as i was it is just to exclude the 'x' in the format like thiss:
struct.pack_into(">b", ba2, 1, 1)
and the right packing will have ben made. With this output:
bytearray(b'\x00\x00') The size: 2
A pack with one byte shift: 0001
(0, 1)

12 bytes array to double python

Im trying to do this:
struct.unpack('d', barray[:HALF_BYTES])[0]
Where barray[:HALF_BYTES] is a 12 bytes array
But I'm getting this error:
Traceback (most recent call last):
File "random_input_sample_drawer.py", line 19, in <module>
print (struct.unpack('d', barray[:HALF_BYTES])[0])
struct.error: unpack requires a bytes object of length 8
How can I solve it ?

You could use:
int.from_bytes(barray[:HALF_BYTES], byteorder='big', signed=False)
to convert those 12 bytes to an integer, if thats what you are trying to do, assuming your barray is a bytearray() and you are using python 3.2 +

python conversion int into arbitrary number of bytes

I am facing a little corner case of the famous struct.pack.
The situation is the following: I have a dll with a thin layer wrapper to python. One of the python method in the wraper accept a byte array as argument. This byte array the representation of a register on a specific hardware bus. Each bus has different register width, typically 8, 16 and 24 bits wide (alignement is the same in all cases).
When calling this method I need to convert my value (whatever that is) to a byte array of 8/16 or 24bits. Such conversion is relatively easy with 8 or 16bits using the struct.pack:
byteList = struct.pack( '>B', regValue ) # For 8 bits case
byteList = struct.pack( '>H', regValue ) # for 16 bits case
I am now looking to make it flexible enough for all three cases 8/16 & 24 bits. I could use a mix of the two previous line to handle the three cases; but I find it quite ugly.
I was hoping this would work:
packformat = ">{0}B".format(regSize)
byteList = struct.pack( packformat, regValue )
But it is not the case as the struct.pack expect an equal amount of arguments.
Any idea how can I convert (neatly) my register value into an arbitrary number of bytes?

You are always packing unsigned integers, and only big endian to boot. Take a look at what happens when you pack them:
>>> import struct
>>> struct.pack('>B', 255)
'\xff'
>>> struct.pack('>H', 255)
'\x00\xff'
>>> struct.pack('>I', 255)
'\x00\x00\x00\xff'
Essentially the value is padded with null bytes at the start. Use this to your advantage:
>>> struct.pack('>I', 255)[-3:]
'\x00\x00\xff'
>>> struct.pack('>I', 255)[-2:]
'\x00\xff'
>>> struct.pack('>I', 255)[-1:]
'\xff'
You won't get an exception now, if your value is too large, but it would simplify your code enormously. You can always add a separate validation step:
def packRegister(value, size):
if value < 0 or value.bit_length() > size:
raise ValueError("Value won't fit in register of size {} bits".format(size))
return struct.pack('>I', value)[-(size // 8):]
Demo:
>>> packRegister(255, 8)
'\xff'
>>> packRegister(1023, 16)
'\x03\xff'
>>> packRegister(324353, 24)
'\x04\xf3\x01'
>>> packRegister(324353, 8)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in packRegister
ValueError: Value won't fit in register of size 8 bits

How to print values of a string full of "chaos question marks"

I'm debugging with python audio, having a hard time with the audio coding.
Here I have a string full of audio data, say, [10, 20, 100].
However the data is stored in a string variable,
data = "����������������"
I want to inspect the values of this string.
Below is the things I tried
Print as int
I tried to use print "%i" % data[0]
ended up with
Traceback (most recent call last):
File "wire.py", line 28, in <module>
print "%i" % data[i]
TypeError: %d format: a number is required, not str
Convert to int
int(data[0]) ended up with
Traceback (most recent call last):
File "wire.py", line 27, in <module>
print int(data[0])
ValueError: invalid literal for int() with base 10: '\xd1'
Any idea on this? I want to print the string in a numerical way since the string is actually an array of sound wave.
EDIT
All your answers turned out to be really helpful.
The string is actually generated from the microphone so I believe it to be raw wave form, or vibration data. Further this should be referred to the audio API document, PortAudio.
After looking into PortAudio, I find this helpful example.
** This routine will be called by the PortAudio engine when audio is needed.
** It may called at interrupt level on some machines so don't do anything
** that could mess up the system like calling malloc() or free().
static int patestCallback( const void *inputBuffer, void *outputBuffer,
unsigned long framesPerBuffer,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void *userData )
{
paTestData *data = (paTestData*)userData;
float *out = (float*)outputBuffer;
unsigned long i;
(void) timeInfo; /* Prevent unused variable warnings. */
(void) statusFlags;
(void) inputBuffer;
for( i=0; i<framesPerBuffer; i++ )
{
*out++ = data->sine[data->left_phase]; /* left */
*out++ = data->sine[data->right_phase]; /* right */
data->left_phase += 1;
if( data->left_phase >= TABLE_SIZE ) data->left_phase -= TABLE_SIZE;
data->right_phase += 3; /* higher pitch so we can distinguish left and right. */
if( data->right_phase >= TABLE_SIZE ) data->right_phase -= TABLE_SIZE;
}
return paContinue;
}
This indicates that there is some way that I can interpret the data as float

To be clear, your audio data is a byte string. The byte string is a representation of the bytes stored in the audio file. You are not going to simply be able to convert those bytes into meaningful values without knowing what is in the binary first.
As an example, the mp3 specification says that each mp3 contains header frames (described here: http://en.wikipedia.org/wiki/MP3). To read the header you would either need to use something like bitstring, or if you feel comfortable doing the bitwise manipulation yourself then you would just need to unpack an integer (4 bytes) and do some math to figure out the values of the 32 individual bits.
It really all depends on what you are trying to read, and how the data was generated. If you have whole byte numbers, then struct will serve you well.

If you're ok with the \xd1 mentioned above:
for item in data: print repr(item),
Note that for x in data will iterate over each value in the list rather than its location. If you want the location you can use for i in range(len(data)): ...
If you want them in numerical form, replace repr(item) with ord(item).

It is better if you use the new {}.format method:
data = "����������������"
print '{0}'.format(data[3])

You could use ord to map each byte to its numeric value between 0-255:
print map(ord, data)
Or, for Python 3 compatibility, do:
print([ord(c) for c in data])
It will also work with Unicode glyphs, which might not be what you want, so make sure you have a bytearray or an actual str or bytes object in Python 2.