Im in no way an experienced python programmer,thats why i believe there may be an obvious answer to this but i just can't wrap my head around the struct.pack and unpack.
i have the following code:
struct.pack("<"+"I"*elements, *self.buf[:elements])
I want to reverse the the packing of this, however im not sure how, i know that "<" means little endian and "I" is unsigned int and thats about it, im not sure how to use struct.unpack to reverse the packing.
struct.pack takes non-byte values (e.g. integers, strings, etc.) and converts them to bytes. And conversely, struct.unpack takes bytes and converts them to their 'higher-order' equivalents.
For example:
>>> from struct import pack, unpack
>>> packed = pack('hhl', 1, 2, 3)
>>> packed
b'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> unpacked = unpack('hhl', packed)
>>> unpacked
(1, 2, 3)
So in your instance, you have little-endian unsigned integers (elements many of them). You can unpack them using the same structure string (the '<' + 'I' * elements part) - e.g. struct.unpack('<' + 'I' * elements, value).
Example from: https://docs.python.org/3/library/struct.html
Looking at the documentation: https://docs.python.org/3/library/struct.html
obj = struct.pack("<"+"I"*elements, *self.buf[:elements])
struct.unpack("<"+"I"*elements, obj)
Does this work for you?
Related
I am trying to convert a number stored as a list of ints to a float type. I got the number via a serial console and want to reassemble it back together into a float.
The way I would do it in C is something like this:
bit_data = ((int16_t)byte_array[0] << 8) | byte_array[1];
result = (float)bit_data;
What I tried to use in python is a much more simple conversion:
result = int_list[0]*256.0 + int_list[1]
However, this does not preserve the sign of the result, as the C code does.
What is the right way to do this in python?
UPDATE:
Python version is 2.7.3.
My byte array has a length of 2.
in the python code byte_array is list of ints. I've renamed it to avoid misunderstanding. I can not just use the float() function because it will not preserve the sign of the number.
I'm a bit confused by what data you have, and how it is represented in Python. As I understand it, you have received two unsigned bytes over a serial connection, which are now represented by a list of two python ints. This data represents a big endian 16-bit signed integer, which you want to extract and turn into a float. eg. [0xFF, 0xFE] -> -2 -> -2.0
import array, struct
two_unsigned_bytes = [255, 254] # represented by ints
byte_array = array.array("B", two_unsigned_bytes)
# change above to "b" if the ints represent signed bytes ie. in range -128 to 127
signed_16_bit_int, = struct.unpack(">h", byte_array)
float_result = float(signed_16_bit_int)
I think what you want is the struct module.
Here's a round trip snippet:
import struct
sampleValue = 42.13
somebytes = struct.pack('=f', sampleValue)
print(somebytes)
result = struct.unpack('=f', somebytes)
print(result)
result may be surprising to you. unpack returns a tuple. So to get to the value you can do
result[0]
or modify the result setting line to be
result = struct.unpack('=f', some bytes)[0]
I personally hate that, so use the following instead
result , = struct.unpack('=f', some bytes) # tuple unpacking on assignment
The second thing you'll notice is that the value has extra digits of noise. That's because python's native floating point representation is double.
(This is python3 btw, adjust for using old versions of python as appropriate)
I am not sure I really understand what you are doing, but I think you got 4 bytes from a stream and know them to represent a float32 value. The way you handling this suggests big-endian byte-order.
Python has the struct package (https://docs.python.org/2/library/struct.html) to handle bytestreams.
import struct
stream = struct.pack(">f", 2/3.)
len(stream) # 4
reconstructed_float = struct.unpack(">f", stream)
Okay, so I think int_list isn't really just a list of ints. The ints are constrained to 0-255 and represent bytes that can be built into a signed integer. You then want to turn that into a float. The trick is to set the sign of the first byte properly and then procede much like you did.
float((-(byte_array[0]-127) if byte_array[0]>127 else byte_array[0])*256 + byte_array[1])
s = '\x01\x00\x02\x00\x00\x00'
struct.unpack('hi',s)
I expect to get (1,2), but instead get the error:
error: unpack requires a string argument of length 8
If I perform the two unpacks separately it works:
myshort = struct.unpack('h',s[:2])
myint = struct.unpack('i',s[2:])
Also, interestingly, it will accept it if the format string is 'ih' instead of 'hi'.
What am I missing?
This is because of C structure alignment. If you actually want your data items to remain unaligned, prefix a = sign before the formatted string
>>> s = '\x01\x00\x02\x00\x00\x00'
>>> struct.unpack('=hi',s)
(1, 2)
Refer the documentation 7.3.2.1. Byte Order, Size, and Alignment
I want to convert an integer value to a string of hex values, in little endian. For example, 5707435436569584000 would become '\x4a\xe2\x34\x4f\x4a\xe2\x34\x4f'.
All my googlefu is finding for me is hex(..) which gives me '0x4f34e24a4f34e180' which is not what I want.
I could probably manually split up that string and build the one I want but I'm hoping somone can point me to a better option.
You need to use the struct module:
>>> import struct
>>> struct.pack('<Q', 5707435436569584000)
'\x80\xe14OJ\xe24O'
>>> struct.pack('<Q', 5707435436569584202)
'J\xe24OJ\xe24O'
Here < indicates little-endian, and Q that we want to pack a unsigned long long (8 bytes).
Note that Python will use ASCII characters for any byte that falls within the printable ASCII range to represent the resulting bytestring, hence the 14OJ, 24O and J parts of the above result:
>>> struct.pack('<Q', 5707435436569584202).encode('hex')
'4ae2344f4ae2344f'
>>> '\x4a\xe2\x34\x4f\x4a\xe2\x34\x4f'
'J\xe24OJ\xe24O'
I know it is an old thread, but it is still useful. Here my two cents using python3:
hex_string = hex(5707435436569584202) # '0x4f34e24a4f34e180' as you said
bytearray.fromhex(hex_string[2:]).reverse()
So, the key is convert it to a bytearray and reverse it.
In one line:
bytearray.fromhex(hex(5707435436569584202)[2:])[::-1] # bytearray(b'J\xe24OJ\xe24O')
PS: You can treat "bytearray" data like "bytes" and even mix them with b'raw bytes'
Update:
As Will points in coments, you can also manage negative integers:
To make this work with negative integers you need to mask your input with your preferred int type output length. For example, -16 as a little endian uint32_t would be bytearray.fromhex(hex(-16 & (2**32-1))[2:])[::-1], which evaluates to bytearray(b'\xf0\xff\xff\xff')
I'm trying to convert the following perl code:
unpack(.., "Z*")
to python, however the lack of a "*" format modifier in struct.unpack() seems to make this impossible. Is there a way I can do this in python?
P.S. The "*" modifier in perl from the perldoc - Supplying a * for the repeat count instead of a number means to use however many items are left, ...
So although python has a numeric repeat count like perl, it seems to lack a * repeat count.
python's struct.unpack doesn't have the Z format
Z A null-terminated (ASCIZ) string, will be null padded.
i think this
unpack(.., "Z*")
would be:
data.split('\x00')
although that strips the nulls
I am assuming that you create the struct datatype and you know the size of the struct. If that is the case, then you can create a buffer allocated for that struct and the pack the value into the buffer. While unpacking, you can use the same buffer to unpack directly by just specifying the starting point.
For e.g.
import ctypes
import struct
s = struct.Struct('I')
b = ctypes.create_string_buffer(s.size)
s.pack_into(b, 0, 42)
s.unpack_from(b, 0)
You must calculate the repeat count yourself:
n = len(s) / struct.calcsize(your_fmt_string)
f = '%d%s' % (n, your_fmt_string)
data = struct.unpack(s, f)
I am assuming your_fmt_string doesn't unpack more than one element, and len(s) is perfectly divided by that element's packed size.
I have a long Hex string that represents a series of values of different types. I need to convert this Hex String into bytes or bytearray so that I can extract each value from the raw data. How can I do this?
For example, the string "ab" should convert to the bytes b"\xab" or equivalent byte array. Longer example:
>>> # what to use in place of `convert` here?
>>> convert("8e71c61de6a2321336184f813379ec6bf4a3fb79e63cd12b")
b'\x8eq\xc6\x1d\xe6\xa22\x136\x18O\x813y\xeck\xf4\xa3\xfby\xe6<\xd1+'
Suppose your hex string is something like
>>> hex_string = "deadbeef"
Convert it to a bytearray (Python 3 and 2.7):
>>> bytearray.fromhex(hex_string)
bytearray(b'\xde\xad\xbe\xef')
Convert it to a bytes object (Python 3):
>>> bytes.fromhex(hex_string)
b'\xde\xad\xbe\xef'
Note that bytes is an immutable version of bytearray.
Convert it to a string (Python ≤ 2.7):
>>> hex_data = hex_string.decode("hex")
>>> hex_data
"\xde\xad\xbe\xef"
There is a built-in function in bytearray that does what you intend.
bytearray.fromhex("de ad be ef 00")
It returns a bytearray and it reads hex strings with or without space separator.
provided I understood correctly, you should look for binascii.unhexlify
import binascii
a='45222e'
s=binascii.unhexlify(a)
b=[ord(x) for x in s]
Assuming you have a byte string like so
"\x12\x45\x00\xAB"
and you know the amount of bytes and their type you can also use this approach
import struct
bytes = '\x12\x45\x00\xAB'
val = struct.unpack('<BBH', bytes)
#val = (18, 69, 43776)
As I specified little endian (using the '<' char) at the start of the format string the function returned the decimal equivalent.
0x12 = 18
0x45 = 69
0xAB00 = 43776
B is equal to one byte (8 bit) unsigned
H is equal to two bytes (16 bit) unsigned
More available characters and byte sizes can be found here
The advantages are..
You can specify more than one byte and the endian of the values
Disadvantages..
You really need to know the type and length of data your dealing with
You can use the Codecs module in the Python Standard Library, i.e.
import codecs
codecs.decode(hexstring, 'hex_codec')
You should be able to build a string holding the binary data using something like:
data = "fef0babe"
bits = ""
for x in xrange(0, len(data), 2)
bits += chr(int(data[x:x+2], 16))
This is probably not the fastest way (many string appends), but quite simple using only core Python.
A good one liner is:
byte_list = map(ord, hex_string)
This will iterate over each char in the string and run it through the ord() function. Only tested on python 2.6, not too sure about 3.0+.
-Josh