Develop Reference

How can I convert from bytes to binary numbers in Python?

So I'm a total Python beginner and I got this byte object:
byte_obj = b'\x45\x10\x00\x4c\xcc\xde\x40\x00\x40\x06\x6c\x80\xc0\xa8\xd9\x17\x8d\x54\xda\x28'
But I have no idea how to put this in a binary number, I only know it's gonna have 32 bits.

You could try int.from_bytes(...), documented here e.g.:
>>> byte_obj = b'\x45\x10\x00\x4c\xcc\xde\x40\x00\x40\x06\x6c\x80\xc0\xa8\xd9\x17\x8d\x54\xda\x28'
>>> int.from_bytes(byte_obj, byteorder='big')
394277201243797802270421732363840487422965373480
Where byteorder is used to specify whether the input is big- or little-endian (i.e. most or least significant byte first).
(Looks a bit bigger than 32 bits though!)

Extract 12-bit integer from 2 byte big endian (motorola) bytearray

I am trying to extract an integer which occupies up to 12 bits in a 2 byte (16 bit) message, which is in big-endian format. I have done some research already and expect that I will have to use bit_manipulation (bit shifting) to achieve this, but I am unsure how this can be applied to big-endian format.
A couple of answers on here used the python 'Numpy' package, but I don't have access to that on Micropython. I do have access to the 'ustruct' module, which I use to unpack certain other parts of the message, but it only seems to apply to 8 bit, 16bit and 32bit messages.
So far the only thing I have come up with is:
int12 = (byte1 << 4) + (byte2)
expected_value = int.from_bytes(int12)
but this isn't giving me the number's I am expecting. For example 0x02,0x15 should present decimal 533 .
Where am I going wrong?
I'm new to bit manipulation and extracting data from bytes so any help is greatly appreciated, Thanks!

This should work:
import struct
val, _ = struct.unpack( '!h', b'23' )
val = (val >> 4) & 0xFFF
gives:
>>> hex(val)
'0x333'
However, you should check what 12 bits out of 16 are occupied. My previous code assumes that those are the upper 3 nibbles. If the number occupies lower 3 nibbles, you don't need any shifts, just the mask with 0xFFF.

Why is the first packed data in struct little endian, but the rest is big endian?

import struct
port = 1331
fragments = [1,2,3,4]
flags = bytes([64])
name = "Hello World"
data = struct.pack('HcHH', port, flags, len(fragments), len(name))
print(int.from_bytes(data[3:5], byteorder='big'))
print(int.from_bytes(data[5:7], byteorder='big'))
print(int.from_bytes(data[0:2], byteorder='little'))
When I print them like this, they come out correctly. It seems port is in little endian, while len(fragments) and len(name) are in big endian. If I also do big endian on the port, it gets the wrong value.
So why does struct behave like this? Or am I missing something?

There is some funny alignment taking place because of the 'c' in the middle of 'H'. You can see it with calcsize:
>>> struct.calcsize('HcHH')
8
>>> struct.calcsize('HHHc')
7
So your data is not aligned as you thought. The correct unpacking is:
print(int.from_bytes(data[4:6], byteorder='little'))
# 4
print(int.from_bytes(data[6:], byteorder='little'))
# 11
It turns out that by chance, the added byte of the 'c' is '\x00', and made your byte-chain correct in big-endian:
>>> data
b'3\x05#\x00\x04\x00\x0b\x00'
^^^^
this is the intruder

By default, your call to pack is equivalent to the following:
struct.pack('#HcHH', port, flags, len(fragments), len(name))
The result looks like this (printed with '.'.join(f'{x:02X} for x in data')):
33.05.40.00.04.00.0B.00
0 1 2 3 4 5 6 7
The number 4 is encoded in bytes 4 and 5, in little endian, and 11 is encoded in bytes 6 and 7. Byte 3 is a padding byte, inserted by pack to properly align the following shorts on an even boundary.
Per the docs:
Note By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details.
To remove the alignment byte and justify your assumptions about the positions of the bytes while keeping native byte order, use
struct.pack('=HcHH', port, flags, len(fragments), len(name))
You can also use a fixed byte order by using < or > as the prefix.
The "correct" solution is to use unpack to get your numbers back, so you don't have to worry about endianness, padding or anything else, really.

Base56 conversion etc

It seems base58 and base56 conversion treat input data as a single Big Endian number; an unsigned bigint number.
If I'm encoding some integers into shorter strings by trying to use base58 or base56 it seems in some implementations the integer is taken as a native (little endian in my case) representation of bytes and then converted to a string, while in other implementations the number is converted to big endian representation first. It seems the loose specifications of these encoding don't clarify which approach is right. Is there an explicit specification of which to do, or a more wildly popular option of the two I'm not aware of?
I was trying to compare some methods of making a short URL. The source is actually a 10 digit number that's less than 4 billion. In this case I was thinking to make it an unsigned 4 byte integer, possibly Little Endian, and then encode it with a few options (with alphabets):
base64 A…Za…z0…9+/
base64 url-safe A…Za…z0…9-_
Z85 0…9a…zA…Z.-:+=^!/*?&<>()[]{}#%$#
base58 1…9A…HJ…NP…Za…km…z (excluding 0IOl+/ from base64 & reordered)
base56 2…9A…HJ…NP…Za…kmnp…z (excluding 1o from base58)
So like, base16, base32 and base64 make pretty good sense in that they're taking 4, 5 or 6 bits of input data at a time and looking them up in an alphabet index. The latter uses 4 symbols per 3 bytes. Straightforward, and this works for any data.
The other 3 have me finding various implementations that disagree with each other as to the right output. The problem appears to be that no amount of bytes has a fixed number of lookups in these. EG taking 2^1 to 2^100 and getting the remainders for 56, 58 and 85 results in no remainders of 0.
Z85 (ascii85 and base85 etal.) approach this by grabbing 4 bytes at a time and encoding them to 5 symbols and accepting some waste. But there's byte alignment to some degree here (base64 has alignment per 16 symbols, Z85 gets there with 5). But the alphabet is … not great for urls, command-line, nor sgml/xml use.
base58 and base56 seem intent on treating the input bytes like a Big Endian ordered bigint and repeating: % base; lookup; -= % base; /= base on the input bigint. Which… I mean, I think that ends up modifying most of the input for every iteration.
For my input that's not a huge performance concern though.
Because we shouldn't treat the input as string data, or we get output longer than the 10 digit decimal number input and what's the point in that, does anyone know of any indication of which kind of processing for the output results in something canonical for base56 or base58?
Have the Little Endian 4 byte word of the 10 digit number (<4*10^10) turned into a sequence of bytes that represent a different number if Big Endian, and convert that by repeating the steps.
Have the 10 digit number (<4*10^10) represented in 4 bytes Big Endian before converting that by repeating the steps.
I'm leaning towards going the route of the 2nd way.
For example given the number: 3003295320
The little endian representation is 58 a6 02 b3
The big endian representation is b3 02 a6 58, Meaning
base64 gives:
>>> base64.b64encode(int.to_bytes(3003295320,4,'little'))
b'WKYCsw=='
>>> base64.b64encode(int.to_bytes(3003295320,4,'big'))
b'swKmWA=='
>>> base64.b64encode('3003295320'.encode('ascii'))
b'MzAwMzI5NTMyMA==' # Definitely not using this
Z85 gives:
>>> encode(int.to_bytes(3003295320,4,'little'))
b'sF=ea'
>>> encode(int.to_bytes(3003295320,4,'big'))
b'VJv1a'
>>> encode('003003295320'.encode('ascii')) # padding to 4 byte boundary
b'fFCppfF+EAh8v0w' # Definitely not using this
base58 gives:
>>> base58.b58encode(int.to_bytes(3003295320,4,'little'))
b'3GRfwp'
>>> base58.b58encode(int.to_bytes(3003295320,4,'big'))
b'5aPg4o'
>>> base58.b58encode('3003295320')
b'3soMTaEYSLkS4w' # Still not using this
base56 gives:
>>> b56encode(int.to_bytes(3003295320,4,'little'))
b'4HSgyr'
>>> b56encode(int.to_bytes(3003295320,4,'big'))
b'6bQh5q'
>>> b56encode('3003295320')
b'4uqNUbFZTMmT5y' # Longer than 10 digits so...

STL binary file reader with Python

I'm trying to write my "personal" python version of STL binary file reader, according to WIKIPEDIA : A binary STL file contains :
an 80-character (byte) headern which is generally ignored.
a 4-byte unsigned integer indicating the number of triangular facets in the file.
Each triangle is described by twelve 32-bit floating-point numbers: three for the normal and then three for the X/Y/Z coordinate of each vertex – just as with the ASCII version of STL. After these follows a 2-byte ("short") unsigned integer that is the "attribute byte count" – in the standard format, this should be zero because most software does not understand anything else. --Floating-point numbers are represented as IEEE floating-point numbers and are assumed to be little-endian--
Here is my code :
#! /usr/bin/env python3
with open("stlbinaryfile.stl","rb") as fichier :
head=fichier.read(80)
nbtriangles=fichier.read(4)
print(nbtriangles)
The output is :
b'\x90\x08\x00\x00'
It represents an unsigned integer, I need to convert it without using any package (struct,stl...). Are there any (basic) rules to do it ?, I don't know what does \x mean ? How does \x90 represent one byte ?
most of the answers in google mention "C structs", but I don't know nothing about C.
Thank you for your time.

Since you're using Python 3, you can use int.from_bytes. I'm guessing the value is stored little-endian, so you'd just do:
nbtriangles = int.from_bytes(fichier.read(4), 'little')
Change the second argument to 'big' if it's supposed to be big-endian.
Mind you, the normal way to parse a fixed width type is the struct module, but apparently you've ruled that out.
For the confusion over the repr, bytes objects will display ASCII printable characters (e.g. a) or standard ASCII escapes (e.g. \t) if the byte value corresponds to one of them. If it doesn't, it uses \x##, where ## is the hexadecimal representation of the byte value, so \x90 represents the byte with value 0x90, or 144. You need to combine the byte values at offsets to reconstruct the int, but int.from_bytes does this for you faster than any hand-rolled solution could.
Update: Since apparent int.from_bytes isn't "basic" enough, a couple more complex, but only using top-level built-ins (not alternate constructors) solutions. For little-endian, you can do this:
def int_from_bytes(inbytes):
res = 0
for i, b in enumerate(inbytes):
res |= b << (i * 8) # Adjust each byte individually by 8 times position
return res
You can use the same solution for big-endian by adding reversed to the loop, making it enumerate(reversed(inbytes)), or you can use this alternative solution that handles the offset adjustment a different way:
def int_from_bytes(inbytes):
res = 0
for b in inbytes:
res <<= 8 # Adjust bytes seen so far to make room for new byte
res |= b # Mask in new byte
return res
Again, this big-endian solution can trivially work for little-endian by looping over reversed(inbytes) instead of inbytes. In both cases inbytes[::-1] is an alternative to reversed(inbytes) (the former makes a new bytes in reversed order and iterates that, the latter iterates the existing bytes object in reverse, but unless it's a huge bytes object, enough to strain RAM if you copy it, the difference is pretty minimal).

The typical way to interpret an integer is to use struct.unpack, like so:
import struct
with open("stlbinaryfile.stl","rb") as fichier :
head=fichier.read(80)
nbtriangles=fichier.read(4)
print(nbtriangles)
nbtriangles=struct.unpack("<I", nbtriangles)
print(nbtriangles)
If you are allergic to import struct, then you can also compute it by hand:
def unsigned_int(s):
result = 0
for ch in s[::-1]:
result *= 256
result += ch
return result
...
nbtriangles = unsigned_int(nbtriangles)
As to what you are seeing when you print b'\x90\x08\x00\x00'. You are printing a bytes object, which is an array of integers in the range [0-255]. The first integer has the value 144 (decimal) or 90 (hexadecimal). When printing a bytes object, that value is represented by the string \x90. The 2nd has the value eight, represented by \x08. The 3rd and final integers are both zero. They are presented by \x00.
If you would like to see a more familiar representation of the integers, try:
print(list(nbtriangles))
[144, 8, 0, 0]
To compute the 32-bit integers represented by these four 8-bit integers, you can use this formula:
total = byte0 + (byte1*256) + (byte2*256*256) + (byte3*256*256*256)
Or, in hex:
total = byte0 + (byte1*0x100) + (byte2*0x10000) + (byte3*0x1000000)
Which results in:
0x00000890
Perhaps you can see the similarities to decimal, where the string "1234" represents the number:
4 + 3*10 + 2*100 + 1*1000