I am reading in a series of CAN BUS frames from python-can represented as hex strings, e.g. '9819961F9FFF7FC1' and I know the values in each frame are laid out as follows:
Signal Startbit Length
A 0 8
B 8 4
C 12 4
D 16 12
E 28 12
F 40 16
G 56 4
With each value being an unsigned integer, with little endian byte order. Where I am struggling is how to deal with the 12 bit signals, and how to do it fast as this will be running in real time. As far as I understand struct.unpack only supports 1,2,4, and 8 byte integers. The Bitstring package also only supports whole-byte bitstrings when you specify the endianness.
I clearly don't understand binary well enough to do it by manipulating the bits directly because I have been tearing my hair out trying to get sensible values...
I was able to decode the frame successfully and reasonably quickly with the bitstruct library, which can handle values with any number of bits, as in the code below.
However I found I also had to swap the location of the hex characters if two signals are present on the same byte, as in the CAN frame layout. I'm still not sure why, but it does work.
swapped_frame = frame[0:2] + frame[3] + frame[2] + frame[4:6] + frame[7] + \
frame[6] + frame[8:]
ba = bytearray(swapped_frame.decode('hex'))
A,B,C,D,E,F,G = bitstruct.unpack('<u8u4u4u12u12u16u4', ba)
Related
[I cannot use numpy so please refrain from talking about it]
I (apparently naively) thought Python array.frombytes() would read from a series of bytes representing various machine format integers, depending on how you create the array object. On creation you are required to provide a letter type code telling it (or so I thought) the machine type of integer making up the byte stream.
import array
b = b"\x01\x00\x02\x00\x03\x00\x04\x00"
a = array.array('i') #signed int (2 bytes)
a.frombytes(b)
print(a)
array('i', [131073, 262147])
and in the debugger:
array('i', [131073, 262147])
itemsize: 4
typecode: 'i'
The bytes in b are a series of little endian int16s (type code = 'i'). Despite being told this, it interpreted the bytes as 4-byte integers. This is Python 3.7.8.
I really need to convert the varying ints into an array (or list) of Python ints to deal with image data coming in byte-streams but which is actually either 16-bit or 32-bit integer, or 64 bit double floating format. What did I miss or do wrong? Or what is the right way to accomplish this?
Note that the documentation doesn't specify the exact size of each type, it specifies the minimum size. Which means it may use a larger size if it wants, probably based on the types in the C compiler that was used to build Python.
Here are all the sizes on my system:
for c in 'bBuhHiIlLqQfd':
print(c, array(c).itemsize)
b 1
B 1
u 2
h 2
H 2
i 4
I 4
l 4
L 4
q 8
Q 8
f 4
d 8
I would suggest using the 'h' or 'H' type.
I'm having some troubles converting a hexadecimal value to decimal. Theoretically is easy with Python:
value = '000100000001f8c65400fefe3195000001230000000000000000000000000000000000000000642b00090700000000001e'
print(int(value,16))
And this is the result:
153914086775326342143664486282692693880080977757509806775956245674142536051238079779640236240803190331364310253598
Since here all is ok.
This string represents a payload of different bytes and I know two things:
1 byte = 8 bits.
Hexadecimal values are usually represented with two hexadecimal values, like FF 2E 32 etc.
The problem comes when I want to work with some concrete byte, because theoretically I know that in the byte 18, 19, 20 and 21 I have some decimal number that starts in 39 (I don't know the other numbers that follow). But when I want to decode it I can't find it.
# First try
a = value[36:43] # 18*2 to 21*2
print(a)
print(int(a,16))
# Second try
a = value[18:22] # 18 to 21
print(a)
print(int(a,16))
With a naked eye, I can see that the third and fourth value in the first result is this 39,
153914086775326342143664486282692693880080977757509806775956245674142536051238079779640236240803190331364310253598
But another time, if I do
# third try
a = value[2:4]
print(a)
print(int(a,16))
I can't find this 39, and the values change from the first result.
How I have to do it? I'm sure is easy but I don't know how to do it. I want to learn to access to the different bytes but I can't understand the logic.
EDIT trying to explain it better
I have this hexadecimal payload:
153914086775326342143664486282692693880080977757509806775956245674142536051238079779640236240803190331364310253598
And this represents the set of different values collected in bytes.
Therefore, what I try is to be able to access a byte (or a set) to know what its value would be in decimal. For example, I know that byte 18 to 21 is the latitude and byte 39 the battery. How can I decode it with python?
(In my city the latitude always starts in 39, that's what I said before this)
Thank you very much
You seem to be very confused about number bases. Please look up proper terminology, as well. For instance:
Hexadecimal values are usually represented with two hexadecimal values, like FF 2E 32 etc
I'm not sure what you're trying to say here. You seem to have noticed that hex values are often separated into bytes (each hex-digit is 4 bits), but the way you express it makes me wonder what you mean. The byte-level separation is not a "representation"; rather, it's a reading convenience.
Your central question seems to be how to find a particular decimal subsequence coded into only a subsequence of the hexidecimal version. Have you confused character coding (such as ASCII, UniCode, or EBCDIC) with simple base representation? Character coding allows you to make this sub-string conversion; changing number bases is not at all the same operation. For instance:
base 16 base 10
1 1
21 33 = 2 * 16 + 1
B21 2849 = 11 * 16*16 + 2 * 16 + 1
B2 178 = 11 * 16 + 2
B 11
There is no subsequence in one base that is coded with a subsequence in the other base. For instance, the "84" in base 10 is a notational feature of the entire hexadecimal number, not any subsequence.
I am trying to extract an integer which occupies up to 12 bits in a 2 byte (16 bit) message, which is in big-endian format. I have done some research already and expect that I will have to use bit_manipulation (bit shifting) to achieve this, but I am unsure how this can be applied to big-endian format.
A couple of answers on here used the python 'Numpy' package, but I don't have access to that on Micropython. I do have access to the 'ustruct' module, which I use to unpack certain other parts of the message, but it only seems to apply to 8 bit, 16bit and 32bit messages.
So far the only thing I have come up with is:
int12 = (byte1 << 4) + (byte2)
expected_value = int.from_bytes(int12)
but this isn't giving me the number's I am expecting. For example 0x02,0x15 should present decimal 533 .
Where am I going wrong?
I'm new to bit manipulation and extracting data from bytes so any help is greatly appreciated, Thanks!
This should work:
import struct
val, _ = struct.unpack( '!h', b'23' )
val = (val >> 4) & 0xFFF
gives:
>>> hex(val)
'0x333'
However, you should check what 12 bits out of 16 are occupied. My previous code assumes that those are the upper 3 nibbles. If the number occupies lower 3 nibbles, you don't need any shifts, just the mask with 0xFFF.
It seems base58 and base56 conversion treat input data as a single Big Endian number; an unsigned bigint number.
If I'm encoding some integers into shorter strings by trying to use base58 or base56 it seems in some implementations the integer is taken as a native (little endian in my case) representation of bytes and then converted to a string, while in other implementations the number is converted to big endian representation first. It seems the loose specifications of these encoding don't clarify which approach is right. Is there an explicit specification of which to do, or a more wildly popular option of the two I'm not aware of?
I was trying to compare some methods of making a short URL. The source is actually a 10 digit number that's less than 4 billion. In this case I was thinking to make it an unsigned 4 byte integer, possibly Little Endian, and then encode it with a few options (with alphabets):
base64 A…Za…z0…9+/
base64 url-safe A…Za…z0…9-_
Z85 0…9a…zA…Z.-:+=^!/*?&<>()[]{}#%$#
base58 1…9A…HJ…NP…Za…km…z (excluding 0IOl+/ from base64 & reordered)
base56 2…9A…HJ…NP…Za…kmnp…z (excluding 1o from base58)
So like, base16, base32 and base64 make pretty good sense in that they're taking 4, 5 or 6 bits of input data at a time and looking them up in an alphabet index. The latter uses 4 symbols per 3 bytes. Straightforward, and this works for any data.
The other 3 have me finding various implementations that disagree with each other as to the right output. The problem appears to be that no amount of bytes has a fixed number of lookups in these. EG taking 2^1 to 2^100 and getting the remainders for 56, 58 and 85 results in no remainders of 0.
Z85 (ascii85 and base85 etal.) approach this by grabbing 4 bytes at a time and encoding them to 5 symbols and accepting some waste. But there's byte alignment to some degree here (base64 has alignment per 16 symbols, Z85 gets there with 5). But the alphabet is … not great for urls, command-line, nor sgml/xml use.
base58 and base56 seem intent on treating the input bytes like a Big Endian ordered bigint and repeating: % base; lookup; -= % base; /= base on the input bigint. Which… I mean, I think that ends up modifying most of the input for every iteration.
For my input that's not a huge performance concern though.
Because we shouldn't treat the input as string data, or we get output longer than the 10 digit decimal number input and what's the point in that, does anyone know of any indication of which kind of processing for the output results in something canonical for base56 or base58?
Have the Little Endian 4 byte word of the 10 digit number (<4*10^10) turned into a sequence of bytes that represent a different number if Big Endian, and convert that by repeating the steps.
Have the 10 digit number (<4*10^10) represented in 4 bytes Big Endian before converting that by repeating the steps.
I'm leaning towards going the route of the 2nd way.
For example given the number: 3003295320
The little endian representation is 58 a6 02 b3
The big endian representation is b3 02 a6 58, Meaning
base64 gives:
>>> base64.b64encode(int.to_bytes(3003295320,4,'little'))
b'WKYCsw=='
>>> base64.b64encode(int.to_bytes(3003295320,4,'big'))
b'swKmWA=='
>>> base64.b64encode('3003295320'.encode('ascii'))
b'MzAwMzI5NTMyMA==' # Definitely not using this
Z85 gives:
>>> encode(int.to_bytes(3003295320,4,'little'))
b'sF=ea'
>>> encode(int.to_bytes(3003295320,4,'big'))
b'VJv1a'
>>> encode('003003295320'.encode('ascii')) # padding to 4 byte boundary
b'fFCppfF+EAh8v0w' # Definitely not using this
base58 gives:
>>> base58.b58encode(int.to_bytes(3003295320,4,'little'))
b'3GRfwp'
>>> base58.b58encode(int.to_bytes(3003295320,4,'big'))
b'5aPg4o'
>>> base58.b58encode('3003295320')
b'3soMTaEYSLkS4w' # Still not using this
base56 gives:
>>> b56encode(int.to_bytes(3003295320,4,'little'))
b'4HSgyr'
>>> b56encode(int.to_bytes(3003295320,4,'big'))
b'6bQh5q'
>>> b56encode('3003295320')
b'4uqNUbFZTMmT5y' # Longer than 10 digits so...
Between each word in the wav file I have full silence (I checked with Hex workshop and silence is represented with 0's).
How can I cut the non-silence sound?
I'm programming using python.
Thanks!
Python has a wav module. You can use it to open a wav file for reading and use the `getframes(1)' command to walk through the file frame by frame.
import wave
w = wave.open('beeps.wav', 'r')
for i in range():
frame = w.readframes(1)
The frame returned will be a byte string with hex values in it. If the file is stereo the result will look something like this (4 bytes):
'\xe2\xff\xe2\xff'
If its mono, it will have half the data (2 bytes):
'\xe2\xff'
Each channel is 2 bytes long because the audio is 16 bit. If is 8 bit, each channel will only be one byte. You can use the getsampwidth() method to determine this. Also, getchannels() will determine if its mono or stereo.
You can loop over these bytes to see if they all equal zero, meaning both channels are silent. In the following example I use the ord() function to convert the '\xe2' hex values to integers.
import wave
w = wave.open('beeps.wav', 'r')
for i in range(w.getnframes()):
### read 1 frame and the position will updated ###
frame = w.readframes(1)
all_zero = True
for j in range(len(frame)):
# check if amplitude is greater than 0
if ord(frame[j]) > 0:
all_zero = False
break
if all_zero:
# perform your cut here
print 'silence found at frame %s' % w.tell()
print 'silence found at second %s' % (w.tell()/w..getframerate())
It is worth noting that a single frame of silence doesn't necessarily denote empty space since the amplitude may cross the 0 mark normal frequencies. Therefore, it is recommended that a certain number of frames at 0 be observed before deciding if the region is, in fact, silent.
I have been doing some research on this topic for a project I'm working on and I came across a few problems with the solution provided, namely the method for determining silence is incorrect. A "more correct" implementation would be:
import struct
import wave
wave_file = wave.open("sound_file.wav", "r")
for i in range(wave_file.getnframes()):
# read a single frame and advance to next frame
current_frame = wave_file.readframes(1)
# check for silence
silent = True
# wave frame samples are stored in little endian**
# this example works for a single channel 16-bit per sample encoding
unpacked_signed_value = struct.unpack("<h", current_frame) # *
if abs(unpacked_signed_value[0]) > 500:
silent = False
if silent:
print "Frame %s is silent." % wave_file.tell()
else
print "Frame %s is not silent." % wave_file.tell()
References and Useful Links
*Struct Unpacking will be useful here: https://docs.python.org/2/library/struct.html
**A good reference I found explaining the format of wave files for dealing with different size bit-encodings and multiple channels is: http://www.piclist.com/techref/io/serial/midi/wave.html
Using the built-in ord() function in Python on the first element of the string object returned by the readframes(x) method will not work correctly.
Another key point is that multiple channel audio is interleaved and thus a little extra logic is needed for dealing with channels. Again, the link above goes into detail about this.
Hopefully this helps someone in the future.
Here are some of the more important points from the link, and what I found helpful.
Data Organization
All data is stored in 8-bit bytes, arranged in Intel 80x86 (ie, little endian) format. The bytes of multiple-byte values are stored with the low-order (ie, least significant) bytes first. Data bits are as follows (ie, shown with bit numbers on top):
7 6 5 4 3 2 1 0
+-----------------------+
char: | lsb msb |
+-----------------------+
7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8
+-----------------------+-----------------------+
short: | lsb byte 0 | byte 1 msb |
+-----------------------+-----------------------+
7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24
+-----------------------+-----------------------+-----------------------+-----------------------+
long: | lsb byte 0 | byte 1 | byte 2 | byte 3 msb |
+-----------------------+-----------------------+-----------------------+-----------------------+
Interleaving
For multichannel sounds (for example, a stereo waveform), single sample points from each channel are interleaved. For example, assume a stereo (ie, 2 channel) waveform. Instead of storing all of the sample points for the left channel first, and then storing all of the sample points for the right channel next, you "mix" the two channels' sample points together. You would store the first sample point of the left channel. Next, you would store the first sample point of the right channel. Next, you would store the second sample point of the left channel. Next, you would store the second sample point of the right channel, and so on, alternating between storing the next sample point of each channel. This is what is meant by interleaved data; you store the next sample point of each of the channels in turn, so that the sample points that are meant to be "played" (ie, sent to a DAC) simultaneously are stored contiguously.
See also How to edit raw PCM audio data without an audio library?
I have no experience with this, but have a look at the wave module present in the standard library. That may do what you want. Otherwise you'll have to read the file as a byte stream an cut out sequences of 0-bytes (but you cannot just cut out all 0-bytes, as that would invalidate the file...)
You might want to try using sox, a command-line sound processing tool. It has many modes, one of them is silence:
silence: Removes silence from the beginning, middle, or end of a sound file. Silence is anything below a specified threshold.
It supports multiple sound formats and it's quite fast, so parsing large files shouldn't be a problem.
To remove silence from the middle of a file, specify a below_periods that is negative. This value is then treated as a positive value and is also used to indicate the effect should restart processing as specified by the above_periods, making it suitable for removing periods of silence in the middle of the sound file.
I haven't found any python building for libsox, though, but You can use it as You use all command line programs in python (or You can rewrite it - use sox sources for guidance then).
You will need to come up with some threshold value of a minimum number of consecutive zeros before you cut them. Otherwise you'll be removing perfectly valid zeros from the middle of normal audio data. You can iterate through the wave file, copying any non-zero values, and buffering up zero values. When you're buffering zeroes and eventually come across the next non-zero, if the buffer has fewer samples that the threshold, copy them over, otherwise discard it.
Python is not a great tool for this sort of task though. :(