I'm looking for a Python3 lib or an implemented way to process on binary data.
The example will tell you more than words :
first part of the packet:
packet.data1 = '0x0EDD'
I want to separate the beginning from the 2 last bits so i use the arrays methodes:
my_id = int(bin(int(packet.data1,16))[-2:],2)
my_len_of_len = int(bin(int(packet.data1,16))[:-2],2)
Now in the second part if my_len_of_len equals 1 i have to catch the following byte like :
packet.data2 = '0x08'
And then i have to convert it to int to know the number of bytes are following, they are the content of the message:
my_len = int(packet.data2,16)
And now i can catch the message from the data. I'm trying to understand a game protocol but with the methods i know, it's "slow" and hard to find myself with all array indices.
A solution for me ?
Thank you.
I recommend using a library for this. My personal recommendation is the awesome construct library.
Methods such as what you've started with, using bit-wise operators or using struct.unpack are highly error-prone and difficult to maintain.
Related
I am trying to stream bitcoin data using the alpaca-py trading documentation but I keey getting a invalid syntax error. This is taken exactly from the alpaca-py documentation. Does anyone know what I am doing wrong?
from typing import Any
from alpaca.data.live import CryptoDataStream
wss_client = CryptoDataStream(key-id, secret-key)
# async handler
async def quote_data_handler(data: Any):
# quote data will arrive here
print(data)
wss_client.subscribe_quotes(quote_data_handler, "BTC")
wss_client.run()
Take a look at the dashes in your parameters. Usually a no-no in most languages since the "-" or dash usually refers to a minus which is a binary operator or an operator that operates on two operands to produce a new value or result."
Make sure parameters are set before passing them.
Try the underscore instead as in: key_id = "".
Also useful is the following link to a comprehensive list of crypto pairs supported by Alpaca: https://alpaca.markets/support/alpaca-crypto-coin-pair-faq/#:~:text=For%20the%20initial%20launch%20of,%2C%20SOL%2C%20TRX%2C%20UNI)
Stay up to date on the above list as it's membership may be a bit volatile at the moment.
I am attempting to decode some data from a Shark 100 Power Meter via TCP modbus. I have successfully pulled down the registers that I need, and am left with two raw values from the registers like so:
[17138, 59381]
From the manual, I know that I need to convert these two numbers into a 32bit IEEE floating-point number. I also know from the manual that "The lower-addressed register is the
high order half (i.e., contains the exponent)." The first number in the list shown above is the lower-addressed register.
Using Python (any library will do if needed), how would I take these two values and make them into a 32 bit IEEE floating point value.
I have tried to use various online converters and calculators to figure out a non-programmatic way to do this, however, anything I have tried gets me a result that is way out of bounds (I am reading volts in this case so the end result should be around 120-122 from the supplied values above).
Update for Python 3.6+ (f-strings).
I am not sure why the fill in #B.Go's answer was only 2. Also, since the byte order was big-endian, I hardcoded it as such.
import struct
a = 17138
b = 59381
struct.unpack('>f', bytes.fromhex(f"{a:0>4x}" + f"{b:0>4x}"))[0]
Output: 121.45304107666016
The following code works:
import struct
a=17138
b=59381
struct.unpack('!f', bytes.fromhex('{0:02x}'.format(a) + '{0:02x}'.format(b)))
It gives
(121.45304107666016,)
Adapted from Convert hex to float and Integer to Hexadecimal Conversion in Python
I read in the comments, and #Sanju had posted this link: https://github.com/riptideio/pymodbus/blob/master/examples/common/modbus_payload.py
For anyone using pymodbus, the BinaryPayloadDecoder is useful as it's built in. It's very easy to pass a result.registers, as shown in the example. Also, it has a logging integrated, so you can help debug why a conversion isn't working (ex: wrong endianness).
As such, I made a working example for this question (using pymodbus==2.3.0):
from pymodbus.constants import Endian
from pymodbus.payload import BinaryPayloadDecoder
a = 17138
b = 59381
registers = [a, b]
decoder = BinaryPayloadDecoder.fromRegisters(registers, byteorder=Endian.Big)
decoder.decode_32bit_float() # type: float
Output: 121.45304107666016
I have different uint64 numbers which I want to send via CAN-Bus with SocketCAN in Python. I need to divide the large number in 8 bytes, so I can assign the values to the CAN data-bytes. But I'm struggling with the implementation.
I am grateful for any help or suggestion.
Thanks for your help!
When it comes to convert numbers to their byte representation, the struct module is your friend:
i = 65357
print(hex(i))
bigendian = struct.pack(">Q", i)
littleendian = struct.pack("<Q", i)
print(repr(bigendian)
print(repr(littleendian)
output is as expected:
'0x10001'
b'\x00\x00\x00\x00\x00\x01\x00\x01'
b'\x01\x00\x01\x00\x00\x00\x00\x00'
That means that you can easily use the individual bytes (in the order you need) to send them via CAN-bus
Assuming you are using Python 3 you can simply use Python int's to_bytes method like so:
i = 65357
print(hex(i))
print(i.to_bytes(8, 'big'))
print(i.to_bytes(8, 'little'))
Output:
0xff4d
b'\x00\x00\x00\x00\x00\x00\xffM'
b'M\xff\x00\x00\x00\x00\x00\x00'
Not sure if you're using the python-can library, but if you are you can pass either bytes, a list of ints or a bytesarray to the can.Message.
Recently I found this awesome 2-factor authentication code generator written in Python 3. I was trying to convert it to Swift 3, but I am having trouble with one specific part, though:
def get_hotp_token(secret, intervals_no):
key = base64.b32decode(secret)
msg = struct.pack(">Q", intervals_no)
h = hmac.new(key, msg, hashlib.sha1).digest()
o = h[19] & 15
h = (struct.unpack(">I", h[o:o+4])[0] & 0x7fffffff) % 1000000
return h
I so far have only been able to do the first line of the function body :p using code from here
func getHotpToken(secret: String) -> [Int] {
let data = secret.base32DecodedData
<...>
return theTokens
}
I tried reading the documentation on struct.pack here and reading about what packing actually is here, but I still find the concept/implementation confusing, and I have no idea what the equivalent would be in Swift.
According to the documentation, struct.pack returns a string in the given format. The format in my case is >Q, which means that the byte order is little-endian and the C Type is an unsigned long long. Again, I am not exactly sure how this is supposed to look in Swift.
... And that is only the second line! I don't really understand how HMAC works (I can't even find the actual 'background' code), so I can't even translate the entire function. I could not find any native library for Swift that has this behavior.
Any pointers or help translating this function will be highly appreciated!
P.S. I checked and I think that this is on topic
Relevant imports:
import base64, struct, hmac
I just finished converting my code to Swift 3. This is a little different from the Python version, since this is more of a framework-type thing. It took a lot of experimentation to get the get_hotp_token to work (for example the Wikipedia page says it used SHA256 but it actually uses SHA1.
You can find it here.
When you use this, be sure to add a bridging header with #import <CommonCrypto/CommonHMAC.h>
Enjoy!
Let's say I need to save a matrix(each line corresponds one row) that could be loaded from fortran later. What method should I prefer? Is converting everything to string is the only one approach?
You can save them in binary format as well. Please see the documentation on the struct standard module, it has a pack function for converting Python object into binary data.
For example:
import struct
value = 3.141592654
data = struct.pack('d', value)
open('file.ext', 'wb').write(data)
You can convert each element of your matrix and write to a file. Fortran should be able to load that binary data. You can speed up the process by converting a row as a whole, like this:
row_data = struct.pack('d' * len(matrix_row), *matrix_row)
Please note, that 'd' * len(matrix_row) is a constant for your matrix size, so you need to calculate that format string only once.
I don't know fortran, so it's hard to tell what is easy for you to perform on that side for parsing.
It sounds like your options are either saving the doubles in plaintext (meaning, 'converting' them to string), or in binary (using struct and the likes). The decision for which one is better depends.
I would go with the plaintext solution, as it means the files will be easily readable, and you won't have to mess with different kinds of details (endianity, default double sizes).
But, there are cases where binary is better (for example, if you have a really big list of doubles and space is of importance, or if it is easier for you to parse it and you need the optimization) - but this is likely not your case.
You can use JSON
import json
matrix = [[2.3452452435, 3.34134], [4.5, 7.9]]
data = json.dumps(matrix)
open('file.ext', 'wb').write(data)
File content will look like:
[[2.3452452435, 3.3413400000000002], [4.5, 7.9000000000000004]]
If legibility and ease of access is important (and file size is reasonable), Fortran can easily parse a simple array of numbers, at least if it knows the size of the matrix beforehand (with something like READ(FILE_ID, '2(F)'), I think):
1.234 5.6789e4
3.1415 9.265358978
42 ...
Two nested for loops in your Python code can easily write your matrix in this form.