Develop Reference

How to parse protobuf in python, byte data

Hello I receive binary data with protobuf, I would like to decode this msg.
my proto file:
syntax = "proto3";
message Server {
oneof msg {
Values values = 3;
Ack ack = 5;
}
int64 ack_id = 8;
}
message Ack {
int64 ack_id = 1;
bool is_ok = 2;
string error_msg = 3;
}
message Values {
message Value {
int64 timestamp = 1;
int64 id = 2;
oneof value {
string string_value = 3;
double double_value = 4;
sint32 int_value = 5;
}
}
repeated Value values = 1;
}
I compiled my proto file using protoc.
here is my example python code to decode this proto msg.
import server_pb2
values = server_pb2.Values.Value()
values.ParseFromString(data)
print(values.timestamp)
print(values.id)
print(values.value)
but always returned values are 0.
0
0
0.0
example byte data input:
b'\x1a\x17\n\x15\x08\xc0\xd6\xb2\x9f\x06\x10\xaa\xed\xe3\xe4\n!\xe9\x0b!\xe7\xfd\xff\xef?#\x1a'
bin ascii:
1a170a1508c0d6b29f0610aaede3e40a21e90b21e7fdffef3f401a
base64:
GhcKFQjA1rKfBhCq7ePkCiHpCyHn/f/vP0Aa
I don't know if these proto data are correct

Try:
import server_pb2
data = b'\x1a\x17\n\x15\x08\xc0\xd6\xb2\x9f\x06\x10\xaa\xed\xe3\xe4\n!\xe9\x0b!\xe7\xfd\xff\xef?#\x1a'
server = server_pb2.Server()
server.ParseFromString(data)
print(server)
Yields:
values {
values {
timestamp: 1676454720
id: 2895705770
double_value: 0.999999
}
}
ack_id: 26
Update responding to comment
Continuing the above code:
for value in server.values.values:
print(value.timestamp)
print(value.id)
print(value.double_value)
Yields:
1676454720
2895705770
0.999999
The proto duplicates the field name values. Server has a field values of type Values and Values contains a field values of type Value. So the Python code has some redundancy:
server_values = server.values
values_values = server.values.values # Or server_values.values

Swift: Incorrect Base64 Encoding

I am attempting to convert a block of code from python and it involved encoding a json string to base64. My attempt on Swift does not produce the same base64 encoded string.
Python:
payload_nonce = datetime.datetime(2022, 10, 10, 0, 0, 0).timestamp()
payload = {"request": "/v1/mytrades", "nonce": payload_nonce}
encoded_payload = json.dumps(payload).encode()
b64 = base64.b64encode(encoded_payload)
print(b64)
//prints b'eyJyZXF1ZXN0IjogIi92MS9teXRyYWRlcyIsICJub25jZSI6IDE2NjUzMzEyMDAuMH0='
Swift:
let formatter = DateFormatter()
formatter.dateFormat = "dd/MM/yyyy"
let date = formatter.date(from: "10/10/2022")
let payloadNonce = date!.timeIntervalSince1970
payload = [
"request": "/v1/mytrades",
"nonce": String(describing: payloadNonce)
]
do {
let json = try JSONSerialization.data(withJSONObject: payload)
let b64 = json.base64EncodedString()
print(b64)
//prints eyJyZXF1ZXN0IjoiXC92MVwvbXl0cmFkZXMiLCJub25jZSI6IjE2NjUzMzEyMDAuMCJ9
} catch {//handle error}
What am I missing?

Decoding the Python payload:
{"request": "/v1/mytrades", "nonce": 1665331200.0}
Decoding the Swift payload:
{"request":"\/v1\/mytrades","nonce":"1665331200.0"}
Firstly, it's clear the payloads are different.
You're using the String(describing:) initializer in Swift so nonce is being converted to a String rather than the raw floating-point value.
Secondly, JSONSerialization.data is escaping the forward slashes automatically when encoding. We can disable this optionally.
Now, other than the space between the keys in Python, the two outputs are the same.
Fixed example:
let formatter = DateFormatter()
formatter.dateFormat = "dd/MM/yyyy"
let date = formatter.date(from: "10/10/2022")
let payloadNonce = date!.timeIntervalSince1970
let payload: [String: Any] = [
"request": "/v1/mytrades",
"nonce": payloadNonce
]
do {
let json = try JSONSerialization.data(withJSONObject: payload, options: .withoutEscapingSlashes)
print(String(data: json, encoding: .utf8)!)
let b64 = json.base64EncodedString()
print(b64)
} catch {
}

When I decode the base64 strings, I get this for the Python code:
echo "eyJyZXF1ZXN0IjogIi92MS9teXRyYWRlcyIsICJub25jZSI6IDE2NjUzMzEyMDAuMH0=" | base64 -d
{"request": "/v1/mytrades", "nonce": 1665331200.0}
And this for the Swift code:
echo "eyJyZXF1ZXN0IjoiXC92MVwvbXl0cmFkZXMiLCJub25jZSI6IjE2NjUzMzEyMDAuMCJ9" | base64 -d
{"request":"\/v1\/mytrades","nonce":"1665331200.0"}
It appears that the slashes in the Swift code are escaped. To fix that, see this Stack Overflow answer: Swift String escaping when serializing to JSON using Codable.
The nonce is also a float in the Python response and a string in the Swift response.

Return the enum key when decoding Protobuf in Python

Currently, I am decoding protobuf messages in Python where the output is:
{
"lat": 12.345678,
"lng": -12.345678,
"firmware_status": 3
}
In my case 3 corresponds to FINISHED per the .proto file enum definition. * Note I'm using v3 of protobuf.
enum FirmwareStatus {
UNKNOWN = 0;
STARTED = 1;
IN_PROGRESS = 2;
FINISHED = 3;
CANCELED = 4;
RESTARTED = 5;
}
How would I pull the enum "key" or definition from protobuf so that my output would be:
{
"lat": 12.345678,
"lng": -12.345678,
"firmware_status": "FINISHED"
}
I couldn't find any functions in the protobuf python library to do this easily, but perhaps I missed something.
Currently, this is my decode function:
def decode_protobuf(uplink):
"""Decode a base64 encoded protobuf message.
Paramaters:
uplink (str): base64 encoded protobuf message
Returns:
output (dict): decoded protobuf message in dictionary format
"""
protobuf = proto.Uplink()
decode_base64 = base64.b64decode(uplink)
protobuf.ParseFromString(decode_base64)
output = {}
elems = protobuf.ListFields()
for elem in elems:
output[elem[0].name] = elem[1]
return output

You can use json_format.MessageToDict which has an option so save enums as string
https://github.com/protocolbuffers/protobuf/blob/main/python/google/protobuf/json_format.py#L134

JAVA to Python MD5 hashlib

I am trying to convert java code into python for doing some encryption using md5. Can you please help me to solve this.
public static void main(String args[]){
String result = "";
JSONObject o1 = new JSONObject();
o1.put("stationId","1298491919448667556");
String body = o1.toJSONString();
System.out.println(body);
try {
MessageDigest md = MessageDigest.getInstance("MD5");
md.update(body.getBytes());
byte[] b = md.digest();
result = java.util.Base64.getEncoder().encodeToString(b);
System.out.println("ContentMD5:"+result);
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
Output: 5pxCd9dmyPR/V2jerFJdXQ==
Python code sample:
import json
import base64, hashlib
body_content = {}
body_content["stationId"] = "1298491919448667556"
body = json.dumps(body_content, sort_keys=True).encode()
hash = hashlib.md5()
hash.update(body)
content_md5 = base64.b64encode(hash.digest()).decode()
print("content-md5",content_md5)
Output: nc9o/UcyLURh7uEbz6A+8w==
Please help me to get same output as java.

proto3 grpc clients: can they all omit sending message fields?

Given the following proto definition
Can all autogenerated grpc clients produced by google code generation omit sending fields in PagingInfo?
In the python client, I can omit sending that field by using code like:
request = SearchRequest(paging_info=dict(a=OptionalField(d='d', e='e')), query='blah')
grpc proto definition:
syntax = "proto3";
message OptionalField {
string d = 1;
string e = 2;
}
message PagingInfo {
OptionalField a = 1;
OptionalField b = 2;
OptionalField c = 3;
}
message SearchRequest {
string query = 1;
PagingInfo paging_info = 2;
}
message SearchResponse {
string a = 1;
}
service SearchService {
rpc Search (SearchRequest) returns (SearchResponse);
}

In proto3, all elements are considered optional (in the proto2 sense), so yes: any compliant implementation should be able to send a message that omits that element.