Hello I receive binary data with protobuf, I would like to decode this msg.
my proto file:
syntax = "proto3";
message Server {
oneof msg {
Values values = 3;
Ack ack = 5;
}
int64 ack_id = 8;
}
message Ack {
int64 ack_id = 1;
bool is_ok = 2;
string error_msg = 3;
}
message Values {
message Value {
int64 timestamp = 1;
int64 id = 2;
oneof value {
string string_value = 3;
double double_value = 4;
sint32 int_value = 5;
}
}
repeated Value values = 1;
}
I compiled my proto file using protoc.
here is my example python code to decode this proto msg.
import server_pb2
values = server_pb2.Values.Value()
values.ParseFromString(data)
print(values.timestamp)
print(values.id)
print(values.value)
but always returned values are 0.
0
0
0.0
example byte data input:
b'\x1a\x17\n\x15\x08\xc0\xd6\xb2\x9f\x06\x10\xaa\xed\xe3\xe4\n!\xe9\x0b!\xe7\xfd\xff\xef?#\x1a'
bin ascii:
1a170a1508c0d6b29f0610aaede3e40a21e90b21e7fdffef3f401a
base64:
GhcKFQjA1rKfBhCq7ePkCiHpCyHn/f/vP0Aa
I don't know if these proto data are correct
Try:
import server_pb2
data = b'\x1a\x17\n\x15\x08\xc0\xd6\xb2\x9f\x06\x10\xaa\xed\xe3\xe4\n!\xe9\x0b!\xe7\xfd\xff\xef?#\x1a'
server = server_pb2.Server()
server.ParseFromString(data)
print(server)
Yields:
values {
values {
timestamp: 1676454720
id: 2895705770
double_value: 0.999999
}
}
ack_id: 26
Update responding to comment
Continuing the above code:
for value in server.values.values:
print(value.timestamp)
print(value.id)
print(value.double_value)
Yields:
1676454720
2895705770
0.999999
The proto duplicates the field name values. Server has a field values of type Values and Values contains a field values of type Value. So the Python code has some redundancy:
server_values = server.values
values_values = server.values.values # Or server_values.values
Related
How can I convert the following dict to protobuf ?
I have to send a protobuf as a payload to an mqtt broker.
I'm using python 3.8
publish_msg = {
"token":"xxxxxxxx",
"parms":{
"fPort":8,
"data":b"MDQzYzAwMDE=",
"confirmed":False,
"devEUI":"8CF9572000023509"
}
}
My protobuf is defined as follows:
syntax = "proto3";
package publish;
message DLParams{
string DevEUI = 1;
int32 FPort = 2;
bytes Data = 3;
bool Confirm = 4;
}
message DeviceDownlink {
string Token = 1;
DLParams Params = 2;
}
How about this?
import json
from google.protobuf import json_format
# Create an empty message
dl = DeviceDownlink_pb2.DeviceDownlink()
# Get the json string for your dict
json_string = json.dumps(publish_msg)
# Put the json contents into your message
json_format.Parse(json_string, dl)
Given the following proto definition
Can all autogenerated grpc clients produced by google code generation omit sending fields in PagingInfo?
In the python client, I can omit sending that field by using code like:
request = SearchRequest(paging_info=dict(a=OptionalField(d='d', e='e')), query='blah')
grpc proto definition:
syntax = "proto3";
message OptionalField {
string d = 1;
string e = 2;
}
message PagingInfo {
OptionalField a = 1;
OptionalField b = 2;
OptionalField c = 3;
}
message SearchRequest {
string query = 1;
PagingInfo paging_info = 2;
}
message SearchResponse {
string a = 1;
}
service SearchService {
rpc Search (SearchRequest) returns (SearchResponse);
}
In proto3, all elements are considered optional (in the proto2 sense), so yes: any compliant implementation should be able to send a message that omits that element.
This question is an evolution of this: Issue returning header byte from Python to Unity
Now the issue about header byte is solved, but I have a problem during the loop.
The data incoming good but after "n" times the loop break because receive a json string with an entire structure of data with open/close bracket PLUS the next structure of data with open bracket, some data and in the "image:" some data and many of the "\0" and unless the close bracket.. And the json parser obviously breaks. Why and what can I do? Thanks
Below the code revised:
IEnumerator Client()
{
while (!IsConnected(client))
{
try
{
client = new TcpClient(host, port);
s = client.GetStream();
byte[] byteBuffer = Encoding.UTF8.GetBytes("Connected to client");
s.Write(byteBuffer, 0, byteBuffer.Length);
while (true)
{
if (s.DataAvailable)
{
while (s.DataAvailable)
{
var header = new byte[4];
s.Read(header, 0, header.Length);
var fileSize = BitConverter.ToUInt32(header,0);
StringBuilder myCompleteMessage = new StringBuilder();
MemoryStream ms = new MemoryStream();
int increment = 0;
while (ms.Length < fileSize)
{
byte[] dataReceived = new byte[fileSize];
increment = s.Read(dataReceived, 0, dataReceived.Length);
ms.Write(dataReceived.Take(increment).ToArray(), 0, increment);
}
myCompleteMessage.AppendFormat("{0}", Encoding.UTF8.GetString(ms.GetBuffer()));
JSONNode data = JSONNode.Parse(myCompleteMessage.ToString());
....
// Do work with myCompleteMessage
yield return null;
}
}
}
}
}
}
I don't have your Unity environment here, so I can't have tested this.
Something like a loop (likely run in a coroutine or something that doesn't block Unity's default handling) like this should be enough to handle a stream of messages of the shape we established before (a 4-byte header signifying the message length, followed by that many bytes of JSON).
while (!IsConnected(client))
{
try
{
client = new TcpClient(host, port);
s = client.GetStream();
while (true)
{
var data = ReadPacket(s);
// Do work with data
}
}
}
ReadPacket should be something like
JSONNode ReadPacket(Stream s)
{
var buffer = new byte[131072];
// Read 4 bytes (we will assume we can always read that much)
var n = s.Read(buffer, 0, 4);
if(n < 4) throw new Exception("short read");
var fileSize = BitConverter.ToUInt32(buffer, 0);
Debug.Print("Reading a blob of length", fileSize);
// Memory stream to accumulate the reads into.
var ms = new MemoryStream();
while (ms.Length < fileSize)
{
// Figure out how much we can read -- if we're near the end,
// don't overread.
var maxRead = Math.Min(buffer.Length, fileSize - ms.Length);
var increment = s.Read(buffer, 0, maxRead);
// ... and write them into the stream.
ms.Write(buffer, 0, increment);
Debug.Print("Read", ms.Length, "of", fileSize);
}
// Decode the bytes to UTF-8 and parse.
return JSONNode.Parse(Encoding.UTF8.GetString(ms.GetBuffer()));
}
Hi I have a question concerend about protocol buffers for list of lists in json:
Example .json (testjson)
"outcome": {
"col": [
"datetime",
"return"
],
"val": [[1199232000000, -0.0066], [1199318400000, -0.0033]]
}
My .proto File (cum_ret.proto)
message CumReturn {
message period_value {
oneof types{
int32 day = 1;
float return = 2;
}
}
message period_values {
repeated period_value value = 1;
}
message outcome {
repeated string col = 1;
repeated period_value val = 2;
}
outcome outcome_returns = 2;
}
I parse the json with the following code:
testjson = {
"outcome_returns": {
"col": [
"datetime",
"cum_returns"
],
"val": [[1199232000000, -0.0066705691], [1199318400000, -0.0033641154]]
}
}
import cum_ret_pb2 as CumRet
from google.protobuf.json_format import Parse
cumrets = Parse(json.dumps(test_json), CumRet.CumReturn())
But I got the error msg:
Failed to parse 1199232000000 field: expected string or bytes-like object...
Could someone help: getting a List of List of int and float into .proto schema?
One approach to implementing a list of lists is to create a new message for your list:
message ListOfInt {
repeated int32 value = 1;
}
and when you call this, use
message outcome {
repeated string col = 1;
repeated ListOfInt val = 2;
}
However, I think you have a different issue in your code. Your "period_value" message expects either an int32 or a float. The maximum int32 value is 2,147,483,647, but you are trying to fit 1,199,232,000,000 into that field. Hence, the error message that the value cannot be parsed. Try changing the int32 to an int64:
message period_value {
oneof types {
int64 day = 1;
float return = 2;
}
}
Suppose we have this Json blob:
{
"thing": {
"x": 1,
"str": "hello,
"params": {
"opaque": "yes",
"unknown": 1,
"more": ...
}
}
}
The contents of params is unknown. All we know is that it's a dictionary.
How do we define a protobuf message that can parse it?
// file: thing.proto
message Thing {
uint32 x = 1;
string str = 2;
WhatGoesHere? params = 3;
}
[Edit] Moved solution to answer as per comment.
Solutions: Use google provided messages.
// file: solution.proto
import "google/protobuf/struct.proto";
message Solution1 {
uint32 x = 1;
string str = 2;
google.protobuf.Struct params = 3;
}
message Solution2 {
uint32 x = 1;
string str = 2;
map<string, google.protobuf.Value> params = 3;
}