Hi I have a question concerend about protocol buffers for list of lists in json:
Example .json (testjson)
"outcome": {
"col": [
"datetime",
"return"
],
"val": [[1199232000000, -0.0066], [1199318400000, -0.0033]]
}
My .proto File (cum_ret.proto)
message CumReturn {
message period_value {
oneof types{
int32 day = 1;
float return = 2;
}
}
message period_values {
repeated period_value value = 1;
}
message outcome {
repeated string col = 1;
repeated period_value val = 2;
}
outcome outcome_returns = 2;
}
I parse the json with the following code:
testjson = {
"outcome_returns": {
"col": [
"datetime",
"cum_returns"
],
"val": [[1199232000000, -0.0066705691], [1199318400000, -0.0033641154]]
}
}
import cum_ret_pb2 as CumRet
from google.protobuf.json_format import Parse
cumrets = Parse(json.dumps(test_json), CumRet.CumReturn())
But I got the error msg:
Failed to parse 1199232000000 field: expected string or bytes-like object...
Could someone help: getting a List of List of int and float into .proto schema?
One approach to implementing a list of lists is to create a new message for your list:
message ListOfInt {
repeated int32 value = 1;
}
and when you call this, use
message outcome {
repeated string col = 1;
repeated ListOfInt val = 2;
}
However, I think you have a different issue in your code. Your "period_value" message expects either an int32 or a float. The maximum int32 value is 2,147,483,647, but you are trying to fit 1,199,232,000,000 into that field. Hence, the error message that the value cannot be parsed. Try changing the int32 to an int64:
message period_value {
oneof types {
int64 day = 1;
float return = 2;
}
}
Related
Hello I receive binary data with protobuf, I would like to decode this msg.
my proto file:
syntax = "proto3";
message Server {
oneof msg {
Values values = 3;
Ack ack = 5;
}
int64 ack_id = 8;
}
message Ack {
int64 ack_id = 1;
bool is_ok = 2;
string error_msg = 3;
}
message Values {
message Value {
int64 timestamp = 1;
int64 id = 2;
oneof value {
string string_value = 3;
double double_value = 4;
sint32 int_value = 5;
}
}
repeated Value values = 1;
}
I compiled my proto file using protoc.
here is my example python code to decode this proto msg.
import server_pb2
values = server_pb2.Values.Value()
values.ParseFromString(data)
print(values.timestamp)
print(values.id)
print(values.value)
but always returned values are 0.
0
0
0.0
example byte data input:
b'\x1a\x17\n\x15\x08\xc0\xd6\xb2\x9f\x06\x10\xaa\xed\xe3\xe4\n!\xe9\x0b!\xe7\xfd\xff\xef?#\x1a'
bin ascii:
1a170a1508c0d6b29f0610aaede3e40a21e90b21e7fdffef3f401a
base64:
GhcKFQjA1rKfBhCq7ePkCiHpCyHn/f/vP0Aa
I don't know if these proto data are correct
Try:
import server_pb2
data = b'\x1a\x17\n\x15\x08\xc0\xd6\xb2\x9f\x06\x10\xaa\xed\xe3\xe4\n!\xe9\x0b!\xe7\xfd\xff\xef?#\x1a'
server = server_pb2.Server()
server.ParseFromString(data)
print(server)
Yields:
values {
values {
timestamp: 1676454720
id: 2895705770
double_value: 0.999999
}
}
ack_id: 26
Update responding to comment
Continuing the above code:
for value in server.values.values:
print(value.timestamp)
print(value.id)
print(value.double_value)
Yields:
1676454720
2895705770
0.999999
The proto duplicates the field name values. Server has a field values of type Values and Values contains a field values of type Value. So the Python code has some redundancy:
server_values = server.values
values_values = server.values.values # Or server_values.values
So, I need some help returning an ID having found a certain string. My JSON looks something like this:
{
"id": "id1"
"field1": {
"subfield1": {
"subrield2": {
"subfield3": {
"subfield4": [
"string1",
"string2",
"string3"
]
}
}
}
}
"id": "id2"
"field1": {
"subfield1": {
"subrield2": {
"subfield3": {
"subfield4": [
"string4",
"string5",
"string6"
]
}
}
}
}
}
Now, I need to get the ID from a certain string, for example:
For "string5" I need to return "id2"
For "string2" I need to return "id1"
In order to find these strings I have used objectpath python module like this: json_Tree.execute('$..subfield4'))
After doing an analysis on a huge amount of strings, I need to return the ones that are meeting my criterias. I have the strings that I need (for example "string3"), but now I have to return the IDs.
Thank you!!
Note: I don't have a lot of experience with coding, I just started a few months ago to work on a project in Python and I have been stuck on this for a while
Making some assumptions about the actual structure of the data as being:
[
{
"id": "id1",
"subfield1": {
"subfield2": {
"subfield3": {
"subfield4": [
"string1",
"string2",
"string3"
]
}
}
}
}
// And so on
]
And assuming that each string1, string2 etc. is in only one id, then you can construct this mapping like so:
data: List[dict] # The json parsed as a list of dicts
string_to_id_mapping = {}
for record in data:
for string in record["subfield1"]["subfield2"]["subfield3"]["subfield4"]:
string_to_id_mapping[string] = record["id"]
assert string_to_id_mapping["string3"] == "id1"
If each string can appear in multiple ids then the following will catch all of them:
from collections import defaultdict
data: List[dict] # The json parsed as a list of dicts
string_to_id_mapping = defaultdict(set)
for record in data:
for string in record["subfield1"]["subfield2"]["subfield3"]["subfield4"]:
string_to_id_mapping[string].add(record["id"])
assert string_to_id_mapping["string3"] == {"id1"}
i tried this but it didn't work, i dont know how to get the array labels which is in the object chartData to pass the data to it
state = {
chartData: {
labels:[],
datasets:[{
label:'items',
data:[]
}]
}
}
componentDidMount() {
axios.get('http://127.0.0.1:8000/orders/chart/').then(result => {
this.setState({
labels:result.data.label,
data:result.data.quantity
});
console.log(this.state.chartData)
});
}
and console.log() gave me an empty arrays and here is the result
Object
labels: Array(0)
length: 0
__proto__: Array(0)
datasets: Array(1)
0: {label: "items", data: Array(0), _meta: {…}}
length: 1
__proto__: Array(0)
__proto__: Object
and here is my backend function:
class chart(ListAPIView):
serializer_class=chartserializer
def get(self,*args,**kwrgs):
names=[]
quant=[]
od=order_details.objects.order_by('product__name','quantity').distinct('product__name')
for i in od :
name=products.objects.get(name=i.product.name)
qty=list(order_details.objects.filter(product=i.product).aggregate(Sum('quantity')).values())[0]
names.append(name.name)
quant.append(qty)
data={
"label":names,
"quantity":quant
}
return Response(data)
and it's result:
{
"label": [
"dress",
"jacket"
],
"quantity": [
10,
17
]
}
From what I see, I think you are not organizing you data correctly when returning a json. They should be grouped by order_detail so you can get meaning from each of them i.e. I would do return something like this from backend.
[
{
"label": "dress",
"quantity": 10
},
{
"label": "jacket",
"quantity": 17
},
]
There your json is self explaining. You have an array of order_details, each object inside the array is an order_detail which has a label and a quantity.
Then in the frontend you would just iterate on each of them. Otherwise you will have to keep indexes to refer to each of them since you are receiving two arrays of not directly related values.
this being used in arrow brackets will point to the function result
so your function should be
componentDidMount() {
self = this;
axios.get('http://127.0.0.1:8000/orders/chart/').then(result => {
self.setState({
labels:result.data.label,
data:result.data.quantity
});
console.log(this.state.chartData)
});
}
Suppose we have this Json blob:
{
"thing": {
"x": 1,
"str": "hello,
"params": {
"opaque": "yes",
"unknown": 1,
"more": ...
}
}
}
The contents of params is unknown. All we know is that it's a dictionary.
How do we define a protobuf message that can parse it?
// file: thing.proto
message Thing {
uint32 x = 1;
string str = 2;
WhatGoesHere? params = 3;
}
[Edit] Moved solution to answer as per comment.
Solutions: Use google provided messages.
// file: solution.proto
import "google/protobuf/struct.proto";
message Solution1 {
uint32 x = 1;
string str = 2;
google.protobuf.Struct params = 3;
}
message Solution2 {
uint32 x = 1;
string str = 2;
map<string, google.protobuf.Value> params = 3;
}
I want to display JSON in specific format and data will be from database.So how to form in specific format in python.Format of JSON is
{
"data":{"Device Id" : value from db,"Date":Current Time},
"Data":
{
"Temperature":
{
value from db
},
"Consumptions":
{
value from db
},
"Error":
[
{
value from db
}
]
}
}
I have tried this:
dict((query.description[i][0], value) for i, value in enumerate(row)) for row in query.fetchall()
But I'm not getting desired output and i'm unable to pass data to json. So how to pass data to json.
Error Im getting is Dictionay update sequence element 0# has length 1; 2 is required
Did you use json module?
import json
print json.dumps(yourStructure, indent=6)