I'm using Protobuf with the C++ API and I have a standart message I send between 2 different softwares and I want to add a raw nested message as data.
So I added a message like this:
Message main{
string id=1;
string data=2;
}
I tried to serialize some nested messages I made to a string and send it as "data" with "main" message but it doesn't work well on the parser side.
How can I send nested serialized message inside a message using c++ and python api.
Basically, use bytes:
message main {
string id=1;
bytes data=2;
}
In addition to not corrupting the data (string is strictly UTF-8), as long as the payload is a standard message, this is also compatible with changing it later (at either end, or both) to the known type:
``` proto
message main {
string id=1;
TheOtherMessageType data=2;
}
message TheOtherMessageType {...}
(or even using both versions at different times depending on which is most convenient)
Related
I want to Hash a CAN message received from a vehicle. The following code written in Python is used to receive the CAN message(dev.recv()) from the vehicle and print the received message (dev.send()).I want to hash the CAN message present in dev.recv()function before sending the message using dev.send().Is this possible? If so how can it be done?
from canard.hw import socketcan
dev = socketcan.SocketCanDev(’can0’)
dev.start()
while True:
f = dev.recv()
dev.send(f)
`
I am not sure what the data type is for "f", the data you receive from the function recv.
I am guessing that SocketCanDev is just a wrapper for the device, and recv acts very similar to the function, read().
So, "f" in your code might be interpreted as an array of bytes, or chars.
Hashing is done to an array of bytes, regardless of the format of the
string.
And, the result of the hashing does not depend on the input format or data type.
Therefore, in your case,
while True:
f = dev.recv()
result = (hashFunction)(f)
dev.send(result) // result should in the data type that the function send can accept as a parameter
(hashFunction) can be replaced with an actual function from a hashing library, such as "hashlib".
If you are interested in cryptographic hashing, you should take a look at hashlib
With it you should be able to hash the message and send the hash like this
H = hashlib.new('sha256', message)
dev.send(H.digest())
If you want to still send the original message besides the hash, you could make two calls to send.
I'm trying to build a bridge between a Redis server and MQTT, so that when the Redis database is updated, these updates are dispatched via MQTT to clients.
For this a client (only one, the bridge) connects to the Redis database and starts to monitor it.
My issue is with parsing the commands, more specifically the arguments contained with it, which is a whitespace-separated list of stings.
For example, when I store the following hash in Redis
data = {
"key-3-1-json": "value-1",
"key-3-2-json": 'this "this is \'quoted\' text"',
}
print r18.hmset("test-hash", {
"key-1": "value-1",
"key-2": 'this "this is \'quoted\' text"',
"key-3": json.dumps(data),
})
the client recieves the following
1549578825.1 0 HMSET test-hash "key-3" "{\"key-3-1-json\": \"value-1\", \"key-3-2-json\": \"this \\\"this is 'quoted' text\\\"\"}" "key-2" "this \"this is 'quoted' text\"" "key-1" "value-1"
As you can see I'm already parsing the timestamp, database id, command and key, but the last part, I don't know how to create a list of strings from it.
This message would then be sent over MQTT as
mqtt.publish("redis/mon/0/HMSET/test-hash", json.dumps(args))
where args would be
[
"key-3",
"{\"key-3-1-json\": \"value-1\", \"key-3-2-json\": \"this \\\"this is 'quoted' text\\\"\"}",
"key-2",
"this \"this is 'quoted' text\"",
"key-1",
"value-1"
]
which would probably be the most complex case, since usually the args would be one single string, in the case where r18.set would have been used instead of r18.hmset.
I think there must be some built-in module in Python which could to this as it is like parsing a command line string.
The subprocess module's documentation states that subprocess.Popen() makes use of shlex.split() (shlex: Simple lexical analysis)
Calling shlex.split(args_str) effectively converts the arguments string into the desired list of substrings.
I'm using python to read messages coming from various topics. Some topics have got their messages encoded in plain JSON, while others are using Avro binary serialization, with confluent schema registry.
When I receive a message I need to know if it has to be decoded. At the moment I'm only relying on the fact that the binary encoded messages are starting with a MAGIC_BYTE which value is zero:
from confluent_kafka.cimpl import Consumer
consumer = Consumer(config)
consumer.subsrcibe(...)
msg = consumer.poll()
# check the msg is not null or error etc
if msg.values()[0] == 0:
# It is binary encoded
else:
# It is json
I was wondering is there's a better way to do that?
You could get bytes 0-5 of your message, then
magic_byte = message_bytes[0]
schema_id = message_bytes[1:5]
Then, perform a lookup against your registry for GET /schemas/{schema_id}, and cache the ID + schema (if needed) when you get a 200 response code.
Otherwise, the message is either JSON, or the producer had sent its data to a different registry (if there is more than one in your environment). Note: this means the data could still be Avro
You can simply query the schema registry through REST first, and build a local cache of the topics that are registered there. Then, when you're trying to decode a message from a particular topic, simply compare the topic to the contents of that list. If it's there, you know it has be decoded.
Of course, this only works if all the topics that are Avro encoded are using Schema Registry. If you ever receive an Avro-encoded message that is not registered with Schema Registry, then it won't work.
What is the data type for "message" in pubsub used by volttron? I have checked the documentation but there is nothing mentioned about this. When checking the source I found this function comment source :
param headers: header info for the message,
type headers: None or dict,
param message: actual message,
type message: None or any
Are the above info correct? Does that "any" type refer to this: typing.Any?
The message can be any Python object that can be serialized into JSON. Typically this will be something specifically defined by the Agent publishing the message that aligns with the purpose of the message. Usually this will be a dictionary or list, but occasionally messages will be numbers or strings. VOLTTRON does not place any restrictions on the structure of the data as long as it can be serialized.
It is up to agents define the datatype of the message and document it for use by other agents.
Nested data structures are allowed as they are in JSON.
Goal:
I am writing a socket server/client program (c++ is the server, python is the client) to send xml strings that carry data. My goal is to be able to receive an xml message from c++ in Python via socket.
Method
VS2013 pro
Python 2.7.2 via Vizard 4.1
1) socket communication is created just fine, no problems. I can send/receive stuff
2) after communications are initialized, c++ begins creating xml objects using Cmarkup
3) c++ converts the xml object to std::string type
4) c++ sends the std::string over the stream to Python
Problem:
The "string" received in python from C++ is interpreted as garbage symbols (not trying to offend, someone may have strong feelings for them, I do not ;) that look like symbols you'd see in notepad if you opened a binary file. This is not surprising, since data sent over the stream is binary.
What I cannot figure out is how to get Python to make sense of the stream.
Failed Attempts to fix:
1) made sure that VS2013 project uses Unicode characters
2) tried converting stream to python string and decoding it string.decode()
3) tried using Unicode()
4) also tried using binascii() methods to get something useful, small improvement but still not the same characters I sent from c++
If anyone can lend some insight on why this is happening I'd be most grateful. I have read several forums about the way data is sent over sockets, but this aspect of encoding and decoding is still spam-mackerel-casserole to my mind.
Here's the server code that creates xml, converts to string, then sends
MCD_CSTR rootname("ROOT");//initializes name for root node
MCD_CSTR Framename("FRAME");//creates name for child node
CMarkup xml;//initializes xml object using Cmarkup method
xml.AddElem(rootname);//create the root node
xml.IntoElem();//move into it
xml.AddElem(Framename, MyClient.GetFrameNumber().FrameNumber);//create child node with data from elsewhere, FrameNumber is an int
CStringA strXML = xml.GetDoc();//convert the xml object to a string using Cmarkup method
std::string test(strXML);//convert the CstringA to a std::string type
std::cout << test << '\n';//verify that the xml as a string looks right
std::cout << typeid(test).name() << '\n';//make sure it is the right type
iSendResult = send(ClientSocket, (char *)&test, sizeof(test), 0);//send the string to the client
Here is the code to receive the xml string in Python:
while 1:
data = s.recv(1024)#receive the stream with larger than required buffer
print(data)#see what is in there
if not data: break#if no data then stop listening for more
Since test is a string, this cannot work:
iSendResult = send(ClientSocket, (char *)&test, sizeof(test), 0);//send the string
The std::string is not a character array. It is an object, and all that line does is send nonsensical bytes to the socket. You want to send the data, not the object.
iSendResult = send(ClientSocket, (char *)test.c_str(), test.length(), 0);//send the string
You can't just write the memory at the location of a std::string and think that's serialization. Depending on how the C++ library implemented it, std::string is likely to be a structure containing a pointer to the actual character data. If you transmit the pointer, not only will you fail to send the character data, but the pointer value is meaningless in any other context than the current instance of the program.
Instead, serialize the important contents of the string. Send the length, then send the character data itself. Something like this.
uint32_t len = test.length();
send(..., &len, sizeof(uint32_t), ...);
send(..., test.c_str(), len, ...);