Develop Reference

Python grpc - reading in all messages before sending responses

I'm trying to understand if grpc server using streams is able to wait for all client messages to be read in prior to sending responses.
I have a trivial application where I send in several numbers I'd like to add and return.
I've set up a basic proto file to test this:
syntax = "proto3";
message CalculateRequest{
int64 x = 1;
int64 y = 2;
};
message CalculateReply{
int64 result = 1;
}
service Svc {
rpc CalculateStream (stream CalculateRequest) returns (stream CalculateReply);
}
On my server-side I have implemented the following code which returns the answer message as the message is received:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
resultToOutput = request.x + request.y
yield contracts_pb2.CalculateReply(result=resultToOutput)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
contracts_pb2_grpc.add_SvcServicer_to_server(
CalculatorServicer(), server)
server.add_insecure_port('localhost:9000')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
print( "We're up")
logging.basicConfig()
serve()
I'd like to tweak this to first read in all the numbers and then send these out at a later stage - something like the following:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
listToReturn = []
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
listToReturn.append (request.x + request.y)
# ...
# do some other stuff first before returning
for item in listToReturn:
yield contracts_pb2.CalculateReply(result=resultToOutput)
Currently, my implementation to write out later doesn't work as the code at the bottom is never reached. Is this by design that the connection seems to "close" before reaching there?
The grpc.io website suggests that this should be possible with BiDirectional streaming:
for example, the server could wait to receive all the client messages before writing its responses, or it could alternately read a message then write a message, or some other combination of reads and writes.
Thanks in advance for any help :)

The issue here is the definition of "all client messages." At the transport level, the server has no way of knowing whether the client has finished independent of the client closing its connection.
You need to add some indication of the client's having finished sending requests to the protocol. Either add a bool field to the existing CalculateRequest or add a top-level oneof with one of the options being something like a StopSendingRequests

Python CAN receiving data (pdo mapping)

I am currently working on making a CAN tracer with Python. The connection as well as the received data from CAN work. My question now is: how can I change the PDO mapping and stop and start the transmission via CAN like it works with CANopen?
import canopen
# CAN Setting
can_interface = '0'
can_filters = [{"can_id":0x018A, "can_mask": 0xFFFF, "extended": True}]
bus = can.interface.Bus(can_interface, bustype='ixxat',can_filters=can_filters)
while True:
message = bus.recv()
print(message)

You should be in pre operational mode and send SDO request to change PDO mapping.
You can get these information in the documentation or in the EDS file.
If doing modification do not forget to send the save SDO request at the end else the node will restart with its default value

receiving and sending mavlink messages using pymavlink library

I have created a proxy between QGC(Ground Control Station) and vehicle in Python. Here is the code:
gcs_conn = mavutil.mavlink_connection('tcpin:localhost:15795')
gcs_conn.wait_heartbeat()
print("Heartbeat from system (system %u component %u)" %(gcs_conn.target_system, gcs_conn.target_system))
vehicle = mavutil.mavlink_connection('tcp:localhost:5760')
vehicle.wait_heartbeat() # recieving heartbeat from the vehicle
print("Heartbeat from system (system %u component %u)" %(vehicle.target_system, vehicle.target_system))
while True:
gcs_msg = gcs_conn.recv_match()
if gcs_msg == None:
pass
else:
vehicle.mav.send(gcs_msg)
print(gcs_msg)
vcl_msg = vehicle.recv_match()
if vcl_msg == None:
pass
else:
gcs_conn.mav.send(vcl_msg)
print(vcl_msg)
I need to receive the messages from the QGC and then forward them to the vehicle and also receive the messages from the vehicle and forward them to the QGC.
When I run the code I get this error.
is there any one who can help me?

If you print your message before sending you'll notice it always fails when you try to send a BAD_DATA message type.
So this should fix it (same for vcl_msg):
if gcs_msg and gcs_msg.get_type() != 'BAD_DATA':
vehicle.mav.send(gcs_msg)
PD: I noticed that you don't specify tcp as input or output, it defaults to input. Than means both connections are inputs. I recommend setting up the GCS connection as output:
gcs_conn = mavutil.mavlink_connection('tcp:localhost:15795', input=False)
https://mavlink.io/en/mavgen_python/#connection_string

For forwarding MAVLink successfully a few things need to happen. I'm assuming you need a usable connection to a GCS, like QGroundControl or MissionPlanner. I use QGC, and my design has basic testing with it.
Note that this is written with Python3. This snippet is not tested, but I have a (much more complex) version tested and working.
from pymavlink import mavutil
import time
# PyMAVLink has an issue that received messages which contain strings
# cannot be resent, because they become Python strings (not bytestrings)
# This converts those messages so your code doesn't crash when
# you try to send the message again.
def fixMAVLinkMessageForForward(msg):
msg_type = msg.get_type()
if msg_type in ('PARAM_VALUE', 'PARAM_REQUEST_READ', 'PARAM_SET'):
if type(msg.param_id) == str:
msg.param_id = msg.param_id.encode()
elif msg_type == 'STATUSTEXT':
if type(msg.text) == str:
msg.text = msg.text.encode()
return msg
# Modified from the snippet in your question
# UDP will work just as well or better
gcs_conn = mavutil.mavlink_connection('tcp:localhost:15795', input=False)
gcs_conn.wait_heartbeat()
print(f'Heartbeat from system (system {gcs_conn.target_system} component {gcs_conn.target_system})')
vehicle = mavutil.mavlink_connection('tcp:localhost:5760')
vehicle.wait_heartbeat()
print(f'Heartbeat from system (system {vehicle.target_system} component {vehicle.target_system})')
while True:
# Don't block for a GCS message - we have messages
# from the vehicle to get too
gcs_msg = gcs_conn.recv_match(blocking=False)
if gcs_msg is None:
pass
elif gcs_msg.get_type() != 'BAD_DATA':
# We now have a message we want to forward. Now we need to
# make it safe to send
gcs_msg = fixMAVLinkMessageForForward(gcs_msg)
# Finally, in order to forward this, we actually need to
# hack PyMAVLink so the message has the right source
# information attached.
vehicle.mav.srcSystem = gcs_msg.get_srcSystem()
vehicle.mav.srcComponent = gcs_msg.get_srcComponent()
# Only now is it safe to send the message
vehicle.mav.send(gcs_msg)
print(gcs_msg)
vcl_msg = vehicle.recv_match(blocking=False)
if vcl_msg is None:
pass
elif vcl_msg.get_type() != 'BAD_DATA':
# We now have a message we want to forward. Now we need to
# make it safe to send
vcl_msg = fixMAVLinkMessageForForward(vcl_msg)
# Finally, in order to forward this, we actually need to
# hack PyMAVLink so the message has the right source
# information attached.
gcs_conn.mav.srcSystem = vcl_msg.get_srcSystem()
gcs_conn.mav.srcComponent = vcl_msg.get_srcComponent()
gcs_conn.mav.send(vcl_msg)
print(vcl_msg)
# Don't abuse the CPU by running the loop at maximum speed
time.sleep(0.001)
Notes
Make sure your loop isn't being blocked
The loop must quickly check if a message is available from one connection or the other, instead of waiting for a message to be available from a single connection. Otherwise a message on the other connection will not go through until the blocking connection has a message.
Check message validity
Check that you actually got a valid message, as opposed to a BAD_DATA message. Attempting to send BAD_DATA will crash
Make sure the recipient gets the correct information about the sender
By default PyMAVLink, when sending a message, will encode YOUR system and component IDs (usually left at zero), instead of the IDs from the message. A GCS receiving this may be confused (ie, QGC) and not properly connect to the vehicle (despite showing the messages in MAVLink inspector).
This is fixed by hacking PyMAVLink such that your system and component IDs match the forwarded message. This can be revered after the message is sent if necessary. See the example to see how I did it.
Loop update rate
It's important that the update rate is fast enough to handle high traffic conditions (especially, say, for downloading params), but it shouldn't peg out the CPU either. I find that a 1000hz update rate works well enough.

KDB/Q Websocket example, getting `badmsg error: How to serialize my data when sending it over to my KDB server?

Lately, I've been trying to stream data from KDB to python.
I'm now using websockets and have gone through the doc https://code.kx.com/q/wp/kdb_and_websockets.pdf
On the python side, I've been trying ws4py, autobahn and websocket-client.
All do work fine, essentially my problem resides in the format of the message sent to the server to subscribe to the feed.
A little (open source) example:
class DummyClient(WebSocketClient):
def opened(self):
self.send(*what should I put here?*, binary=True)
def closed(self, code, reason=None):
print("Closed down", code, reason)
def received_message(self, m):
# not yet implemented
if __name__ == '__main__':
try:
ws = DummyClient('ws://host:port/', protocols=['http-only', 'chat'])
ws.connect()
ws.run_forever()
except KeyboardInterrupt:
ws.close()
When opening, I'm supposed to subscribe to the feed by calling the server function loadPage.
I've tried different things when to encode the list containing the function name and the argument, without success.
What I've tried:
np.array("['loadPage',[]]").tobytes()
"['loadPage',[]]".encode('utf8')
json formating
hexadecimal formating
Any help would be much appreciated!
Best,
Yael

I think what you'll need to do in this case is define .z.ws in your kdb server. This function will be called with whatever you pass over the websocket as an argument, so for example if you defined .z.ws like so:
.z.ws:{show x}
and then send a message over the WebSocket like so:
var ws = new WebSocket("ws://localhost:1234")
ws.send("Hello World")
This will be output in your kdb session e.g.
λ q
KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems
w64/ 4(16)core 8082MB jonat laptop-o8a8co1o 10.255.252.249 EXPIRE 2019.05.21 jonathon.mcmurray#aquaq.co.uk KOD #4160315
q).z.ws:{show x}
q)\p 1234
q)"Hello World"
So in your case if you want to call loadPage, it might be as simple as defining .z.ws like so:
.z.ws:loadPage
Then whatever you pass over the socket will be passed into loadPage (note this is will not update if you change loadPage elsewhere, you can instead use .z.ws:{loadPage x} if you need to be able to update it dynamically). For a niladic function, this will just ignore whatever is passed in

Azure IoTHub DeviceMessage and route filter

I use python and paho.mqtt for sending messages to cloud
I set up endpoint and route. When I set query string to true, everything works fine
messageDict = {}
systemPropertiesDict = {"contentType": "application/json", "contentEncoding": "utf-8", "iothub-message-source": "deviceMessages", "iothub-enqueuedtime": "2017-05-08T18:55:31.8514657Z"}
messageDict = {"systemProperties": systemPropertiesDict}
messageDict["appProperties"] = {}
body = '{id:1}'
messageDict["body"] = body
root = {"message":messageDict}
msg = json.dumps(root, indent=2).encode('utf-8')
print("Message to send", msg)
self.client.publish(topicName, msg)
But if I set the query string to $body.id = 1, then I don't receive any messages.
Any ideas, guys?

The route not working because the content encoding type is not set. All the "systemProperties" in your code actually as message body not system properties. Content encoding type set by this method doesn't take effect.
Add "$.ct=application%2Fjson&$.ce=utf-8" to the topic. Then it will look like this:
devices/{yourDeviceId}/messages/events/$.ct=application%2Fjson&$.ce=utf-8
But to make the route query works on your message you need use this query string: $body.message.body.id = 1
Two edits to make:
First, change body = '{id:1}' to body = {"id":1} to make the id as a string.
Second, change topicName value to this one:
devices/{yourDeviceId}/messages/events/$.ct=application%2Fjson&$.ce=utf-8
If possible, it is suggesting to use Azure IoT SDK for Python to communicate with Azure IoT Hub.

If you are using a third-party library (like paho-mqtt) you have to specify the content type and the encoding of the message.
IoT Hub route messages with content type "application/json" and content encoding: "utf-8" or "utf-16" or "utf-32".
Using MQTT protocol you can set this info with $.ct and $.ce.
topic example:
devices/{MY_DEVICE_ID}/messages/events/%24.ct=application%2fjson&%24.ce=utf-8
URL encoding of
devices/{MY_DEVICE_ID}/messages/events/$.ct=application/json&$.ce=utf-8
Here you could found more info:
https://azure.microsoft.com/it-it/blog/iot-hub-message-routing-now-with-routing-on-message-body/