I'm trying to improve network performance between gRPC client and server.
The client network speed is 1Gps, the
Assuming my server takes 200ms to respond, and I measure the latency in the client.
Now, if the server processing time goes up, say to 700ms for a response. Where the requests will accumulate? Will they stay in the client network queue, or they will still be sent to the server and will wait in the server queue?
In other words, does a grpc client holds a queue for requests, or every request is always sent - which means that the latency does not depends on the server processing time.
And is there a setting for it in grpc-python?
I suggest you to check the Client-side Interceptor and Server-side Interceptor classes.
Also, if you want to debug the requests, you can try to create immediate or with time interval multiple requests using JMeter or Postman Runner.
Related
As the title says, I have a question about the architecture and interaction of the client, NestJS backend, and Python microservice. I need to develop a recognition service. I have a client application that needs to send an HTTP request with an image to the NestJS backend, which should contact a third-party Python microservice, this service will recognize the text from the image (this can take a long time) and we should receive this response on the client application. What is the best way to implement this? I have an idea to connect NestJS with a Python microservice via RabbitMQ, the client sends a POST HTTP request to the NestJS backend, the backend sends a message to create a task via RPC to the Python microservice, which creates a Redis RQ queue with the task, returns the backend to the NestJS id of the task and starts to perform a long-running task, we return the task id to the client from the NestJS backend. After that, the client, with a certain interval, can send a GET HTTP request to the NestJS backend, which in turn will send a message to get the status of the task to the microservice and return the status to the client. Is this a good way to do it or is it possible to somehow optimize this process or implement it more competently?
I think you're on the right track here.
Send image to nest via HTTP - yes
Post job to redis queue - yes, use nestjs builtin queue handling (see docs), this will make it easier to consume the product of the job as well
Instead of having your client poll for a result, check out Server-sent Events
Server sent events are intended exactly for the use-case you are using.
In python gRPC, I tried grpc aio streaming. If there is just one client, the speed is acceptable. But when the number of clients increases, data transfer speed is slower and slower. I tried multi processing with sync gRPC, but sometimes it still route the request to a busy one, then the result is slow than expected. I tried multiprocessing with aio gRPC as well, by each process serves an aio gRPC server. It seems no big difference with multiprocessing with sync gRPC.
In addition, during the testing, it seems that the server is waiting till the client gets the message, then the server send the next one. The server may be affected by some very slow client.
Overall, my question is: If python is the only option, what is the best practice of creating a Data transfer service using python gRPC?
gRPC is based on HTTP2 protocol, and giving gRPC the advantage of multiplexing over one connection. It's possible that the bandwidth between client and server is consumed by one client and one server setup. As for multiprocessing, the SO_REUSEPORT is not a perfectly balanced load balancing for HTTP. You might need to configure your own routing mechanism(proxy/gateway).
it seems that the server is waiting till the client gets the message
Emm... technically, the server implementation hands over the message buffer to transport, then yield to event loop. If there are multiple ongoing RPCs, the server will start processing other coroutines. See code: https://github.com/grpc/grpc/blob/3ffa94f5e7334b7fdee6b173c25d71d42b512ac7/src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi#L142
If python is the only option, what is the best practice of creating a Data transfer service using python gRPC?
This question is a bit generic. If the throughput is the highest priority for the service, I would suggest to tune up the size of messages to MB-level. If there is no other compute-heavy workload, I would suggest to keep only one server process.
Is there a simple method or library to allow a websocket to drop certain messages if bandwidth doesn't allow? Or any one of the following?
to measure the queue size of outgoing messages that haven't yet reached a particular client
to measure the approximate bitrate that a client has been receiving recent messages at
to measure the time that a particular write_message finished being transmitted to the client
I'm using Tornado on the server side (tornado.websocket.WebSocketHandler) and vanilla JS on the client side. In my use case it's really only important that the server realize that a client is slow and throttle its messages (or use lossier compression) when it realizes that condition.
You can implement this on top of what you have by having the client confirm every message it gets and then use that information on the server to adapt the sending of messages to each client.
This is the only way you will know which outgoing messages haven't yet reached the client, be able to approximate bitrate or figure out the time it took for the message to reach the client. You must consider that the message back to the server will also take time and that if you use timestamps on the client, they will likely not match your servers as clients have their time set incorrectly more often than not.
I'm working with django-websocket-redis lib, that allow establish websockets over uwsgi in separated django loop.
By the documentation I understand well how to send data from server through websockets, but I don't understand how to receive.
Basically I have client and I want to send periodically from the client to server status. I don't understand what I need to do, to handle receiving messages from client on server side? What URL I should use on client?
You can achieve that by using periodically ajax calls from client to server. From documentation:
A client wishing to trigger events on the server side, shall use
XMLHttpRequests (Ajax), as they are much more suitable, rather than
messages sent via Websockets. The main purpose for Websockets is to
communicate asynchronously from the server to the client.
Unfortunately I was unable to find the way to achieve it using just websocket messages.
I'm trying to push some messages to client (browser) without any request.
For that I use WebSockets and python to do. I know WebSockets provide a full-duplex communication, but I just need server to client push. I could only find bi-directional communication examples for WebSockets over internet, for a request a response model. When I tried to send in an infinite loop from server after handshake process, the browser hung up.
the code I used is in this post
Is there any solution to do that or whether it is better to go for SSE..
Once the client initiates the web socket connection the server then is eligible to send anything to client side