Develop Reference

Architecture and interaction of the client, NestJS backend, and Python microservice

As the title says, I have a question about the architecture and interaction of the client, NestJS backend, and Python microservice. I need to develop a recognition service. I have a client application that needs to send an HTTP request with an image to the NestJS backend, which should contact a third-party Python microservice, this service will recognize the text from the image (this can take a long time) and we should receive this response on the client application. What is the best way to implement this? I have an idea to connect NestJS with a Python microservice via RabbitMQ, the client sends a POST HTTP request to the NestJS backend, the backend sends a message to create a task via RPC to the Python microservice, which creates a Redis RQ queue with the task, returns the backend to the NestJS id of the task and starts to perform a long-running task, we return the task id to the client from the NestJS backend. After that, the client, with a certain interval, can send a GET HTTP request to the NestJS backend, which in turn will send a message to get the status of the task to the microservice and return the status to the client. Is this a good way to do it or is it possible to somehow optimize this process or implement it more competently?

I think you're on the right track here.
Send image to nest via HTTP - yes
Post job to redis queue - yes, use nestjs builtin queue handling (see docs), this will make it easier to consume the product of the job as well
Instead of having your client poll for a result, check out Server-sent Events
Server sent events are intended exactly for the use-case you are using.

ZeroMQ bidirectional async communication with subprocesses

I have a server process which receives requests from a web clients.
The server has to call an external worker process ( another .py ) which streams data to the server and the server streams back to the client.
The server has to monitor these worker processes and send messages to them ( basically kill them or send messages to control which kind of data gets streamed ). These messages are asynchronous ( e.g. depend on the web client )
I thought in using ZeroMQ sockets over an ipc://-transport-class , but the call for socket.recv() method is blocking.
Should I use two sockets ( one for streaming data to the server and another to receive control messages from server )?

Using a separate socket for signalling and messaging is always better
While a Poller-instance will help a bit, the cardinal step is to use separate socket for signalling and another one for data-streaming. Always. The point is, that in such setup, both the Poller.poll() and the event-loop can remain socket-specific and spent not more than a predefined amount of time, during a real-time controlled code-execution.
So, do not hesitate to setup a bit richer signalling/messaging infrastructure as an environment where you will only enjoy the increased simplicity of control, separation of concerns and clarity of intents.
ZeroMQ is an excellent tool for doing this - including per-socket IO-thread affinity, so indeed a fine-grain performance tuning is available at your fingertips.

I think if figured out a solution, but I don't know if there is a better (more efficient, safer, ...) way of doing this.
The client makes a request to the server, which spawns N processes worker to attend the request.
This is the relevant excerpt from worker.py:
for i in range(start_counter,10):
# Check if there is any message from server
while True:
try:
msg = worker.recv(zmq.DONTWAIT)
print("Received {} from server".format(msg))
except zmq.Again:
break
# Send data to server
worker.send(b"Message {} from {}".format(i, worker_id))
# Take some sleep
time.sleep(random.uniform(0.3, 1.1))
In this way, the worker a) does not need a separate socket and b) does not need a separate thread to process messages from server.
In the real implementation, worker must stream 128 byte messages at 100Hz to the server, and the server must receive lots of this messages (many clients asking requests that need 3-10 worker each).
Will this approach suffer a performance hit if implemented this way?

HTTP Callback URL vs. WebSocket for ansynchronous response?

I have two servers: Golang and Python (2.7). The Python (Bottle) server has a computation intensive task to perform and exposes a RESTful URI to start the execution of the process. That is, the Go server sends:
HTTP GET to myserver.com/data
The python server performs the computation and needs to inform the Go server of the completion of the processing. There are two ways in which I see this can be designed:
Go sends a callback URL/data to Python and python responds by hitting that URL. E.g:
HTTP GET | myserver.com/data | Data{callbackURI:goserver.com/process/results, Type: POST, response:"processComplete"}
Have a WebSocket based response be sent back from Python to Go.
What would be a more suitable design? Are there pros/cons of doing one over the other? Other than error conditions (server crashed etc.,) the only thing that the Python server needs to actually "inform" the client is about completing the computation. That's the only response.
The team working on the Go server is not very well versed with having a Go client based on websockets/ajax (nor do I. But I've never written a single line of Go :) #1 seems to be easier but am not aware of whether it is an accepted design approach or is it just a hack? What's the recommended way to proceed in this regard?

If you want to do it RESTful, then when the client requests HTTP GET myserver.com/data the server should return a 202 Accepted status code:
202 Accepted
The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.
The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.
The Python server could return an ETA and an URL to a temporary resource to request the current status of the operation (e.g.: myserver.com/temp_data?processing_status). Then it's up to the Go client to wait for the task to fulfill by requesting this resource and reading the ETA. Once the processing is done, the Python server could return a 410 Gone status with the definitive URL of the new resource.

It depends on how often these signal are being sent. If it's many times per second, keeping a websocket open might make more sense. Otherwise, use option #1 since it will have less overhead and be more loosely coupled.

Distributing task with python and zeromq

I have a working application using python and zeromq and I would like to optimize it.
Briefly, a master node send the same request to all workers (about 200) and the then collect the answers. Based on the answer, it sends a message back to one node and the node answers back.
Right now I implemented a very simple pattern. Each worker has one REP socket and the server has a list of REQ sockets. The server iterates through all sockets sending the general message and then iterates through all sockets to collect the answers. Finally, based on the answers the server picks one worker, sends a message to it and waits for the reply.
This is, of course, quite slow. The slowest part is sending 200 times the same message. Collecting is also slow. The solutions that I have found to distribute tasks and collect answers do load balance which is not what I need. I need that each worker receives the message and responds.
What is the pattern recommended for this situation?
Thanks

If each worker should need a different job, Pub/sub wont work . What you need then is a worker pool Implementation. Either you push out the jobs in a round robin style (just use the push socket bound on the server and have each client pull from it, zeromq will do the round robin) or you have each worker request a job from the server, if the jobs are at least of some minimum complexity and the variance between jobs is high that is a better approach. There are numerous examples in the zeromq guide on its homepage and on the net:
http://zguide.zeromq.org/page:all (look at the paronoid pirate pattern)
http://blog.garambrogne.net/post/2010/10/23/simple-python-work-queue-with-zeromq
https://github.com/marklit/zeromq-worker-queue
The concrete implementation also depends on whether or not you need to have jobs reliably processed.

I don't know zmq. Here's a pattern that might not work, just to get started:
a master node send the same request to all workers (about 200)
master PUB bind *:3140 send
worker SUB connect masterhost:3140 SUBSCRIBE recv
the then collect the answers
worker PUSH connect masterhost:3141 send
master PULL bind *:3141 recv
Based on the answer, it sends a message back to one node and the node answers back.
master REQ connect workerhost:3142 send recv
worker REP bind *:3142 recv send

Message queue proxy in Python + Twisted

I want to implement a lightweight Message Queue proxy. It's job is to receive messages from a web application (PHP) and send them to the Message Queue server asynchronously. The reason for this proxy is that the MQ isn't always avaliable and is sometimes lagging, or even down, but I want to make sure the messages are delivered, and the web application returns immediately.
So, PHP would send the message to the MQ proxy running on the same host. That proxy would save the messages to SQLite for persistence, in case of crashes. At the same time it would send the messages from SQLite to the MQ in batches when the connection is available, and delete them from SQLite.
Now, the way I understand, there are these components in this service:
message listener (listens to the messages from PHP and writes them to a Incoming Queue)
DB flusher (reads messages from the Incoming Queue and saves them to a database; due to SQLite single-threadedness)
MQ connection handler (keeps the connection to the MQ server online by reconnecting)
message sender (collects messages from SQlite db and sends them to the MQ server, then removes them from db)
I was thinking of using Twisted for #1 (TCPServer), but I'm having problem with integrating it with other points, which aren't event-driven. Intuition tells me that each of these points should be running in a separate thread, because all are IO-bound and independent of each other, but I could easily put them in a single thread. Even though, I couldn't find any good and clear (to me) examples on how to implement this worker thread aside of Twisted's main loop.
The example I've started with is the chatserver.py, which uses service.Application and internet.TCPServer objects. If I start my own thread prior to creating TCPServer service, it runs a few times, but the it stops and never runs again. I'm not sure, why this is happening, but it's probably because I don't use threads with Twisted correctly.
Any suggestions on how to implement a separate worker thread and keep Twisted? Do you have any alternative architectures in mind?

You're basically considering writing an ad-hoc extension to your messaging server, the job of which it is to provide whatever reliability guarantees you've asked of it.
Instead, perhaps you should take the hardware where you were planning to run this new proxy and run another MQ node on it. The new node should take care of persisting and relaying messages that you deliver to it while the other nodes are overloaded or offline.

Maybe it's not the best bang for your buck to use a separate thread in Twisted to get around a blocking call, but sometimes the least evil solution is the best. Here's a link that shows you how to integrate threading into Twisted:
http://twistedmatrix.com/documents/10.1.0/core/howto/threading.html
Sometimes in a pinch easy-to-implement is faster than hours/days of research which may all turn out to be for nought.

A neat solution to this problem would be to use the Key Value store Redis. Its a high speed persistent data store, with plenty of clients - it has a php and a python client (if you want to use a timed/batch process to process messages - it saves you creating a database, and also deals with your persistence stories. It runs fine on Cywin/Windows + posix environments.
PHP Redis client is here.
Python client is here.
Both have a very clean and simple API. Redis also offers a publish/subscribe mechanism, should you need it, although it sounds like it would be of limited value if you're publishing to an inconsistent queue.