SQLAlchemy + Requests Asynchronous Pattern - python

I am currently working on an application where a client makes some call to a web services, some small amount of processing is done on the JSON data returned, and then that is stored in a database. I am currently using Requests and SQLAlchemy. The amount of processing is very small (just changing the data to a more relational format). I am not using the ORM for SA. I am just using the engine + transactions.
I was wondering what the a good pattern to do this asynchronously would be (request returned -> handed off to database -> the next request starts without waiting for the DB to finish transaction).
I know that there are a number of tools available in Python (multiprocessing, threads, coroutines, asyncore, etc). However, I am having difficulty finding a good tutorial for my use case.
I was wondering if anyone had suggestions, libraries I should look at, or async patterns that would help me solve this problem.
Thanks.

You can push each request in a Queue and let a set of worker threads handle each one of them and push them to the DB.
Here is a simple example of the worker body:
import threading
import time
from Queue import Queue, Empty
from random import choice
class worker(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.q = Queue()
def run(self):
while True:
try:
r = self.q.get_nowait()
except Empty:
r = None
if r is None:
time.sleep(1.0)
continue
# do something with 'r'
print '%s: Handled request %s' % (self, r)
def push(self, r):
self.q.put(r)
workers = [worker() for i in range(5)]
for w in workers:
w.start()
Then distribute the requests to workers like this:
choice(workers).push(req)

Related

The gRPC object creates extra processes that don't close

I developed the application with gRPC servicer. The point of my application is:
gRPC servicer (class DexFxServicer in the code below) has Transmit method which is called by gRPC client outside.
Transmit method creates multiple channels and stubs for the different hosts from hostList.
Further application creates the process pool and launches it.
Each child process calls gRPC method SendHostListAndGetMetrics for its own stub and receives response iterator.
This code works well, the application invokes Transmit method and receive all needed results from the process pool. But I noticed when outside gRPC client calls Transmit method multiple times, this code didn't close some of its child processes. And it leads to extra nonclosing processes creation as htop shows.
When I try to close gRPC channels by channel.close() method, extra processes are being created more intensively.
Python 2.7.12
grpcio==1.16.1
grpcio-tools==1.16.1
Ubuntu 16.04.6 LTS 4.4.0-143-generic
from concurrent import futures
import sleep
import grpc
import sys
import cascade_pb2
import cascade_pb2_grpc
import metrics_pb2
import metrics_pb2_grpc
from multiprocessing import Pool
class DexFxServicer(cascade_pb2_grpc.DexFxServicer):
def __init__(self, args):
self.args = args
def Transmit(self, request, context):
entrypoint = request.sender.host_address # entrypoint is a string
hostList = [] # hostList is a list of strings
for rec in request.sender.receiver:
hostList.append(rec.host_address)
channels = {}
stubs = {}
for host in hostList:
try:
channels[host] = grpc.insecure_channel('%s:%d' % (host, self.args.cascadePort))
except Exception as e:
print(e)
sys.exit(0)
else:
stubs[host] = metrics_pb2_grpc.MetricsStub(channels[host])
def collect_metrics(host):
mtrx = []
hosts = (metrics_pb2.Host(hostname = i) for i in hostList + [entrypoint])
for i in stubs[host].SendHostListAndGetMetrics(hosts):
mtrx.append(i.mtrx)
return mtrx
pool = Pool(len(hostList))
results = pool.map(collect_metrics, hostList)
pool.close()
pool.terminate()
pool.join()
# Return the iterator of the results
I expect to see the code which doesn't create extra nonclosing processes. Please, suggest me what to do in this case.
The problem was solved by means of update grpcio version to 1.23.0. gRPC issue

Multithreading/Multiprocessing in Python 3.4 web app for Azure

I'm working on writing a Python web app with Flask using Azure to host. I need to do some background work. When I test the app locally, everything works great. However, as soon as I push the update to Azure, it stops functioning. Right now, I have a multithreading.Process set up, and based on the log files, Azure isn't starting another process. Here is the relevant parts of my code:
#task queue and comm pipes
tasks = Queue()
parent_pipe, child_pipe = Pipe()
def handle_queue_execution(tasks, pipe):
logging.info("starting task queue handler")
while True:
if pipe.recv():
logging.debug("preparing to get task from queue")
task = tasks.get()
args = tasks.get()
logging.debug("executing task %s(%s)", get_fn_name(task), clean_args(args))
task(args)
logging.debug("task %s(%s) executed successfully", get_fn_name(task), clean_args(args))
queue_handler = Process(target=handle_queue_execution, args=(tasks, child_pipe,))
queue_handler.daemon = True
if __name__ == '__main__':
queue_handler.start()
There are a few semi-related questions I have on this:
1) Why won't Azure start another process?
You'll note that the handle_queue_execution function begins with a logger call. That message doesn't appear in the log file when hosted on Azure, nor do the queued tasks appear to execute. Again, both aspects of this work as expected when running on localhost.
2) Is there a better way?
I'm fairly new to both Python and Azure, so if there's a better way to do this type of task handling, I'm open to hear about it. I've looked into using something like Celery, but I can't figure out how to set it up, and I'd prefer to make my own implementation as I'm learning these new skills.
Thanks very much.
Python has multiple other ways to start new processes. Threading would most likely be the easiest here.
#task queue and comm pipes
import threading
tasks = Queue()
parent_pipe, child_pipe = Pipe()
def handle_queue_execution(tasks, pipe):
logging.info("starting task queue handler")
while True:
if pipe.recv():
logging.debug("preparing to get task from queue")
task = tasks.get()
args = tasks.get()
logging.debug("executing task %s(%s)", get_fn_name(task), clean_args(args))
task(args)
logging.debug("task %s(%s) executed successfully", get_fn_name(task), clean_args(args))
T1 = threading.Thread(target=handle_que_execution, args=(tasks, child_pipe,))
if __name__ == '__main__':
T1.start()

Python how can I do a multithreading/asynchronous HTTP server with twisted

Now I wrote ferver by this tutorial:
https://twistedmatrix.com/documents/14.0.0/web/howto/web-in-60/asynchronous-deferred.html
But it seems to be good only for delayng process, not actually concurently process 2 or more requests. My full code is:
from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import Site, NOT_DONE_YET
from twisted.internet import reactor, threads
from time import sleep
class DelayedResource(Resource):
def _delayedRender(self, request):
print 'Sorry to keep you waiting.'
request.write("<html><body>Sorry to keep you waiting.</body></html>")
request.finish()
def make_delay(self, request):
print 'Sleeping'
sleep(5)
return request
def render_GET(self, request):
d = threads.deferToThread(self.make_delay, request)
d.addCallback(self._delayedRender)
return NOT_DONE_YET
def main():
root = Resource()
root.putChild("social", DelayedResource())
factory = Site(root)
reactor.listenTCP(8880, factory)
print 'started httpserver...'
reactor.run()
if __name__ == '__main__':
main()
But when I passing 2 requests console output is like:
Sleeping
Sorry to keep you waiting.
Sleeping
Sorry to keep you waiting.
But if it was concurrent it should be like:
Sleeping
Sleeping
Sorry to keep you waiting.
Sorry to keep you waiting.
So the question is how to make twisted not to wait until response is finished before processing next?
Also make_delayIRL is a large function with heavi logic. Basically I spawn lot of threads and make requests to other urls and collecting results intro response, so it can take some time and not easly to be ported
Twisted processes everything in one event loop. If somethings blocks the execution, it also blocks Twisted. So you have to prevent blocking calls.
In your case you have time.sleep(5). It is blocking. You found the better way to do it in Twisted already: deferLater(). It returns a Deferred that will continue execution after the given time and release the events loop so other things can be done meanwhile. In general all things that return a deferred are good.
If you have to do heavy work that for some reason can not be deferred, you should use deferToThread() to execute this work in a thread. See https://twistedmatrix.com/documents/15.5.0/core/howto/threading.html for details.
You can use greenlents in your code (like threads).
You need to install the geventreactor - https://gist.github.com/yann2192/3394661
And use reactor.deferToGreenlet()
Also
In your long-calculation code need to call gevent.sleep() for change context to another greenlet.
msecs = 5 * 1000
timeout = 100
for xrange(0, msecs, timeout):
sleep(timeout)
gevent.sleep()

Asynchronous Client/Server pattern in Python ZeroMQ

I have 3 programs written in Python, which need to be connected. 2 programs X and Y gather some information, which are sent by them to program Z. Program Z analyzes the data and send to program X and Y some decisions. Number of programs similar to X and Y will be expanded in the future. Initially I used named pipe to allow communication from X, Y to Z. But as you can see, I need bidirectional relation. My boss told me to use ZeroMQ. I have just found pattern for my use case, which is called Asynchronous Client/Server. Please see code from ZMQ book (http://zguide.zeromq.org/py:all) below.
The problem is my boss does not want to use any threads, forks etc. I moved client and server tasks to separate programs, but I am not sure what to do with ServerWorker class. Can this be somehow used without threads? Also, I am wondering, how to establish optimal workers amount.
import zmq
import sys
import threading
import time
from random import randint, random
__author__ = "Felipe Cruz <felipecruz#loogica.net>"
__license__ = "MIT/X11"
def tprint(msg):
"""like print, but won't get newlines confused with multiple threads"""
sys.stdout.write(msg + '\n')
sys.stdout.flush()
class ClientTask(threading.Thread):
"""ClientTask"""
def __init__(self, id):
self.id = id
threading.Thread.__init__ (self)
def run(self):
context = zmq.Context()
socket = context.socket(zmq.DEALER)
identity = u'worker-%d' % self.id
socket.identity = identity.encode('ascii')
socket.connect('tcp://localhost:5570')
print('Client %s started' % (identity))
poll = zmq.Poller()
poll.register(socket, zmq.POLLIN)
reqs = 0
while True:
reqs = reqs + 1
print('Req #%d sent..' % (reqs))
socket.send_string(u'request #%d' % (reqs))
for i in range(5):
sockets = dict(poll.poll(1000))
if socket in sockets:
msg = socket.recv()
tprint('Client %s received: %s' % (identity, msg))
socket.close()
context.term()
class ServerTask(threading.Thread):
"""ServerTask"""
def __init__(self):
threading.Thread.__init__ (self)
def run(self):
context = zmq.Context()
frontend = context.socket(zmq.ROUTER)
frontend.bind('tcp://*:5570')
backend = context.socket(zmq.DEALER)
backend.bind('inproc://backend')
workers = []
for i in range(5):
worker = ServerWorker(context)
worker.start()
workers.append(worker)
poll = zmq.Poller()
poll.register(frontend, zmq.POLLIN)
poll.register(backend, zmq.POLLIN)
while True:
sockets = dict(poll.poll())
if frontend in sockets:
ident, msg = frontend.recv_multipart()
tprint('Server received %s id %s' % (msg, ident))
backend.send_multipart([ident, msg])
if backend in sockets:
ident, msg = backend.recv_multipart()
tprint('Sending to frontend %s id %s' % (msg, ident))
frontend.send_multipart([ident, msg])
frontend.close()
backend.close()
context.term()
class ServerWorker(threading.Thread):
"""ServerWorker"""
def __init__(self, context):
threading.Thread.__init__ (self)
self.context = context
def run(self):
worker = self.context.socket(zmq.DEALER)
worker.connect('inproc://backend')
tprint('Worker started')
while True:
ident, msg = worker.recv_multipart()
tprint('Worker received %s from %s' % (msg, ident))
replies = randint(0,4)
for i in range(replies):
time.sleep(1. / (randint(1,10)))
worker.send_multipart([ident, msg])
worker.close()
def main():
"""main function"""
server = ServerTask()
server.start()
for i in range(3):
client = ClientTask(i)
client.start()
server.join()
if __name__ == "__main__":
main()
So, you grabbed the code from here: Asynchronous Client/Server Pattern
Pay close attention to the images that show you the model this code is targeted to. In particular, look at "Figure 38 - Detail of Asynchronous Server". The ServerWorker class is spinning up 5 "Worker" nodes. In the code, those nodes are threads, but you could make them completely separate programs. In that case, your server program (probably) wouldn't be responsible for spinning them up, they'd spin up separately and just communicate to your server that they are ready to receive work.
You'll see this often in ZMQ examples, a multi-node topology mimicked in threads in a single executable. It's just to make reading the whole thing easy, it's not always intended to be used that way.
For your particular case, it could make sense to have the workers be threads or to break them out into separate programs... but if it's a business requirement from your boss, then just break them out into separate programs.
Of course, to answer your second question, there's no way to know how many workers would be optimal without understanding the work load they'll be performing and how quickly they'll need to respond... your goal is to have the worker complete the work faster than new work is received. There's a fair chance, in many cases, that that can be accomplished with a single worker. If so, you can have your server itself be the worker, and just skip the entire "worker tier" of the architecture. You should start there, for the sake of simplicity, and just do some load testing to see if it will actually cope with your workload effectively. If not, get a sense of how long it takes to complete a task, and how quickly tasks are coming in. Let's say a worker can complete a task in 15 seconds. That's 4 tasks a minute. If tasks are coming in 5 tasks a minute, you need 2 workers, and you'll have a little headroom to grow. If things are wildly variable, then you'll have to make a decision about resources vs. reliability.
Before you get too much farther down the trail, make sure you read Chapter 4, Reliable Request/Reply Patterns, it will provide some insight for handling exceptions, and might give you a better pattern to follow.

concurrency of heavy tasks in tornado

my code:
import tornado.tcpserver
import tornado.ioloop
import itertools
import socket
import time
class Talk():
def __init__(self, id):
self.id = id
#tornado.gen.coroutine
def on_connect(self):
try:
while "connection alive":
self.said = yield self.stream.read_until(b"\n")
response = yield tornado.gen.Task(self.task) ### LINE 1
yield self.stream.write(response) ### LINE 2
except tornado.iostream.StreamClosedError:
print('error: socket closed')
return
#tornado.gen.coroutine
def task(self):
if self.id == 1:
time.sleep(3) # sometimes request is heavy blocking
return b"response"
#tornado.gen.coroutine
def on_disconnect(self):
yield []
class Server(tornado.tcpserver.TCPServer):
def __init__(self, io_loop=None, ssl_options=None, max_buffer_size=None):
tornado.tcpserver.TCPServer.__init__(self,
io_loop=io_loop,
ssl_options=ssl_options,
max_buffer_size=max_buffer_size)
self.talk_id_alloc = itertools.count(1)
return
#tornado.gen.coroutine
def handle_stream(self, stream, address):
talk_id = next(self.talk_id_alloc)
talk = Talk(talk_id)
stream.set_close_callback(talk.on_disconnect)
stream.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
stream.socket.setsockopt(socket.IPPROTO_TCP, socket.SO_KEEPALIVE, 1)
talk.stream = stream
yield talk.on_connect()
return
Server().listen(8888)
tornado.ioloop.IOLoop.instance().start()
problem:
I need a tornado as tcp server - it looks like a good choice for handling many requests with low computation.
however:
99% of requests will last less than 0,05 sec, but
1% of them can last even 3 sec (special cases).
single response must be returned at once, not partially.
what is best aproach here?
how to achieve a code where LINE #1 is never blocking more than 0.1 sec
yield tornado.gen.with_timeout(
datetime.timedelta(seconds=0.1), tornado.gen.Task(self.task))
doesnt work form me - do nothing
tornado.ioloop.IOLoop.current().add_timeout(
datetime.timedelta(seconds=0.1),
lambda: result.set_exception(TimeoutError("Timeout")))
either nothing.
looking for better solutions:
task can detect if need high computation (API ...) - using timeout?,
then run/fork to another thread or even process
and send to tornado server execption - "receive" me later from results queue (consumer/producer)
i dont want case where timeout kill heavy task without saving results, and task is reopened within special wrapper - so consumer/producer pattern should be for all tasks?
adding new ioloop when current is blocked - how detect blocking?
I dont see any solution in tornado.
task in line #1 could be simple (~99%) or complicated, which can require:
I/O:
- disk/DB access
- ram/redis access
network:
- API call
CPU:
- algorithms, regex
(the worst task will do all of above).
I know what kind of task it is (the weight) only when I start doing it,
so appriopriate is use a task queue in separate threads.
I dont want delay simple/quick tasks.
so if you manage to cancel the heavy tasks, I recommend cancelling them with a time-out and then spawning them off to another thread. Performance-wise this is not ideal (GIL) but you prevent tornado from blocking - which is your ultimate goal.
A nice write-up about how this can be done can be found here: http://lbolla.info/blog/2013/01/22/blocking-tornado.
If you want to go further you could use something like celery where you can offload to other processes transparently - though this much heavier.

Categories

Resources