Develop Reference

Sharing DB client among multiple processes in Python?

My python application uses concurrent.futures.ProcessPoolExecutor with 5 workers and each process makes multiple database queries.
Between the choice of giving each process its own db client, or alternatively , making all process to share a single client, which is considered more safe and conventional?

Short answer: Give each process (that needs it) its own db client.
Long answer: What problem are you trying to solve?
Sharing a DB client between processes basically doesn't happen; you'd have to have the one process which does have the DB client proxy the queries from the others, using more-or-less your own protocol. That can have benefits, if that protocol is specific to your application, but it will add complexity: you'll now have two different kinds of workers in your program, rather than just one kind, plus the protocol between them. You'd want to make sure that the benefits outweigh the additional complexity.
Sharing a DB client between threads is usually possible; you'd have to check the documentation to see which objects and operations are "thread-safe". However, since your application is otherwise CPU-heavy, threading is not suitable, due to Python limitations (the GIL).
At the same time, there's little cost to having a DB client in each process; you will in any case need some sort of client, it might as well be the direct one.
There isn't going to be much more IO, since that's mostly based on the total number of queries and amount of data, regardless of whether that comes from one process or gets spread among several. The only additional IO will be in the login, and that's not much.
If you're running out of connections at the database, you can either tune/upgrade your database for more connections, or use a separate off-the-shelf "connection pooler" to share them; that's likely to be much better than trying to implement a connection pooler from scratch.
More generally, and this applies well beyond this particular question, it's often better to combine several off-the-shelf pieces in a straightforward way, than it is to try to put together a custom complex piece that does the whole thing all at once.
So, what problem are you trying to solve?

It is better to use multithreading or asynchronous approach instead of multiprocessing because it will consume fewer resources. That way you could use a single db connection, but I would recommend creating a separate session for each worker or coroutine to avoid some exceptions or problems with locking.

Concurrently searching a graph in Python 3

I'd like to create a small p2p application that concurrently processes incoming data from other known / trusted nodes (it mostly stores it in an SQLite database). In order to recognize these nodes, upon connecting, each node introduces itself and my application then needs to check whether it knows this node directly or maybe indirectly through another node. Hence, I need to do a graph search which obviously needs processing time and which I'd like to outsource to a separate process (or even multiple worker processes? See my 2nd question below). Also, in some cases it is necessary to adjust the graph, add new edges or vertices.
Let's say I have 4 worker processes accepting and handling incoming connections via asynchronous I/O. What's the best way for them to access (read / modify) the graph? A single queue obviously doesn't do the trick for read access because I need to pass the search results back somehow.
Hence, one way to do it would be another queue which would be filled by the graph searching process and which I could add to the event loop. The event loop could then pass the results to a handler. However, this event/callback-based approach would make it necessary to also always pass the corresponding sockets to the callbacks and thus to the Queue – which is nasty because sockets are not picklable. (Let alone the fact that callbacks lead to spaghetti code.)
Another idea that's just crossed my mind might be to create a pipe to the graph process for each incoming connection and then, on the graph's side, do asynchronous I/O as well. However, in order to avoid callbacks, if I understand correctly, I would need an async I/O library making use of yield from (i.e. tulip / PEP 3156). Are there other options?
Regarding async I/O on the graph's side: This is certainly the best way to handle many incoming requests at once but doing graph lookups is a CPU intensive task, thus could profit from using multiple worker threads or processes. The problem is: Multiple threads allow shared data but Python's GIL somewhat negates the performance benefit. Multiple processes on the other hand don't have this problem but how can I share and synchronize data between them? (For me it seems quite impossible to split up a graph.) Is there any way to solve this problem in a nice way? Also, does it make sense in terms of performance to mix asynchronous I/O with multithreading / multiprocessing?

Answering your last question: It does! But, IMHO, the question is: does it makes sense mix Events and Threads? You can check this article about hybrid concurrency models: http://bibliotecadigital.sbc.org.br/download.php?paper=3027
My tip: Start with just one process and an event loop, like in the tulip model. I'll try to explain how can you use tulip to have Events+async I/O (and threads or other processes) without callbacks at all.
You could have something like accept = yield from check_incoming(), which should be a tulip coroutine (check_incoming), and inside this function you could use loop.run_in_executor() to run your graph search in a thread/process pool (I'll explain more about this later). This function run_in_executor() returns a Future, in which you can also yield from tasks.wait([future_returned_by_run_in_executor], loop=self). The next step would be result = future_returned_by_run_in_executor.result() and finally return True or False.
The process pool requires that only pickable objects can be executed and returned. This requirement is not a problem but it's implicit that the graph operation must be self contained in a function and must obtain the graph instance somehow. The Thread pool has the GIL problem since you mentioned CPU bound tasks which can lead to 'acquiring-gil-conflicts' but this was improved in the new Python 3.x GIL. Both solutions have limitations..
So.. instead of a pool, you can have another single process with it's own event loop just to manage all the graph work and connect both processes with a unix domain socket for instance..
This second process, just like the first one, must also accept incoming connections (but now they are from a known source) and can use a thread pool just like I said earlier but it won't "conflict" with the first event loop process(the one that handles external clients), only with the second event loop. Threads sharing the same graph instance requires some locking/unlocking.
Hope it helped!

Handle communication in multi layer application

I am creating an application using Python.
I first designed an API, that is working fine.
I am now designing my GUI. The GUI starts a Thread that is used to perform tasks against the API.
Up to now, I used the Observer pattern to handle communication through the different layers.
Basically, communication can be of two types (mainly):
- The GUI asking the Thread (and the API subsequently) to START/STOP
- The API giving information back to the Thread, that propagates to the GUI.
Here is a simple schema of the current architecture I am talking about.
One arrow means "notify", basically.
My concern is that when the application Thread communicates, both the Gui and the API receive the message because they subscribed. Thing is, each message is only meant to be read by one of the two.
What I did to solve that is to send an message together with an ID. Each of the three elements have an id and they know whether the message is for them or now.
But I am not sure if this is the "correct" (understand nicest) way to do it. What if I have more parties in the future ?
I started thinking about some kind of manager handling communication, but It would then have to be at the top of the architecture, and I am not sure how to further organize it :s.
I am not asking for a complete solution, but mainly ideas or best practises by more experienced people ;)
I can keep handling multiple Observer pattern in this simple case.
But I was thinking about porting my code on a server. In this case, I am likely to have way more than one thread for the application, and handling API calls will become quite impossible.
Link to the code I am talking about :
GUI, ApplicationThread and Application API.
You want to look at notify and update methods.
Thx for any piece of advice !

One of the nice implementation of the observer pattern I've met is the signal/slot system in Qt. Objects have signals, and slots (which are actually methods) can be connected to signals. The connected slots are called when the signals are emitted.
It seems to me that some of your problems may stem from the fact you have single communication canal in each of your objects. This forces you to have some dispatch mechanism in every update method, and makes the code quite complex.
Taking inspiration from Qt, you could have different signals for each kind of message and recipient. The code for signal would look like :
class Signal:
def __init__(self):
self.subs = []
def subscribe(self, s):
self.subs.append(s)
def signal(self, *args, **kwargs):
for s in self.subs:
s(*args, **kwargs)
For example, the gui would have a signal stop_signal and the thread a method to handle it :
def handle_gui_stop(self):
self.console_logger.debug("Facemovie is going to stop")
self.my_logger.debug("Facemovie is going to stop")
self.stop_process = True
# ...
Somewhere in the initialization code, we would tie everything together :
gui.stop_signal.subscribe(thread.handle_gui_stop)

I recently created a GUI app with similar architecture (GUI thread + a separate work thread), and I end up creating an explicit protocol between threads, in a form of two queues (from Queue Python module). One queue is for requests made by GUI and is consumed by worker thread(s). The other queue is for answers produces by worker threads and consumed by GUI.
I find it much clearer when communication between threads is explicit, you have full control over when and where the updating is done (GUI methods can be called only from the GUI thread).
A natural extension of this model in a server environment is a message queue protocol like AMQP.

Application thread must be more explicit about communications since it's the communicator between GUI and Application API. This may be achieved by separating working sets (queues) coming from GUI and from Application API. Also, Application Thread must be able to handle delivery-type of pattern, which includes command senders and recipients. This includes managing communications between different queues (e.g. GUI queue has pending command which is awaiting for the command in the Application API queue. Once this one completes, then the Application Thread passes results back between the queues). And each queue is the observer itself.
In terms of extending the application, it seems to me that you want to add more GUIs in the future, which will be handled by request/response (or sender/receiver) pattern implementation described above (that will suffice).
If you plan to add more layers vertically instead of horisontally then you should not use the same Application Thread to communicate upper between new layers. Well, physically it can be the same, but virtually it must be different at least. This can be achieved by exactly implementing what I described above again (separate queues). By introducing dynamically adding queues you will open the possibility to add a new layer (new layer simply corresponds to new queue then).

Specially with GUIs, I recomend another pattern: MVC. It includes the Observer patterns in it and is more robust than the Observer alone.
It solves your concern because it separates concerns: each layer has a very specific role and you can change any of them, as long as you don't change the interface between them.

How do I add two integers together with Twisted?

I have two integers in my program; let's call them "a" and "b". I would like to add them together and get another integer as a result. These are regular Python int objects. I'm wondering; how do I add them together with Twisted? Is there a special performAsynchronousAddition function somewhere? Do I need a Deferred? What about the reactor? Is the reactor involved?

OK, to be clear.
Twisted doesn't do anything about cpu bound tasks and for good reason. there's no way to make a compute bound job go any quicker by reordering subtasks; the only thing you could possibly do is add more compute resources; and even that wouldn't work out in python because of a subtlety of its implementation.
Twisted offers special semantics and event loop handling in case the program would become "stuck" waiting for something outside if its control; most normally a process running on another machine and communicating with your twisted process over a network connection. Since you would be waiting anyways, twisted gives you a mechanism to get more things done in the meantime. That is to say, twisted provides concurrency for I/O Bound tasks
tl;dr: twisted is for network code. Everything else is just normal python.

How about this:
c = a + b
That should work, and it doesn't need to be done asynchronously (it's pretty fast).

Good question, and Twisted (or Python) should have a way to at least spawn "a + b" of to several cores (on my 8 core i7).
Unfortunately the Python GIL prevents this from happening, meaning that you will have to wait, not only for the CPU bound task, but for one core doing the job, while the seven others core are doing nothing.
Note: Maybe a better example would be "a() + b()", or even "fact(sqrt(a()**b())" etc. but the important fact is that above operation will lock one core and the GIL pretty much prevents Python for doing anything else during that operation, which could be several ms...

Twisted: Making code non-blocking

I'm a bit puzzled about how to write asynchronous code in python/twisted. Suppose (for arguments sake) I am exposing a function to the world that will take a number and return True/False if it is prime/non-prime, so it looks vaguely like this:
def IsPrime(numberin):
for n in range(2,numberin):
if numberin % n == 0: return(False)
return(True)
(just to illustrate).
Now lets say there is a webserver which needs to call IsPrime based on a submitted value. This will take a long time for large numberin.
If in the meantime another user asks for the primality of a small number, is there a way to run the two function calls asynchronously using the reactor/deferreds architecture so that the result of the short calc gets returned before the result of the long calc?
I understand how to do this if the IsPrime functionality came from some other webserver to which my webserver would do a deferred getPage, but what if it's just a local function?
i.e., can Twisted somehow time-share between the two calls to IsPrime, or would that require an explicit invocation of a new thread?
Or, would the IsPrime loop need to be chunked into a series of smaller loops so that control can be passed back to the reactor rapidly?
Or something else?

I think your current understanding is basically correct. Twisted is just a Python library and the Python code you write to use it executes normally as you would expect Python code to: if you have only a single thread (and a single process), then only one thing happens at a time. Almost no APIs provided by Twisted create new threads or processes, so in the normal course of things your code runs sequentially; isPrime cannot execute a second time until after it has finished executing the first time.
Still considering just a single thread (and a single process), all of the "concurrency" or "parallelism" of Twisted comes from the fact that instead of doing blocking network I/O (and certain other blocking operations), Twisted provides tools for performing the operation in a non-blocking way. This lets your program continue on to perform other work when it might otherwise have been stuck doing nothing waiting for a blocking I/O operation (such as reading from or writing to a socket) to complete.
It is possible to make things "asynchronous" by splitting them into small chunks and letting event handlers run in between these chunks. This is sometimes a useful approach, if the transformation doesn't make the code too much more difficult to understand and maintain. Twisted provides a helper for scheduling these chunks of work, cooperate. It is beneficial to use this helper since it can make scheduling decisions based on all of the different sources of work and ensure that there is time left over to service event sources without significant additional latency (in other words, the more jobs you add to it, the less time each job will get, so that the reactor can keep doing its job).
Twisted does also provide several APIs for dealing with threads and processes. These can be useful if it is not obvious how to break a job into chunks. You can use deferToThread to run a (thread-safe!) function in a thread pool. Conveniently, this API returns a Deferred which will eventually fire with the return value of the function (or with a Failure if the function raises an exception). These Deferreds look like any other, and as far as the code using them is concerned, it could just as well come back from a call like getPage - a function that uses no extra threads, just non-blocking I/O and event handlers.
Since Python isn't ideally suited for running multiple CPU-bound threads in a single process, Twisted also provides a non-blocking API for launching and communicating with child processes. You can offload calculations to such processes to take advantage of additional CPUs or cores without worrying about the GIL slowing you down, something that neither the chunking strategy nor the threading approach offers. The lowest level API for dealing with such processes is reactor.spawnProcess. There is also Ampoule, a package which will manage a process pool for you and provides an analog to deferToThread for processes, deferToAMPProcess.