I am creating an application using Python.
I first designed an API, that is working fine.
I am now designing my GUI. The GUI starts a Thread that is used to perform tasks against the API.
Up to now, I used the Observer pattern to handle communication through the different layers.
Basically, communication can be of two types (mainly):
- The GUI asking the Thread (and the API subsequently) to START/STOP
- The API giving information back to the Thread, that propagates to the GUI.
Here is a simple schema of the current architecture I am talking about.
One arrow means "notify", basically.
My concern is that when the application Thread communicates, both the Gui and the API receive the message because they subscribed. Thing is, each message is only meant to be read by one of the two.
What I did to solve that is to send an message together with an ID. Each of the three elements have an id and they know whether the message is for them or now.
But I am not sure if this is the "correct" (understand nicest) way to do it. What if I have more parties in the future ?
I started thinking about some kind of manager handling communication, but It would then have to be at the top of the architecture, and I am not sure how to further organize it :s.
I am not asking for a complete solution, but mainly ideas or best practises by more experienced people ;)
I can keep handling multiple Observer pattern in this simple case.
But I was thinking about porting my code on a server. In this case, I am likely to have way more than one thread for the application, and handling API calls will become quite impossible.
Link to the code I am talking about :
GUI, ApplicationThread and Application API.
You want to look at notify and update methods.
Thx for any piece of advice !
One of the nice implementation of the observer pattern I've met is the signal/slot system in Qt. Objects have signals, and slots (which are actually methods) can be connected to signals. The connected slots are called when the signals are emitted.
It seems to me that some of your problems may stem from the fact you have single communication canal in each of your objects. This forces you to have some dispatch mechanism in every update method, and makes the code quite complex.
Taking inspiration from Qt, you could have different signals for each kind of message and recipient. The code for signal would look like :
class Signal:
def __init__(self):
self.subs = []
def subscribe(self, s):
self.subs.append(s)
def signal(self, *args, **kwargs):
for s in self.subs:
s(*args, **kwargs)
For example, the gui would have a signal stop_signal and the thread a method to handle it :
def handle_gui_stop(self):
self.console_logger.debug("Facemovie is going to stop")
self.my_logger.debug("Facemovie is going to stop")
self.stop_process = True
# ...
Somewhere in the initialization code, we would tie everything together :
gui.stop_signal.subscribe(thread.handle_gui_stop)
I recently created a GUI app with similar architecture (GUI thread + a separate work thread), and I end up creating an explicit protocol between threads, in a form of two queues (from Queue Python module). One queue is for requests made by GUI and is consumed by worker thread(s). The other queue is for answers produces by worker threads and consumed by GUI.
I find it much clearer when communication between threads is explicit, you have full control over when and where the updating is done (GUI methods can be called only from the GUI thread).
A natural extension of this model in a server environment is a message queue protocol like AMQP.
Application thread must be more explicit about communications since it's the communicator between GUI and Application API. This may be achieved by separating working sets (queues) coming from GUI and from Application API. Also, Application Thread must be able to handle delivery-type of pattern, which includes command senders and recipients. This includes managing communications between different queues (e.g. GUI queue has pending command which is awaiting for the command in the Application API queue. Once this one completes, then the Application Thread passes results back between the queues). And each queue is the observer itself.
In terms of extending the application, it seems to me that you want to add more GUIs in the future, which will be handled by request/response (or sender/receiver) pattern implementation described above (that will suffice).
If you plan to add more layers vertically instead of horisontally then you should not use the same Application Thread to communicate upper between new layers. Well, physically it can be the same, but virtually it must be different at least. This can be achieved by exactly implementing what I described above again (separate queues). By introducing dynamically adding queues you will open the possibility to add a new layer (new layer simply corresponds to new queue then).
Specially with GUIs, I recomend another pattern: MVC. It includes the Observer patterns in it and is more robust than the Observer alone.
It solves your concern because it separates concerns: each layer has a very specific role and you can change any of them, as long as you don't change the interface between them.
Related
I'm trying to implement a plugin architecture in Python.
I've started writing it using the Threading module where each plugin is a thread which I invoke using the Thread.start() method (since all plugins subclass BasePlugin which subclasses Thread). However I've just come across the multiprocessing module.
I'm currently wondering if I should switch to the multiprocessing module and share data using shared memory / Pipes etc...
I'd like to get other's opinions on this.
The plugin architecture I've been working on works as follows:
An event is received by the Plugin Manager. The Plugin Manager checks for all the plugins who've subscribed to that type of event. It activates them and sends them the event object (since it holds additional information). If one of the plugins is already active there is no need to spawn it (just send the event object to it).
In addition there are a few resources which belong only to one plugin at any point in time. Each plugin can request the resource (I'm not worrying about any race condition here since there won't be that many plugins active at once).
Threads share memory with the primary process and each other. For example you can have a list that is available to all threads. An item appended to a list can be seen by other threads. But you have to be careful. You have to understand which operations on data structures are thread safe and which are not. What happens to the behaviour of your program when two threads are checking for the existence of a key in a dictionary and then writing to it?
Multiple processes do not share memory. The new process that you start gets a copy of the memory at the point where it was spawned.
Threads use less resources. But can be hard to reason about. On the other hand communication between processes is tricky. And you can't just access an arbitrary Python data structure. Which it sounds like you want to be able to do.
A badly written plugin, if it was in a thread, could crash your whole program. Whereas if it was in a separate process this wouldn't happen. Maybe that's a consideration?
Anyone who has worked with Multithreaded PyQt4 apps? I was just wondering if the inbuilt signal/slot mechanism coupled with QtThread of PyQt4 framework has any benefit over the standard Python threads (which are designed in my code to handle the UI components in a thread safe way offcourse) using event driven async callback.
I am looking for any major speed or security concerns, any specific run-time exceptions or edge cases. (The UI is quite complex hence a re-write at a later stage would be very counter-productive).
Thanks.
Edit: I realize this might mean replicating some of the already present PyQt core functionality but it is ok if it allows more flexibility within the app.
There's no point really using Qt/PyQt if your're not using the signal-and-slot mechanism and rolling your own event loop. Basically, you'd be reimplementing the core of the framework itself. But I'm guessing this is not what you're asking about.
It would be nice if you could clarify your question a bit, (because of which I've had to make a few assumptions)
but here's the deal:
I think you're a bit confused about what the signal and slot mechanism does. (or maybe not, forgive me for reiterating some stuff that might probably be obvious to you).
The signals-and-slots do not implement threading for you (so the question of using signal/slots having any benefit over standard Python threads is moot)
You're probably assuming that the signal-slot mechanism is multithreaded, and that a slot when called by a signal, executes in a new thread. Well, this is not the case.
The signal and slot mechanism in Qt runs in a single event loop in Qt (implemented by QApplication), which itself runs in a single thread.
So signals and slots are not replacements for multi-threading in any way.
If there's a slot that blocks, then it will block your entire application.
So any blocking I/O or time intensive functions should ideally be in a separate thread from the UI, and your slots should start execution of these threads.
Now whether to use QThread or standard Python threads to implement your own threads is another issue, and it's been asked on StackOverflow before, but I tend to use QThreads for Qt apps.
So if you have a button, and you want to start a file download with the Requests library when its clicked, you'll connect the clicked signal of the QPushButton to a slot say for example downloadButtonClicked, and that slot would start a new QThread which would take care of downloading the file using Requests. You can further connect the finished() signal from the QThread to know when the download is complete and to update your UI
(I'll add a code example if this is indeed what you're asking about. So please clarify your question)
Based on your comment to another reply:
Sorry for the ambiguity, I was talking about QtThread Slot/Signal
Mechanism vs. callbacks using inbuilt Python Threads. I intend on
creating separate threads from UI on event arrival (clicks, etc) and
then use callbacks to the main UI thread from the new threads to
update the UI (all UI logic in the main thread with locks to keep it
thread safe.) I know this might mean replicating some of the already
present PyQt functionality but I feel this way I would have a lot more
control over my app. (The extra work isn't a concern if it allows more
flexibility in the app. Plus it isn't so much of work)
I would say that what you are after is to use QApplication.postEvent() from your threads. With a bit of extra code, you can use this to execute arbitrary methods in the main thread either synchronously or asynchronously.
I'm not sure there are really any advantages or disadvantages to either option (Qt or Python threads). As far as I'm aware, they both still hold the GIL, meaning your program is never truly multithreaded. QThreads come with an event loop, but as you say that isn't hard to write yourself in your own Python thread.
Have you considered using multiple processes instead of multiple threads? While slower to start, you get the advantage of actually having your code run in parallel.
At the end of the day, I think the answer to your question is simply personal preference. Mine would be to avoid using a QThread because it makes it easier to port your application to another widget toolkit in the future if PyQt\PySide\Qt ever die (not that it is very likely, but I've had a bad experience with PyGTK so now I'm wary)
EDIT: Please also look at this, as it has people far better answers than I've given: Threading in a PyQt application: Use Qt threads or Python threads?
I'm quite new to python threading/network programming, but have an assignment involving both of the above.
One of the requirements of the assignment is that for each new request, I spawn a new thread, but I need to both send and receive at the same time to the browser.
I'm currently using the asyncore library in Python to catch each request, but as I said, I need to spawn a thread for each request, and I was wondering if using both the thread and the asynchronous is overkill, or the correct way to do it?
Any advice would be appreciated.
Thanks
EDIT:
I'm writing a Proxy Server, and not sure if my client is persistent. My client is my browser (using firefox for simplicity)
It seems to reconnect for each request. My problem is that if I open a tab with http://www.google.com in it, and http://www.stackoverflow.com in it, I only get one request at a time from each tab, instead of multiple requests from google, and from SO.
I answered a question that sounds amazingly similar to your, where someone had a homework assignment to create a client server setup, with each connection being handled in a new thread: https://stackoverflow.com/a/9522339/496445
The general idea is that you have a main server loop constantly looking for a new connection to come in. When it does, you hand it off to a thread which will then do its own monitoring for new communication.
An extra bit about asyncore vs threading
From the asyncore docs:
There are only two ways to have a program on a single processor do
“more than one thing at a time.” Multi-threaded programming is the
simplest and most popular way to do it, but there is another very
different technique, that lets you have nearly all the advantages of
multi-threading, without actually using multiple threads. It’s really
only practical if your program is largely I/O bound. If your program
is processor bound, then pre-emptive scheduled threads are probably
what you really need. Network servers are rarely processor bound,
however.
As this quote suggests, using asyncore and threading should be for the most part mutually exclusive options. My link above is an example of the threading approach, where the server loop (either in a separate thread or the main one) does a blocking call to accept a new client. And when it gets one, it spawns a thread which will then continue to handle the communication, and the server goes back into a blocking call again.
In the pattern of using asyncore, you would instead use its async loop which will in turn call your own registered callbacks for various activity that occurs. There is no threading here, but rather a polling of all the open file handles for activity. You get the sense of doing things all concurrently, but under the hood it is scheduling everything serially.
I'm using PySide to manage some hardware and perform some relatively simple operations depending on (e.g.) button clicks in the interface. The code for running each of these pieces of hardware resides in another thread. For convenience, to all of those hardware drivers I've added a generic invoke_method signal, such that a UI component can use
my_driver.invoke_method.emit('method_name', [arg, ...], {kwarg, ...})
Obviously this accesses the signal attribute in that other thread directly.... but I'm not sure if this is necessarily okay in a GIL world.
If this is indeed too lazy a solution - are there any other great alternatives for invoking arbitrary methods in arbitrary threads without having to have an operation-specific signal in the UI connected to another signal in the driver?
I could imagine instead using a signal in each bit of UI code that accessed a different piece of hardware - something like do_invocation_driver_1 and do_invocation_driver_2 and connect those to the invoke_method signal of the corresponding driver.
I'd recommend reading this post for a general approach to interface threads with a PyQt GUI. The post discusses a thread that does socket I/O, but this really is applicable to any thread. Specifically, hardware-interface threads usually also use I/O, so this may be a good fit.
The approach discussed is very generic, using Queue.Queue, and may seem like an overkill for simple tasks (but I just want to call "that" function in a thread). However, once your application grows non-trivial, you will appreciate it, because it avoids any thread synchronization problems whatsoever, and is very scalable. I've personally used it to implement complex PyQt GUIs with side-threads doing all kinds of stuff.
Im currently trying to learn Networking with Python asyncore and pyqt4.
I coded a small server, which basically listens on some port, and resends all messages it recieves to the sender.
Since both qts QApplication.exec_() and asyncore.loop() are functions which never return i could not start them both in one thread, so i stared asyncore.loop() in a seperate daemon thread.
Whenever my server class (derived from asyncore.dispatcher) establishes or drops a connection, or sends/recieves a message, it calls methods of my window class (derived from QtGui.QMainWindow), which displays the information in a QPlainTextEdit.
But the text is not visible, unless you mark the text with the mouse.
Python console displays following error msg:
QObject::connect: Cannot queue arguments of type 'QTextBlock'
(Make sure 'QTextBlock' is registered using qRegisterMetaType().)
QObject::connect: Cannot queue arguments of type 'QTextCursor'
(Make sure 'QTextCursor' is registered using qRegisterMetaType().)
I read on some forum, that this may be caused by calling qt-functions from another Thread, and that using signals & slots instead of plain function calling may fix the issue, but i have tried signals aswell, and i still get this error.
So, (if that is really the cause of my problems) whats the correct way to call methods of an qt object from another thread ?
EDIT More Info:
the asyncore.loop() call is located in the child thread, well its not really blocking, but only during the runtime of asyncore.loop() my Server class (asyncore.dispatcher) can do networking.
So, during the runtime of asyncore.loop() the methods of my Server class ARE called by asyncore.loop() (=child thread), and in these i
tried to emit signals to the window class running in the main thread
EDIT: Seems like i got it working now, i had some errors in my code, everything works as intended with signals now.
EDIT: small example: http://paste2.org/p/635612 (dead link)
It appears you're trying to access QtGui classes from a thread other than the main thread. Like in some other GUI toolkits (e.g. Java Swing), that's not allowed. From the Threads and QObjects web page:
Although QObject is reentrant, the GUI
classes, notably QWidget and all its
subclasses, are not reentrant. They
can only be used from the main thread.
A solution is to use signals and slots for communication between the main thread (where the GUI objects live) and your secondary thread(s). Basically, you emit signals in one thread that get delivered to the QObjects via the other thread. The page I linked to above has a good discussion of this. Actually, the whole section on Thread Support in Qt is a good read.
One potential issue you could run into is that, normally, to get full signals and slots support working across threads, you need to start an event loop in the child thread using QThread::exec() (or the PyQt equivalent) so that signals can be delivered to slots in the QObjects that live there. In your case, it sounds like you're making a blocking call to asyncore.loop(), which will prevent you from doing this. But, if you only need to emit signals in one direction (from the child thread to widgets in the main thread), I don't think you'll have a problem.