Is there a process safe persistent (disk-based) Python FIFO queue?
Could someone provide a simple example with a script that writes a string to the queue, and another that reads one-string-at-a-time from the queue? Note that each of these processes can be launched multiple times from the command line. I.e. multiple writers and multiple readers.
Background: I have several scripts that each produces some output every once in a while. I would like to be aware of the output they create, and use another script(s) to process the information they produce. Unfortunately I cannot use the multiprocessing or threading modules of Python, because the scripts can run from different machines on the same file system. I.e. each of the scripts is launched from the command line.
All I need is that each of the queue elements is a string, and that the queue is process-safe.
Edit: I found the modules of queuelib pqueque, but they don't provide process safety. I am thinking of maybe replacing the file handling of queuelib, with atomicfile calls. But I am afraid spending too long on debugging it, since I am short of time, and there are always fine details to care when writing software that should be process safe.
Note1: I am using Python 2.7.
Note2: I am aware of this question, however my requirements are very modest, and I am looking for a simple solution, while the provided answers on that question refer to complex libraries, with no examples on how to use the queues.
EDIT (2019): For anyone landing here, I found that SQLite database provides atomic access, it has a python API, and it is serverless. Therefore, I believe it is possible (and easy) to implement the queue in my question with it.
I'm trying to implement a plugin architecture in Python.
I've started writing it using the Threading module where each plugin is a thread which I invoke using the Thread.start() method (since all plugins subclass BasePlugin which subclasses Thread). However I've just come across the multiprocessing module.
I'm currently wondering if I should switch to the multiprocessing module and share data using shared memory / Pipes etc...
I'd like to get other's opinions on this.
The plugin architecture I've been working on works as follows:
An event is received by the Plugin Manager. The Plugin Manager checks for all the plugins who've subscribed to that type of event. It activates them and sends them the event object (since it holds additional information). If one of the plugins is already active there is no need to spawn it (just send the event object to it).
In addition there are a few resources which belong only to one plugin at any point in time. Each plugin can request the resource (I'm not worrying about any race condition here since there won't be that many plugins active at once).
Threads share memory with the primary process and each other. For example you can have a list that is available to all threads. An item appended to a list can be seen by other threads. But you have to be careful. You have to understand which operations on data structures are thread safe and which are not. What happens to the behaviour of your program when two threads are checking for the existence of a key in a dictionary and then writing to it?
Multiple processes do not share memory. The new process that you start gets a copy of the memory at the point where it was spawned.
Threads use less resources. But can be hard to reason about. On the other hand communication between processes is tricky. And you can't just access an arbitrary Python data structure. Which it sounds like you want to be able to do.
A badly written plugin, if it was in a thread, could crash your whole program. Whereas if it was in a separate process this wouldn't happen. Maybe that's a consideration?
Anyone who has worked with Multithreaded PyQt4 apps? I was just wondering if the inbuilt signal/slot mechanism coupled with QtThread of PyQt4 framework has any benefit over the standard Python threads (which are designed in my code to handle the UI components in a thread safe way offcourse) using event driven async callback.
I am looking for any major speed or security concerns, any specific run-time exceptions or edge cases. (The UI is quite complex hence a re-write at a later stage would be very counter-productive).
Thanks.
Edit: I realize this might mean replicating some of the already present PyQt core functionality but it is ok if it allows more flexibility within the app.
There's no point really using Qt/PyQt if your're not using the signal-and-slot mechanism and rolling your own event loop. Basically, you'd be reimplementing the core of the framework itself. But I'm guessing this is not what you're asking about.
It would be nice if you could clarify your question a bit, (because of which I've had to make a few assumptions)
but here's the deal:
I think you're a bit confused about what the signal and slot mechanism does. (or maybe not, forgive me for reiterating some stuff that might probably be obvious to you).
The signals-and-slots do not implement threading for you (so the question of using signal/slots having any benefit over standard Python threads is moot)
You're probably assuming that the signal-slot mechanism is multithreaded, and that a slot when called by a signal, executes in a new thread. Well, this is not the case.
The signal and slot mechanism in Qt runs in a single event loop in Qt (implemented by QApplication), which itself runs in a single thread.
So signals and slots are not replacements for multi-threading in any way.
If there's a slot that blocks, then it will block your entire application.
So any blocking I/O or time intensive functions should ideally be in a separate thread from the UI, and your slots should start execution of these threads.
Now whether to use QThread or standard Python threads to implement your own threads is another issue, and it's been asked on StackOverflow before, but I tend to use QThreads for Qt apps.
So if you have a button, and you want to start a file download with the Requests library when its clicked, you'll connect the clicked signal of the QPushButton to a slot say for example downloadButtonClicked, and that slot would start a new QThread which would take care of downloading the file using Requests. You can further connect the finished() signal from the QThread to know when the download is complete and to update your UI
(I'll add a code example if this is indeed what you're asking about. So please clarify your question)
Based on your comment to another reply:
Sorry for the ambiguity, I was talking about QtThread Slot/Signal
Mechanism vs. callbacks using inbuilt Python Threads. I intend on
creating separate threads from UI on event arrival (clicks, etc) and
then use callbacks to the main UI thread from the new threads to
update the UI (all UI logic in the main thread with locks to keep it
thread safe.) I know this might mean replicating some of the already
present PyQt functionality but I feel this way I would have a lot more
control over my app. (The extra work isn't a concern if it allows more
flexibility in the app. Plus it isn't so much of work)
I would say that what you are after is to use QApplication.postEvent() from your threads. With a bit of extra code, you can use this to execute arbitrary methods in the main thread either synchronously or asynchronously.
I'm not sure there are really any advantages or disadvantages to either option (Qt or Python threads). As far as I'm aware, they both still hold the GIL, meaning your program is never truly multithreaded. QThreads come with an event loop, but as you say that isn't hard to write yourself in your own Python thread.
Have you considered using multiple processes instead of multiple threads? While slower to start, you get the advantage of actually having your code run in parallel.
At the end of the day, I think the answer to your question is simply personal preference. Mine would be to avoid using a QThread because it makes it easier to port your application to another widget toolkit in the future if PyQt\PySide\Qt ever die (not that it is very likely, but I've had a bad experience with PyGTK so now I'm wary)
EDIT: Please also look at this, as it has people far better answers than I've given: Threading in a PyQt application: Use Qt threads or Python threads?
I'm using PySide to manage some hardware and perform some relatively simple operations depending on (e.g.) button clicks in the interface. The code for running each of these pieces of hardware resides in another thread. For convenience, to all of those hardware drivers I've added a generic invoke_method signal, such that a UI component can use
my_driver.invoke_method.emit('method_name', [arg, ...], {kwarg, ...})
Obviously this accesses the signal attribute in that other thread directly.... but I'm not sure if this is necessarily okay in a GIL world.
If this is indeed too lazy a solution - are there any other great alternatives for invoking arbitrary methods in arbitrary threads without having to have an operation-specific signal in the UI connected to another signal in the driver?
I could imagine instead using a signal in each bit of UI code that accessed a different piece of hardware - something like do_invocation_driver_1 and do_invocation_driver_2 and connect those to the invoke_method signal of the corresponding driver.
I'd recommend reading this post for a general approach to interface threads with a PyQt GUI. The post discusses a thread that does socket I/O, but this really is applicable to any thread. Specifically, hardware-interface threads usually also use I/O, so this may be a good fit.
The approach discussed is very generic, using Queue.Queue, and may seem like an overkill for simple tasks (but I just want to call "that" function in a thread). However, once your application grows non-trivial, you will appreciate it, because it avoids any thread synchronization problems whatsoever, and is very scalable. I've personally used it to implement complex PyQt GUIs with side-threads doing all kinds of stuff.
I'm writing a GUI application that regularly retrieves data through a web connection. Since this retrieval takes a while, this causes the UI to be unresponsive during the retrieval process (it cannot be split into smaller parts). This is why I'd like to outsource the web connection to a separate worker thread.
[Yes, I know, now I have two problems.]
Anyway, the application uses PyQt4, so I'd like to know what the better choice is: Use Qt's threads or use the Python threading module? What are advantages / disadvantages of each? Or do you have a totally different suggestion?
Edit (re bounty): While the solution in my particular case will probably be using a non-blocking network request like Jeff Ober and Lukáš Lalinský suggested (so basically leaving the concurrency problems to the networking implementation), I'd still like a more in-depth answer to the general question:
What are advantages and disadvantages of using PyQt4's (i.e. Qt's) threads over native Python threads (from the threading module)?
Edit 2: Thanks all for you answers. Although there's no 100% agreement, there seems to be widespread consensus that the answer is "use Qt", since the advantage of that is integration with the rest of the library, while causing no real disadvantages.
For anyone looking to choose between the two threading implementations, I highly recommend they read all the answers provided here, including the PyQt mailing list thread that abbot links to.
There were several answers I considered for the bounty; in the end I chose abbot's for the very relevant external reference; it was, however, a close call.
Thanks again.
This was discussed not too long ago in PyQt mailing list. Quoting Giovanni Bajo's comments on the subject:
It's mostly the same. The main difference is that QThreads are better
integrated with Qt (asynchrnous signals/slots, event loop, etc.).
Also, you can't use Qt from a Python thread (you can't for instance
post event to the main thread through QApplication.postEvent): you
need a QThread for that to work.
A general rule of thumb might be to use QThreads if you're going to interact somehow with Qt, and use Python threads otherwise.
And some earlier comment on this subject from PyQt's author: "they are both wrappers around the same native thread implementations". And both implementations use GIL in the same way.
Python's threads will be simpler and safer, and since it is for an I/O-based application, they are able to bypass the GIL. That said, have you considered non-blocking I/O using Twisted or non-blocking sockets/select?
EDIT: more on threads
Python threads
Python's threads are system threads. However, Python uses a global interpreter lock (GIL) to ensure that the interpreter is only ever executing a certain size block of byte-code instructions at a time. Luckily, Python releases the GIL during input/output operations, making threads useful for simulating non-blocking I/O.
Important caveat: This can be misleading, since the number of byte-code instructions does not correspond to the number of lines in a program. Even a single assignment may not be atomic in Python, so a mutex lock is necessary for any block of code that must be executed atomically, even with the GIL.
QT threads
When Python hands off control to a 3rd party compiled module, it releases the GIL. It becomes the responsibility of the module to ensure atomicity where required. When control is passed back, Python will use the GIL. This can make using 3rd party libraries in conjunction with threads confusing. It is even more difficult to use an external threading library because it adds uncertainty as to where and when control is in the hands of the module vs the interpreter.
QT threads operate with the GIL released. QT threads are able to execute QT library code (and other compiled module code that does not acquire the GIL) concurrently. However, the Python code executed within the context of a QT thread still acquires the GIL, and now you have to manage two sets of logic for locking your code.
In the end, both QT threads and Python threads are wrappers around system threads. Python threads are marginally safer to use, since those parts that are not written in Python (implicitly using the GIL) use the GIL in any case (although the caveat above still applies.)
Non-blocking I/O
Threads add extraordinarily complexity to your application. Especially when dealing with the already complex interaction between the Python interpreter and compiled module code. While many find event-based programming difficult to follow, event-based, non-blocking I/O is often much less difficult to reason about than threads.
With asynchronous I/O, you can always be sure that, for each open descriptor, the path of execution is consistent and orderly. There are, obviously, issues that must be addressed, such as what to do when code depending on one open channel further depends on the results of code to be called when another open channel returns data.
One nice solution for event-based, non-blocking I/O is the new Diesel library. It is restricted to Linux at the moment, but it is extraordinarily fast and quite elegant.
It is also worth your time to learn pyevent, a wrapper around the wonderful libevent library, which provides a basic framework for event-based programming using the fastest available method for your system (determined at compile time).
The advantage of QThread is that it's integrated with the rest of the Qt library. That is, thread-aware methods in Qt will need to know in which thread they run, and to move objects between threads, you will need to use QThread. Another useful feature is running your own event loop in a thread.
If you are accessing a HTTP server, you should consider QNetworkAccessManager.
I asked myself the same question when I was working to PyTalk.
If you are using Qt, you need to use QThread to be able to use the Qt framework and expecially the signal/slot system.
With the signal/slot engine, you will be able to talk from a thread to another and with every part of your project.
Moreover, there is not very performance question about this choice since both are a C++ bindings.
Here is my experience of PyQt and thread.
I encourage you to use QThread.
Jeff has some good points. Only one main thread can do any GUI updates. If you do need to update the GUI from within the thread, Qt-4's queued connection signals make it easy to send data across threads and will automatically be invoked if you're using QThread; I'm not sure if they will be if you're using Python threads, although it's easy to add a parameter to connect().
I can't really recommend either, but I can try describing differences between CPython and Qt threads.
First of all, CPython threads do not run concurrently, at least not Python code. Yes, they do create system threads for each Python thread, however only the thread currently holding Global Interpreter Lock is allowed to run (C extensions and FFI code might bypass it, but Python bytecode is not executed while thread doesn't hold GIL).
On the other hand, we have Qt threads, which are basically common layer over system threads, don't have Global Interpreter Lock, and thus are capable of running concurrently. I'm not sure how PyQt deals with it, however unless your Qt threads call Python code, they should be able to run concurrently (bar various extra locks that might be implemented in various structures).
For extra fine-tuning, you can modify the amount of bytecode instructions that are interpreted before switching ownership of GIL - lower values mean more context switching (and possibly higher responsiveness) but lower performance per individual thread (context switches have their cost - if you try switching every few instructions it doesn't help speed.)
Hope it helps with your problems :)
I can't comment on the exact differences between Python and PyQt threads, but I've been doing what you're attempting to do using QThread, QNetworkAcessManager and making sure to call QApplication.processEvents() while the thread is alive. If GUI responsiveness is really the issue you're trying to solve, the later will help.