I'm writing a GUI application that regularly retrieves data through a web connection. Since this retrieval takes a while, this causes the UI to be unresponsive during the retrieval process (it cannot be split into smaller parts). This is why I'd like to outsource the web connection to a separate worker thread.
[Yes, I know, now I have two problems.]
Anyway, the application uses PyQt4, so I'd like to know what the better choice is: Use Qt's threads or use the Python threading module? What are advantages / disadvantages of each? Or do you have a totally different suggestion?
Edit (re bounty): While the solution in my particular case will probably be using a non-blocking network request like Jeff Ober and Lukáš Lalinský suggested (so basically leaving the concurrency problems to the networking implementation), I'd still like a more in-depth answer to the general question:
What are advantages and disadvantages of using PyQt4's (i.e. Qt's) threads over native Python threads (from the threading module)?
Edit 2: Thanks all for you answers. Although there's no 100% agreement, there seems to be widespread consensus that the answer is "use Qt", since the advantage of that is integration with the rest of the library, while causing no real disadvantages.
For anyone looking to choose between the two threading implementations, I highly recommend they read all the answers provided here, including the PyQt mailing list thread that abbot links to.
There were several answers I considered for the bounty; in the end I chose abbot's for the very relevant external reference; it was, however, a close call.
Thanks again.
This was discussed not too long ago in PyQt mailing list. Quoting Giovanni Bajo's comments on the subject:
It's mostly the same. The main difference is that QThreads are better
integrated with Qt (asynchrnous signals/slots, event loop, etc.).
Also, you can't use Qt from a Python thread (you can't for instance
post event to the main thread through QApplication.postEvent): you
need a QThread for that to work.
A general rule of thumb might be to use QThreads if you're going to interact somehow with Qt, and use Python threads otherwise.
And some earlier comment on this subject from PyQt's author: "they are both wrappers around the same native thread implementations". And both implementations use GIL in the same way.
Python's threads will be simpler and safer, and since it is for an I/O-based application, they are able to bypass the GIL. That said, have you considered non-blocking I/O using Twisted or non-blocking sockets/select?
EDIT: more on threads
Python threads
Python's threads are system threads. However, Python uses a global interpreter lock (GIL) to ensure that the interpreter is only ever executing a certain size block of byte-code instructions at a time. Luckily, Python releases the GIL during input/output operations, making threads useful for simulating non-blocking I/O.
Important caveat: This can be misleading, since the number of byte-code instructions does not correspond to the number of lines in a program. Even a single assignment may not be atomic in Python, so a mutex lock is necessary for any block of code that must be executed atomically, even with the GIL.
QT threads
When Python hands off control to a 3rd party compiled module, it releases the GIL. It becomes the responsibility of the module to ensure atomicity where required. When control is passed back, Python will use the GIL. This can make using 3rd party libraries in conjunction with threads confusing. It is even more difficult to use an external threading library because it adds uncertainty as to where and when control is in the hands of the module vs the interpreter.
QT threads operate with the GIL released. QT threads are able to execute QT library code (and other compiled module code that does not acquire the GIL) concurrently. However, the Python code executed within the context of a QT thread still acquires the GIL, and now you have to manage two sets of logic for locking your code.
In the end, both QT threads and Python threads are wrappers around system threads. Python threads are marginally safer to use, since those parts that are not written in Python (implicitly using the GIL) use the GIL in any case (although the caveat above still applies.)
Non-blocking I/O
Threads add extraordinarily complexity to your application. Especially when dealing with the already complex interaction between the Python interpreter and compiled module code. While many find event-based programming difficult to follow, event-based, non-blocking I/O is often much less difficult to reason about than threads.
With asynchronous I/O, you can always be sure that, for each open descriptor, the path of execution is consistent and orderly. There are, obviously, issues that must be addressed, such as what to do when code depending on one open channel further depends on the results of code to be called when another open channel returns data.
One nice solution for event-based, non-blocking I/O is the new Diesel library. It is restricted to Linux at the moment, but it is extraordinarily fast and quite elegant.
It is also worth your time to learn pyevent, a wrapper around the wonderful libevent library, which provides a basic framework for event-based programming using the fastest available method for your system (determined at compile time).
The advantage of QThread is that it's integrated with the rest of the Qt library. That is, thread-aware methods in Qt will need to know in which thread they run, and to move objects between threads, you will need to use QThread. Another useful feature is running your own event loop in a thread.
If you are accessing a HTTP server, you should consider QNetworkAccessManager.
I asked myself the same question when I was working to PyTalk.
If you are using Qt, you need to use QThread to be able to use the Qt framework and expecially the signal/slot system.
With the signal/slot engine, you will be able to talk from a thread to another and with every part of your project.
Moreover, there is not very performance question about this choice since both are a C++ bindings.
Here is my experience of PyQt and thread.
I encourage you to use QThread.
Jeff has some good points. Only one main thread can do any GUI updates. If you do need to update the GUI from within the thread, Qt-4's queued connection signals make it easy to send data across threads and will automatically be invoked if you're using QThread; I'm not sure if they will be if you're using Python threads, although it's easy to add a parameter to connect().
I can't really recommend either, but I can try describing differences between CPython and Qt threads.
First of all, CPython threads do not run concurrently, at least not Python code. Yes, they do create system threads for each Python thread, however only the thread currently holding Global Interpreter Lock is allowed to run (C extensions and FFI code might bypass it, but Python bytecode is not executed while thread doesn't hold GIL).
On the other hand, we have Qt threads, which are basically common layer over system threads, don't have Global Interpreter Lock, and thus are capable of running concurrently. I'm not sure how PyQt deals with it, however unless your Qt threads call Python code, they should be able to run concurrently (bar various extra locks that might be implemented in various structures).
For extra fine-tuning, you can modify the amount of bytecode instructions that are interpreted before switching ownership of GIL - lower values mean more context switching (and possibly higher responsiveness) but lower performance per individual thread (context switches have their cost - if you try switching every few instructions it doesn't help speed.)
Hope it helps with your problems :)
I can't comment on the exact differences between Python and PyQt threads, but I've been doing what you're attempting to do using QThread, QNetworkAcessManager and making sure to call QApplication.processEvents() while the thread is alive. If GUI responsiveness is really the issue you're trying to solve, the later will help.
Related
Anyone who has worked with Multithreaded PyQt4 apps? I was just wondering if the inbuilt signal/slot mechanism coupled with QtThread of PyQt4 framework has any benefit over the standard Python threads (which are designed in my code to handle the UI components in a thread safe way offcourse) using event driven async callback.
I am looking for any major speed or security concerns, any specific run-time exceptions or edge cases. (The UI is quite complex hence a re-write at a later stage would be very counter-productive).
Thanks.
Edit: I realize this might mean replicating some of the already present PyQt core functionality but it is ok if it allows more flexibility within the app.
There's no point really using Qt/PyQt if your're not using the signal-and-slot mechanism and rolling your own event loop. Basically, you'd be reimplementing the core of the framework itself. But I'm guessing this is not what you're asking about.
It would be nice if you could clarify your question a bit, (because of which I've had to make a few assumptions)
but here's the deal:
I think you're a bit confused about what the signal and slot mechanism does. (or maybe not, forgive me for reiterating some stuff that might probably be obvious to you).
The signals-and-slots do not implement threading for you (so the question of using signal/slots having any benefit over standard Python threads is moot)
You're probably assuming that the signal-slot mechanism is multithreaded, and that a slot when called by a signal, executes in a new thread. Well, this is not the case.
The signal and slot mechanism in Qt runs in a single event loop in Qt (implemented by QApplication), which itself runs in a single thread.
So signals and slots are not replacements for multi-threading in any way.
If there's a slot that blocks, then it will block your entire application.
So any blocking I/O or time intensive functions should ideally be in a separate thread from the UI, and your slots should start execution of these threads.
Now whether to use QThread or standard Python threads to implement your own threads is another issue, and it's been asked on StackOverflow before, but I tend to use QThreads for Qt apps.
So if you have a button, and you want to start a file download with the Requests library when its clicked, you'll connect the clicked signal of the QPushButton to a slot say for example downloadButtonClicked, and that slot would start a new QThread which would take care of downloading the file using Requests. You can further connect the finished() signal from the QThread to know when the download is complete and to update your UI
(I'll add a code example if this is indeed what you're asking about. So please clarify your question)
Based on your comment to another reply:
Sorry for the ambiguity, I was talking about QtThread Slot/Signal
Mechanism vs. callbacks using inbuilt Python Threads. I intend on
creating separate threads from UI on event arrival (clicks, etc) and
then use callbacks to the main UI thread from the new threads to
update the UI (all UI logic in the main thread with locks to keep it
thread safe.) I know this might mean replicating some of the already
present PyQt functionality but I feel this way I would have a lot more
control over my app. (The extra work isn't a concern if it allows more
flexibility in the app. Plus it isn't so much of work)
I would say that what you are after is to use QApplication.postEvent() from your threads. With a bit of extra code, you can use this to execute arbitrary methods in the main thread either synchronously or asynchronously.
I'm not sure there are really any advantages or disadvantages to either option (Qt or Python threads). As far as I'm aware, they both still hold the GIL, meaning your program is never truly multithreaded. QThreads come with an event loop, but as you say that isn't hard to write yourself in your own Python thread.
Have you considered using multiple processes instead of multiple threads? While slower to start, you get the advantage of actually having your code run in parallel.
At the end of the day, I think the answer to your question is simply personal preference. Mine would be to avoid using a QThread because it makes it easier to port your application to another widget toolkit in the future if PyQt\PySide\Qt ever die (not that it is very likely, but I've had a bad experience with PyGTK so now I'm wary)
EDIT: Please also look at this, as it has people far better answers than I've given: Threading in a PyQt application: Use Qt threads or Python threads?
I've written a script that uses two thread pools of ten threads each to pull in data from an API. The thread pool implements this code on ActiveState. Each thread pool is monitoring a Redis database via PubSub for new entries. When a new entry is published, python passes the data to a function that uses python's Subprocess.POpen to execute a PHP shell to do the actual work of calling the API.
This system of launching PHP shells is necessary for functionality with my PHP web app, so launching PHP shells with Python can't be avoided.
This script will only be running on Linux servers.
How do I control the niceness (scheduling priority) of the application's threads?
Edit:
It seems controlling scheduling priority for individual threads in Python isn't possible. Is there a python solution, or at the very least a UNIX command I can run along with my script, to control the priority?
Edit 2:
Well I didn't end up finding a python way to handle it. I'm just running my script with nice now like this:
nice -n 19 python MyScript.py
I believe that threading priority is not controllable in python due to how they are implemented using a global interpreter lock (GIL). Having said that, even if you could give one thread more CPU processing priority, the python implementation that hands around the GIL would not be aware of this as it handed around the GIL. If you were able to increase niceness in a single thread in your pool (say it is doing a more important job) you would need to use your own implementation of locks to give the higher priority thread access to the GIL more often.
A google search returns this article which I believe is similar to what you are asking
Explains why it doesnt work
http://www.velocityreviews.com/forums/t329441-threading-priority.html
Explains the workaround I was suggesting
http://bytes.com/topic/python/answers/645966-setting-thread-priorities
The python threading-docs mention explicitly that there is no support for setting thread-priorities:
The design of this module is loosely based on Java’s threading model. However, where Java makes locks and condition variables basic behavior of every object, they are separate objects in Python. Python’s Thread class supports a subset of the behavior of Java’s Thread class; currently, there are no priorities, no thread groups, and threads cannot be destroyed, stopped, suspended, resumed, or interrupted. The static methods of Java’s Thread class, when implemented, are mapped to module-level functions.
It doesn't work, but I tried:
getting the parent pid and priority
launching threads using concurrent.futures.ThreadPoolExecutor
using ctypes to get the (linux) thread id from within the thread(works)
using the tid with os.setpriority(os.PRIO_PROCESS,tid,parent_priority+1)
calling pool.shutdown() from the parent.
Even with liberal sprinkling of os.sched_yield(), the child threads never actually run past the setpriority().
Reading man pages, it seems threads don't have the capability to change (even their) scheduling priority; you have to do something with "capabilities" to give the thread the "CAP_SYS_NICE" capability. Running the process with root permissions didn't help either; child threads still don't run.
I know, a lot of time has passed, but I recently came across this question, and I thought it would be useful to add another option.
Have a look at threading2, which is a drop-in replacement and extension for the default threading module, with support – sort of – for priority and affinity.
I was wondering if this answer at another related question might be useful in this scenario? (link)
As you are already using Subprocess.POpen to launch your PHP script, it strikes me that you can use "preexec_fn" and either a predefined function, or a lambda function (as demonstrated in the above linked answer) to set the nice level of each launched PHP thread?
I'm using PySide to manage some hardware and perform some relatively simple operations depending on (e.g.) button clicks in the interface. The code for running each of these pieces of hardware resides in another thread. For convenience, to all of those hardware drivers I've added a generic invoke_method signal, such that a UI component can use
my_driver.invoke_method.emit('method_name', [arg, ...], {kwarg, ...})
Obviously this accesses the signal attribute in that other thread directly.... but I'm not sure if this is necessarily okay in a GIL world.
If this is indeed too lazy a solution - are there any other great alternatives for invoking arbitrary methods in arbitrary threads without having to have an operation-specific signal in the UI connected to another signal in the driver?
I could imagine instead using a signal in each bit of UI code that accessed a different piece of hardware - something like do_invocation_driver_1 and do_invocation_driver_2 and connect those to the invoke_method signal of the corresponding driver.
I'd recommend reading this post for a general approach to interface threads with a PyQt GUI. The post discusses a thread that does socket I/O, but this really is applicable to any thread. Specifically, hardware-interface threads usually also use I/O, so this may be a good fit.
The approach discussed is very generic, using Queue.Queue, and may seem like an overkill for simple tasks (but I just want to call "that" function in a thread). However, once your application grows non-trivial, you will appreciate it, because it avoids any thread synchronization problems whatsoever, and is very scalable. I've personally used it to implement complex PyQt GUIs with side-threads doing all kinds of stuff.
So I just finished watching this talk on the Python Global Interpreter Lock (GIL) http://blip.tv/file/2232410.
The gist of it is that the GIL is a pretty good design for single core systems (Python essentially leaves the thread handling/scheduling up to the operating system). But that this can seriously backfire on multi-core systems and you end up with IO intensive threads being heavily blocked by CPU intensive threads, the expense of context switching, the ctrl-C problem[*] and so on.
So since the GIL limits us to basically executing a Python program on one CPU my thought is why not accept this and simply use taskset on Linux to set the affinity of the program to a certain core/cpu on the system (especially in a situation with multiple Python apps running on a multi-core system)?
So ultimately my question is this: has anyone tried using taskset on Linux with Python applications (especially when running multiple applications on a Linux system so that multiple cores can be used with one or two Python applications bound to a specific core) and if so what were the results? is it worth doing? Does it make things worse for certain workloads? I plan to do this and test it out (basically see if the program takes more or less time to run) but would love to hear from others as to your experiences.
Addition: David Beazley (the guy giving the talk in the linked video) pointed out that some C/C++ extensions manually release the GIL lock and if these extensions are optimized for multi-core (i.e. scientific or numeric data analysis/etc.) then rather than getting the benefits of multi-core for number crunching the extension would be effectively crippled in that it is limited to a single core (thus potentially slowing your program down significantly). On the other hand if you aren't using extensions such as this
The reason I am not using the multiprocessing module is that (in this case) part of the program is heavily network I/O bound (HTTP requests) so having a pool of worker threads is a GREAT way to squeeze performance out of a box since a thread fires off an HTTP request and then since it's waiting on I/O gives up the GIL and another thread can do it's thing, so that part of the program can easily run 100+ threads without hurting the CPU much and let me actually use the network bandwidth that is available. As for stackless Python/etc I'm not overly interested in rewriting the program or replacing my Python stack (availability would also be a concern).
[*] Only the main thread can receive signals so if you send a ctrl-C the Python interpreter basically tries to get the main thread to run so it can handle the signal, but since it doesn't directly control which thread is run (this is left to the operating system) it basically tells the OS to keep switching threads until it eventually hits the main thread (which if you are unlucky may take a while).
Another solution is:
http://docs.python.org/library/multiprocessing.html
Note 1: This is not a limitation of the Python language, but of CPython implementation.
Note 2: With regard to affinity, your OS shouldn't have a problem doing that itself.
I have never heard of anyone using taskset for a performance gain with Python. Doesn't mean it can't happen in your case, but definitely publish your results so others can critique your benchmarking methods and provide validation.
Personally though, I would decouple your I/O threads from the CPU bound threads using a message queue. That way your front end is now completely network I/O bound (some with HTTP interface, some with message queue interface) and ideal for your threading situation. Then the CPU intense processes can either use multiprocessing or just be individual processes waiting for work to arrive on the message queue.
In the longer term you might also want to consider replacing your threaded I/O front-end with Twisted or some thing like eventlets because, even if they won't help performance they should improve scalability. Your back-end is now already scalable because you can run your message queue over any number of machines+cpus as needed.
An interesting solution is the experiment reported by Ryan Kelly on his blog: http://www.rfk.id.au/blog/entry/a-gil-adventure-threading2/
The results seems very satisfactory.
I've found the following rule of thumb sufficient over the years: If the workers are dependent on some shared state, I use one multiprocessing process per core (CPU bound), and per core a fix pool of worker threads (I/O bound). The OS will take care of assigining the different Python processes to the cores.
The Python GIL is per Python interpreter. That means the only to avoid problems with it while doing multiprocessing is simply starting multiple interpreters (i.e. using seperate processes instead of threads for concurrency) and then using some other IPC primitive for communication between the processes (such as sockets). That being said, the GIL is not a problem when using threads with blocking I/O calls.
The main problem of the GIL as mentioned earlier is that you can't execute 2 different python code threads at the same time. A thread blocking on a blocking I/O call is blocked and hence not executin python code. This means it is not blocking the GIL. If you have two CPU intensive tasks in seperate python threads, that's where the GIL kills multi-processing in Python (only the CPython implementation, as pointed out earlier). Because the GIL stops CPU #1 from executing a python thread while CPU #0 is busy executing the other python thread.
Until such time as the GIL is removed from Python, co-routines may be used in place of threads. I have it on good authority that this strategy has been implemented by two successful start-ups, using greenlets in at least one case.
This is a pretty old question but since everytime I search about information related to python and performance on multi-core systems this post is always on the result list, I would not let this past before me an do not share my thoughts.
You can use the multiprocessing module that rather than create threads for each task, it creates another process of cpython compier interpreting your code.
It would make your application to take advantage of multicore systems.
The only problem that I see on this approach is that you will have a considerable overhead by creating an entire new process stack on memory. (http://en.wikipedia.org/wiki/Thread_(computing)#How_threads_differ_from_processes)
Python Multiprocessing module:
http://docs.python.org/dev/library/multiprocessing.html
"The reason I am not using the multiprocessing module is that (in this case) part of the program is heavily network I/O bound (HTTP requests) so having a pool of worker threads is a GREAT way to squeeze performance out of a box..."
About this, I guess that you can have also a pool of process too: http://docs.python.org/dev/library/multiprocessing.html#using-a-pool-of-workers
Att,
Leo
General tutorial or good resource on how to use threads in Python?
When to use threads, how they are effective, and some general background on threads [specific to Python]?
Threads should be used when you want two things to run at once, or want something to run in the background without slowing down the main process.
My recommendation is to only use threads if you have to. They generally add complexity to a program.
The main documentation for threading is here: http://docs.python.org/library/threading.html
Some examples are here:
http://www.devshed.com/c/a/Python/Basic-Threading-in-Python/1/
http://linuxgazette.net/107/pai.html
http://www.wellho.net/solutions/python-python-threads-a-first-example.html
One thing to remember before spending time and effort in writing a multi-threaded Python application is that there is a Global Interpreter Lock (GIL), so you won't actually be running more than one thread at a time.
This makes threading unsuitable for trying to take advantage of multiple cores or CPUs. You may get some speedup from multiplexing other resources (network, disk, ...), but it's never been particularly noticeable in my experience.
In general, I only use threads when there are several logically separate tasks happening at once, and yet I want them all in the same VM. A thread pulling data from the web and putting it on a Queue, while another thread pops from the Queue and writes to a database, something like that.
With Python 2.6, there is the new multiprocessing module which is pretty cool - it's got a very similar interface to the threading module, but actually spawns new OS processes, sidestepping the GIL.
There is a fantastic pdf, Tutorial on Threads Programming with Python by
Norman Matloff and Francis Hsu, of University of California, Davis.
Threads should be avoided whenever possible. They add much in complexity, synchronization issues and hard to debug issues. However some problems require them (i.e. GUI programming), but I would encourage you to look for a single-threaded solution if you can.
There are several tutorials here.