What is the best way to continuously repeat the execution of a given function at a fixed interval while being able to terminate the executor (thread or process) immediately?
Basically I know two approaches:
use multiprocessing and function with infinite cycle and time.sleep at the end. Processing is terminated with process.terminate() in any state.
use threading and constantly recreate timers at the end of the thread function. Processing is terminated by timer.cancel() while sleeping.
(both “in any state” and “while sleeping” are fine, even though the latter may be not immediate). The problem is that I have to use both multiprocessing and threading as the latter appears not to work on ARM (some fuzzy interaction of python interpreter and vim, outside of vim everything is fine) (I was using the second approach there, have not tried threading+cycle; no code is currently left) and the former spawns way too many processes which I would like not to see unless really required. This leads to a problem of having to code two different approaches while threading with cycle is just a few more imports for drop-in replacements of all multiprocessing stuff wrapped in if/else (except that there is no thread.terminate()). Is there some better way to do the job?
Currently used code is here (currently with cycle for both jobs), but I do not think it will be much useful to answer the question.
Update: The reason why I am using this solution are functions that display file status (and some other things like branch) in version control systems in vim statusline. These statuses must be updated, but updating them immediately cannot be done without using hooks and I have no idea how to set hooks temporary and remove on vim quit without possibly spoiling user configuration. Thus standard solution is cache expiring after N seconds. But when cache expired I need to do an expensive shell call and the delay appears to be noticeable, the more noticeable the heavier IO load is. What I am implementing now is updating values for viewed buffers each N seconds in a separate process thus delays are bothering that process and not me. Threads are likely to also work because GIL does not affect calls to external programs.
I'm not clear on why a single long-lived thread that loops infinitely over the tasks wouldn't work for you? Or why you end up with many processes in the multiprocess option?
My immediate reaction would have been a single thread with a queue to feed it things to do. But I may be misunderstanding the problem.
I do not know how do it simply and/or cleanly in Python, but I was wondering if maybe you couldn't take avantage of an existing system scheduler, e.g. crontab for *nix system.
There is an API in python and it might satisfied your needs.
Related
I have a program that fetches data via an API. I created a function that only takes the target data as an argument and with a for-loop I run this method 10 times.
The programm takes quite some time to display the data because the next function call only happens when the function before has done its work.
I want to use Threads to make it all happen quicker. However, I'm confused. On realpython.org I read this:
A thread is a separate flow of execution. This means that your program will have two things happening at once. But for most Python 3 implementations the different threads do not actually execute at the same time: they merely appear to. It’s tempting to think of threading as having two (or more) different processors running on your program, each one doing an independent task at the same time. That’s almost right. The threads may be running on different processors, but they will only be running one at a time.
First they say: "This means that your program will have two things happening at once" and then they say "but they will only be running one at a time". So my threads are not done simultaneously?
I want to make a decision on whether to use Threads or Multiprocessing but I can't figure it out.
Can somebody help?
With both Threads or Multiprocessing you must assume that execution of your program could jump from one thread/process to another randomly. The difference is that with Threads, code is never really executed at the same time. That means there is always only one CPU core doing your work. With Multiprocessing, your code runs on multiple cores at the same time. So only Multiprocessing will solve your computation N times faster with N processes. (There will be some overhead of course.) If you are not doing any heavy computation, but need to create the illusion of things running in parallel, use threads. This is especially useful for GUIs.
The confusing part is that IO (copying files or loading something from the web for example) is not CPU bound, as it does not require a lot of CPU instructions to happen. So always use threads for this. To understand it a bit more, you should realise that when a thread is waiting for an IO operation to finish, it is actually in a blocked state. This allows other threads to run. So if you use threads to fetch data the first thread will start loading it and then block. This makes room for the the second thread to do the same and so on. When one of the threads has the data ready, it will unblock, run the rest of its code and finish.
(Note that when multiple threads are running they can pause randomly and give room for other threads to run for a while and then carry on. (See first sentence of this answer.))
Generally always use threads unless you need to do something CPU heavy in parallel. Multiprocessing has a lot of limitations when it comes to how it works internally and using it is more complicated and heavy.
This only applies to some implementations of Python tough, for example the most commonly used "official" implementation, CPython. In other languages or less common Python implementations threads are often able to execute instructions on multiple cores at the same time.
I'm writing a program in which I want to evaluate a piece of code asynchronously. I want it to be isolated from the main thread so that it can raise an error, enter an infinite loop, or just about anything else without disrupting the main program. I was hoping to use threading.Thread, but this has a major problem; I can't figure out how to stop it. I have tried Thread._stop(), but that frequently doesn't work. I end up with a thread that I can't control hogging both interpreter time and CPU power. The code in the thread doesn't open any files or do anything else that would cause problems if I hard-killed it.
Python's multiprocessing.Process.terminate() does this really well; unfortunately, initiating a process on Windows takes nearly a second, which is long enough to cause annoying delays in my GUI.
Does anyone know either a: how to kill a Python thread (I don't think I care how dirty the exit is), or b: how to speed up starting a process?
A third possibility would be a third-party library that provides an alternative method for asynchronous execution, but I've never heard of any such thing.
In my case, the best way to do this seems to be to maintain a running worker process, and send the code to it on an as-needed basis. If the process acts up, I kill it and then start a new one immediately to avoid any delay the next time.
My wx GUI shows thumbnails, but they're slow to generate, so:
The program should remain usable while the thumbnails are generating.
Switching to a new folder should stop generating thumbnails for the old folder.
If possible, thumbnail generation should make use of multiple processors.
What is the best way to do this?
Putting the thumbnail generation in a background thread with threading.Thread will solve your first problem, making the program usable.
If you want a way to interrupt it, the usual way is to add a "stop" variable which the background thread checks every so often (e.g., once per thumbnail), and the GUI thread sets when it wants to stop it. Ideally you should protect this with a threading.Condition. (The condition isn't actually necessary in most cases—the same GIL that prevents your code from parallelizing well also protects you from certain kinds of race conditions. But you shouldn't rely on that.)
For the third problem, the first question is: Is thumbnail generation actually CPU-bound? If you're spending more time reading and writing images from disk, it probably isn't, so there's no point trying to parallelize it. But, let's assume that it is.
First, if you have N cores, you want a pool of N threads, or N-1 if the main thread has a lot of work to do too, or maybe something like 2N or 2N-1 to trade off a bit of best-case performance for a bit of worst-case performance.
However, if that CPU work is done in Python, or in a C extension that nevertheless holds the Python GIL, this won't help, because most of the time, only one of those threads will actually be running.
One solution to this is to switch from threads to processes, ideally using the standard multiprocessing module. It has built-in APIs to create a pool of processes, and to submit jobs to the pool with simple load-balancing.
The problem with using processes is that you no longer get automatic sharing of data, so that "stop flag" won't work. You need to explicitly create a flag in shared memory, or use a pipe or some other mechanism for communication instead. The multiprocessing docs explain the various ways to do this.
You can actually just kill the subprocesses. However, you may not want to do this. First, unless you've written your code carefully, it may leave your thumbnail cache in an inconsistent state that will confuse the rest of your code. Also, if you want this to be efficient on Windows, creating the subprocesses takes some time (not as in "30 minutes" or anything, but enough to affect the perceived responsiveness of your code if you recreate the pool every time a user clicks a new folder), so you probably want to create the pool before you need it, and keep it for the entire life of the program.
Other than that, all you have to get right is the job size. Hopefully creating one thumbnail isn't too big of a job—but if it's too small of a job, you can batch multiple thumbnails up into a single job—or, more simply, look at the multiprocessing API and change the way it batches jobs when load-balancing.
Meanwhile, if you go with a pool solution (whether threads or processes), if your jobs are small enough, you may not really need to cancel. Just drain the job queue—each worker will finish whichever job it's working on now, but then sleep until you feed in more jobs. Remember to also drain the queue (and then maybe join the pool) when it's time to quit.
One last thing to keep in mind is that if you successfully generate thumbnails as fast as your computer is capable of generating them, you may actually cause the whole computer—and therefore your GUI—to become sluggish and unresponsive. This usually comes up when your code is actually I/O bound and you're using most of the disk bandwidth, or when you use lots of memory and trigger swap thrash, but if your code really is CPU-bound, and you're having problems because you're using all the CPU, you may want to either use 1 fewer core, or look into setting thread/process priorities.
Is there a way to do long processing loops in Python without freezing the GUI with TideSDK?
or I'll just have to use threads...
Thanks.
There's nothing really specific to TideSDK here—this is a general issue with any program built around an event loop, which means nearly all GUI apps and network servers, among other things.
There are three standard solutions:
Break the long task up into a bunch of small tasks, each of which schedules the next to get run.
Make the task call back to the event loop every so often.
Run the task in parallel.
For the first solution, most event-based frameworks have a method like doLater(func) or setTimeout(func, 0). If not, they have to at least have a way of posting a message to the event loop's queue, and you can pretty easily build a doLater around that. This kind of API can be horrible to use in C-like languages, and a bit obnoxious in JS just because of the bizarre this/scoping rules, but in Python and most other dynamic languages it's nearly painless.
Since TideSDK is built around a browser JS engine, it's almost certainly going to provide this first solution.
The second solution really only makes sense for frameworks built around either cooperative threadlets or explicit coroutines. However, some traditional single-threaded frameworks like classic Mac (and, therefore, modern Win32 and a few cross-platform frameworks like wxWindows) use this for running background jobs.
The first problem is that you have to deal with re-entrancy carefully (at least wx has a SafeYield to help a little), or you can end up with many of the same kinds of problems as threads—or, worse, everything seems to work except that under heavy use you occasionally get a stack crash from infinite recursion. The other problem is that it only really works well when there's only one heavy background task at a time, because it doesn't work so well
If your framework has a way of doing this, it'll have a function like yieldToOtherTasks or processNextEvent, and all you have to do is make sure to call that every once in a while. (However, if there's also a doLater, you should consider that first.) If there is no such method, this solution is not appropriate to your framework.
The third solution is to spin off a task via threading.Thread or multiprocessing.Process.
The problem with this parallelism is that you have to come up with some way to signal safely, and to share data safely. Some event-loop frameworks have a thread-safe "doLater" or "postEvent" method, and if the only signal you need is "task finished" and the only data you need to share are the task startup params and return values, everything is easy. But once that's not sufficient, things can get very complicated.
Also, if you have hundreds of long-running tasks to run, you probably don't want a thread or process for each one. In fact, you probably want a fixed-size pool of threads or processes, and then you'll have to break your tasks into small-enough subtasks so they don't starve each other out, so in effect you're doing all the work of solution #1 anyway.
However, there are cases where threads or processes are the simplest solution.
I use python for video-game-like experiments in cognitive science. I'm testing out a device that detects eye movements via EOG, and this device talks to the computer via USB. To ensure that data is being continuously read from the USB while the experiment does other things (like changing the display, etc), I thought I'd use the multiprocessing module (with a multicore computer of course), put the USB reading work in a separate worker process, and use a queue to tell that worker when events of interest occur in the experiment. However, I've encountered some strange behaviour such that even when there is 1 second between the enqueuing of 2 different messages to the worker, when I look at the worker's output at the end, it seems to have received the second almost immediately after the first. Surely I've coded something awry, but I can't see what, so I'd very much appreciate help anyone can provide.
I've attempted to strip down my code to a minimal example demonstrating this behaviour. If you go to this gist:
https://gist.github.com/914070
you will find "multiprocessing_timetravel.py", which codes the example, and "analysis.R", which analyzes the "temp.txt" file that results from running "multiprocessing_timetravel.py". "analysis.R" is written in R and requires you have the plyr library installed, but I've also included example of the analysis output in the "analysis_results.txt" file at the gist.
Despite working with multiprocessing, your queue still uses synchronization objects (two locks and a semaphore) and the put method spawns another thread (based on the 2.7 source). So GIL contention (and other fun stuff) may come into play, as suggested by BlueRaja. You can try playing with sys.checkinterval and see if decreasing it also decreases the observed discrepancy, although you don't want to run normally in that condition.
Note that, if your USB reading code drops the GIL (e.g. ctypes code, or a Python extension module designed to drop the GIL), you do get true multithreading, and a threaded approach might be more productive than using multiprocessing.
Ah, I solved it and it turned out to be much simpler than I expected. There were 5 events per "trial" and the final event triggered a write of data to the HD. If this final write takes a long time, the worker may not grab the next trial's first event until the second event has already been put into the queue. When this happens, the first event lasts (to the worker's eyes) for only one of its loops before it encounters the second event. I'll have to either figure out a faster way to write out the data or leave the data in memory until a break in the experiment permits a long write.