Python have been really bumpy for me, because the last time I created a GUI client, the client seems to hang when spawning a process, calling a shell script, and calling outside application.
This have been my major problem with Python since then, and now I'm in a new project, can someone give me pointers, and a word of advice in order for my GUI python application to still be interactive when spawning another process?
Simplest (not necessarily "best" in an abstract sense): spawn the subprocess in a separate thread, communicating results back to the main thread via a Queue.Queue instance -- the main thread must periodically check that queue to see if the results have arrived yet, but periodic polling isn't hard to arrange in any event loop.
Your main GUI thread will freeze if you spawn off a process and wait for it to completely. Often, you can simply use subprocess and poll it now and then for completion rather than waiting for it to finish. This will keep your GUI from freezing.
Related
I know that I can run a background process in python using subprocess. But the problem is that when I make a gui and then use subprocess with close_fds=True parameter, the window changes to not responding.
So, what I want is that I need to create a background process but it should run separately along with the main process and when that process is done, it should again combine with the main process.
BTW, I am using PySide2 as the gui framework
Any help would be appreciated
I think what would be more beneficial to you would be threading, you are able to start a process in another thread without blocking the main thread which runs your gui. Once the other thread has completed its task it will join the main thread
Kind all, I'm really new to python and I'm facing a task which I can't completely grasp.
I've created an interface with Tkinter which should accomplish a couple of apparently easy feats.
By clicking a "Start" button two threads/processes will be started (each calling multiple subfunctions) which mainly read data from a serial port (one port per process, of course) and write them to file.
The I/O actions are looped within a while loop with a very high counter to allow them to go onward almost indefinitely.
The "Stop" button should stop the acquisition and essentially it should:
Kill the read/write Thread
Close the file
Close the serial port
Unfortunately I still do not understand how to accomplish point 1, i.e.: how to create killable threads without killing the whole GUI. Is there any way of doing this?
Thank you all!
First, you have to choose whether you are going to use threads or processes.
I will not go too much into differences, google it ;) Anyway, here are some things to consider: it is much easier to establish communication between threads than betweeween processes; in Python, all threads will run on the same CPU core (see Python GIL), but subprocesses may use multiple cores.
Processes
If you are using subprocesses, there are two ways: subprocess.Popen and multiprocessing.Process. With Popen you can run anything, whereas Process gives a simpler thread-like interface to running python code which is part of your project in a subprocess.
Both can be killed using terminate method.
See documentation for multiprocessing and subprocess
Of course, if you want a more graceful exit, you will want to send an "exit" message to the subprocess, rather than just terminate it, so that it gets a chance to do the clean-up. You could do that e.g. by writing to its stdin. The process should read from stdin and when it gets message "exit", it should do whatever you need before exiting.
Threads
For threads, you have to implement your own mechanism for stopping, rather than using something as violent as process.terminate().
Usually, a thread runs in a loop and in that loop you check for a flag which says stop. Then you break from the loop.
I usually have something like this:
class MyThread(Thread):
def __init__(self):
super(Thread, self).__init__()
self._stop_event = threading.Event()
def run(self):
while not self._stop_event.is_set():
# do something
self._stop_event.wait(SLEEP_TIME)
# clean-up before exit
def stop(self, timeout):
self._stop_event.set()
self.join(timeout)
Of course, you need some exception handling etc, but this is the basic idea.
EDIT: Answers to questions in comment
thread.start_new_thread(your_function) starts a new thread, that is correct. On the other hand, module threading gives you a higher-level API which is much nicer.
With threading module, you can do the same with:
t = threading.Thread(target=your_function)
t.start()
or you can make your own class which inherits from Thread and put your functionality in the run method, as in the example above. Then, when user clicks the start button, you do:
t = MyThread()
t.start()
You should store the t variable somewhere. Exactly where depends on how you designed the rest of your application. I would probably have some object which hold all active threads in a list.
When user clicks stop, you should:
t.stop(some_reasonable_time_in_which_the_thread_should_stop)
After that, you can remove the t from your list, it is not usable any more.
First you can use subprocess.Popen() to spawn child processes, then later you can use Popen.terminate() to terminate them.
Note that you could also do everything in a single Python thread, without subprocesses, if you want to. It's perfectly possible to "multiplex" reading from multiple ports in a single event loop.
I wanted to run a function in parallel to the main program in Python. Suppose I have a speech recognition function. I want it to run in the background and interrupt the main program if a particular utterance is heard. But at the same time, I have other tasks to perform. So, the speech recognition should work as a separate process and may be call a function when a command is heard.
I tried the python multiprocessing module, the thread module and the threading module. But all of these required me to wait until the process or thread is finished. What I want is something that will allow me to run functions in the background. They must call some callback function if a specific event occurs.
I hope I will find an effective way of doing this.
I tried the threading module. The code looked like this (pseudocode):
def detected(text):
commands = 'a list of commands'
if text in commands:
#call some function according to the command
def speech_recognition():
#while True:
#If speech detected:
#record it
#process it and covert it to text
#if text is a specified command:
#call the detected(text) function with the recognized text as argument
import threading as t
pr = t.Thread(target=speech_recognition)
pr.start()
#from here starts the main program that does some other functions that
#doesn't need to be mentioned here.
But this doesn't work. The speech recognition runs for a few seconds and then just quits. No Exceptions raised, no system exits, nothing. Its the same when I try the multiprocessing and thread modules.
I don't know how CPU-intense speech recognition is, but I am pretty sure the problem you are describing is best solved with maximum decoupling between entities, i.e. separation in processes. Simple scenario: one of your processes runs your "main program", the other process is entirely responsible for speech recognition. You then implement a communication protocol between those processes. The "main program" still needs some kind of event system and asynchronous execution based on threading, because it needs to be able to listen and immediately react to events sent by the speech-reco-process. Hence, a working model would contain:
one main process which should not be CPU-bound
one child process, spawned via multiprocessing, handling speech recognition
a communication protocol enabling the transmission of data/events between main and child process
one additional thread in the main process that waits for events sent by the child process and makes sure that the main program reacts correspondingly
Main and child process run concurrently as scheduled by the operating system. Of course this works best on a system with at least two CPU cores. In the main process, the main thread and the other thread to not really run synchronously -- only one thread can run at a time due to CPython's global interpreter lock (GIL).
The communication between main process and child process can be implemented with basic multiprocessing tools such as Queue or Pipe.
As you realize, you need to invest some serious thought into a problem like this. Don't try to solve this quick&dirty or just by trial and error. You need to make sure you understand your self-developed architecture.
Just use threading and pass your thread either a function handle to call when it has data ready or the application handle in the case of GUI applications so that the thread can create an event with the data attached.
I'm using Python with wxPython for writing an app.
The method I'm considering to accomplish this may not be the best - if that's the case, let me know because I'm open to refactoring.
Right now, I have one GUI form. The main program start point instantiates an instance of the GUI form then runs wx.mainLoop(), which causes the app's main initial thread to block for the lifetime of the app.
We of course know that when events happen in the UI, the UI thread runs the code for them.
Now, I have another thread - a worker thread. This thread needs to sit idle, and then when something happens in the UI thread, e.g. a button is clicked, I want the worker thread to stop idling and do something else - run a function, say.
I can't envision this right now but I could see as the app gets more complex also having to signal the worker thread while it's actually busy doing something.
I have two questions about this setup:
How can I make my worker thread idle without using up CPU time? Doing something like while True: pass will suck CPU time, while something like while True: time.sleep(0.1) will not allow instantaneous reaction to events.
What's the best way to signal into the worker thread to do something? I don't want the UI thread to execute something, I want the worker thread to be signaled, by the UI thread, that it should change what it's doing. Ideally, I'd have some way for the worker thread to register a callback with the UI itself, so that when a button is clicked or any other UI Event happens, the worker thread is signalled to change what it's doing.
So, is this the best way to accomplish this? And what's the best way to do it?
Thanks!
First: Do you actually need a background thread to sit around idle in the first place?
On most platforms, starting a new thread is cheap. (Except on Windows and Linux, where it's supercheap.) So, why not just kick off a thread whenever you need it? (It's just as easy to keep around a list of threads as a single thread, right?)
Alternatively, why not just create a ThreadPoolExecutor, and just submit jobs to it, and let the executor worry about when they get run and on which thread. Any time you can just think in terms of "tasks that need to get run without blocking the main thread" instead of "worker threads that need to wait on work", you're making your life easier. Under the covers, there's still one or more worker threads waiting on a queue, or something equivalent, but that part's all been written (and debugged and optimized) for you. All you have to write are the tasks, which are just regular functions.
But, if you want to write explicit background threads, you can, so I'll explain that.
How can I make my worker thread idle without using up CPU time? … What's the best way to signal into the worker thread to do something?
The way to idle a thread until a value is ready is to wait on a synchronization object. On any modern OS, waiting on a synchronization object means the operating system stops giving you any CPU time until the object is ready for you.*
There are a variety of different options you can see in the Threading module docs, but the obvious one to use in most cases like this is a Condition. The way to signal the worker thread is then to notify the Condition.
However, often a Queue is a lot simpler. To wait on a Queue, just call its get method with block=True. To signal another thread to wake up, just put something on the Queue. (Under the covers, a Queue wraps up a list or deque or other collection, a Lock, and a Condition, so you just tell it what you want to do—check for a value, block until there's a value, add a value—instead of dealing with waiting and signaling and protecting the collection.)
See the answer to controlling UI elements in wxPython using threading for how to signal in both directions, from a worker thread to a UI thread and vice-versa.
I'd have some way for the worker thread to register a callback with the UI itself, so that when a button is clicked or any other UI Event happens, the worker thread is signalled to change what it's doing.
You can do it this way if you want. Just pass self.queue.put or def callback(value): self.value = value; self.condition.notify() or whatever as a callback, and the GUI thread doesn't even have to know that the callback is triggering another thread.
In fact, that's a pretty nice design that may make you very happy later, when you decide to move some code back and forth between inline and background-threaded, or move it off to a child process instead of a background thread, or whatever.
I can't envision this right now but I could see as the app gets more complex also having to signal the worker thread while it's actually busy doing something.
But what do you want to happen if it's busy?
If you just want to say "If you're idle, wake up and do this task; otherwise, hold onto it and do it whenever you're ready", that's exactly what a Queue, or an Executor, will do for you automatically.
If you want to say, "If you're idle, wake up, otherwise, don't worry about it", that's what a Condition or Event will do.
If you want to say, "If you're idle, wake up and do this, otherwise, cancel what you're doing and do this instead", that's a bit more complicated. You pretty much need to have the background thread periodically check an "interrupt_me" variable while it's busy (and put a Lock around it), and then you'll set that flag as well as notifying the Condition… although in some cases, you can merge the idle and busy cases into a single Condition or Event (by calling an infinite wait() when idle, and a quick-check wait(timeout=0) when busy).
* In some cases—e.g., a Linux futex or a Windows CriticalSection—it may actually spin off a little bit of CPU time in some cases, because that happens to be a good optimization. But the point is, you're not asking for any CPU time until you're ready to use it.
I have a PyQt4 application, which at some point packs a big file using the tarfile module. Since the tarfile module does not implement any callback strategy, it blocks and the Qt GUI gets unresponsive.
I want the GUI to keep updating during that time. The only possibility is a separate thread.
So, I start a QThread. What do I have to do in the QThread to make the GUI update itself?
As soon, as the tar process is finished, I want the thread to finish.
Thanks!
Nathan
QThread's are pretty much identical to normal Python threads so you can just use normal communication methods. However, QThreads also have a few signals available, so if you simply connect to those, than you're done.
In your GUI code do something like this and you're pretty much done:
thread = Thread()
thread.finished.connect(gui.do_update_thingy)
There is also a terminated and started signal available which you can use :)