I am really having a hard time wrapping my head around multithreading in python. My expectation of the following code is that appLoop() will run for 10 seconds and the cease to exist -- which it does when tracing through in PyCharm, but not, when I just run it. This results in an infinite loop.
import time
import multiprocessing
isRunning = True
runningSince = 0
def appLoop():
try:
global isRunning
while isRunning:
time.sleep(1)
global runningSince
runningSince = runningSince + 1
print(f'Looping since {runningSince} seconds.')
except KeyboardInterrupt:
print(f'appLoop stopped after {runningSince} seconds.')
class Process:
class __Process:
def __init__(self):
self.process = multiprocessing.Process(target=appLoop)
self.process.start()
instance = None
def __init__(self):
if not Process.instance:
Process.instance = Process.__Process()
def __del__(self):
print('Instance deleted.')
p = Process()
time.sleep(10)
isRunning = False
print(f'isRunning set to False.')
del p
This brings up (at least...) 2 questions for me:
why is process still running after del p -- am I creating a zombie process here?
why does my appLoop() keep running even after I set isRunning to false when I run the app (according to my observations this works when tracing through the code as mentioned above)?
My use case in the end is to be able to start / stop my appLoop() from a flask web interface -- which is why I am trying to implement a singleton here. Just in case you might wonder...
And: I do know that __del__ is not recommended as you never know when exactly garbage collection will call it -- in this case I just use it for (cave man) debugging.
isRunning = False changes the value of the variable in the parent process. The child process (the one that executes the while loop) has its own copy of isRunning that is not affected by the assignment.
For the same reason del p does not terminate the process: because it has its own copy of p. You should terminate the process explicitly in the destructor:
def __del__(self):
print('Instance deleted.')
Process.instance.process.terminate()
Related
I am trying to spawn a python thread which perform a particular operation repeatedly based on a certain condition. If the condition doesn't met then the thread should exit. I have written the following code, but its running indefinitely.
class dummy(object):
def __init__(self):
# if the flag is set to False,the thread should exit
self.flag = True
def print_hello(self):
while self.flag:
print "Hello!! current Flag value: %s" % self.flag
time.sleep(0.5)
def execute(self):
t = threading.Thread(target=self.print_hello())
t.daemon = True # set daemon to True, to run thread in background
t.start()
if __name__ == "__main__":
obj = dummy()
obj.execute()
#Some other functions calls
#time.sleep(2)
print "Executed" # This line is never executed
obj.flag = False
I am new to python threading module. I have gone through some articles suggesting the use of threading.Timer() function, but this is not what I need.
The problem line is t = threading.Thread(target=self.print_hello()), more specifically target=self.print_hello(). This sets target to the results of self.print_hello(), and since this function never ends it will never be set. What you need to do is set it to the function itself with t = threading.Thread(target=self.print_hello).
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._finished = False
self._end = False
self._running = False
def run(self):
self._running = True
while not self._finished:
time.sleep(0.05)
self._end = True
def stop(self):
if not self._running:
return
self._finished = True
while not self._end:
time.sleep(0.05)
I wish to have a thread on which I can call run() and stop(). The stop method should wait for run to complete in an orderly manner. I also want stop to return without any issues if run hasn't even be called. How should I do this?
I create this thread in a setup() method in my test environment and run stop on it in the teardown(). However, in some tests I dont call run().
UPDATE
Here's my second attempt. Is it correct now?
import threading
import time
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._finished = False
def run(self):
while not self._finished:
print("*")
time.sleep(1)
print("Finished Other")
def finish(self):
self._finished = True
self.join()
m = MyThread()
m.start()
print("After")
time.sleep(5)
m.finish()
print("Finished Main")
You do not need to and should not implement this yourself. What you are looking for already exists, at least in large parts. It is, however, not called "stop". The concept you are describing is usually called "join".
Have a look at the documentation for join: https://docs.python.org/3.4/library/threading.html#threading.Thread.join
You write
The stop method should wait for run to complete in an orderly manner.
Join's documentation says: "Wait until the thread terminates." check ✓
You write
I also want stop to return without any issues if run hasn't even be
called
Join's documentation says: "It is also an error to join() a thread before it has been started"
So, the only thing you need to make sure is that you call join() only after you have started the thread via the start() method. That should be easy for you.
I have a function foo that only stops once a condition has been met. While foo is running, I need to ask for user input the whole time (keeps asking for user input). I want them to run separately without interfering with each other.
In my example below, foo keeps printing 'Hello' and getUserInput keeps looking for user input. I want foo to keep printing hello even if i do not enter anything for user input. It will keep asking for input as long as the user does not enter letter 'e'. I have my attempt below:
import threading
from time import sleep
class test:
def __init__(self):
self.running = True
def foo(self):
while(self.running):
print 'Hello\n'
sleep(2)
def getUserInput(self):
x = ''
while(x != 'e'):
x = raw_input('Enter value: ')
self.running = False
def go(self):
th1 = threading.Thread(target=self.foo)
th2 = threading.Thread(target=self.getUserInput)
th1.start()
th2.start()
t = test()
t.go()
My code prints out the first hello and asks for input but nothing after that. What am I doing wrong? Thanks for your help in advance.
Update: The opener was running his code on Windows in IDLE. Regarding I/O it behaves differently than a shell or the Windows command line. His code works on the Windows command line.
In principle, your code works for me. I am running Python 2.6.5.
Several comments here:
1) In your case it would be fine to only have two threads: the main thread and another one. However, it will also work with three. It's just that your main thread does nothing else than waiting for the other threads to finish.
2) You should to explicitly join() all threads you spawn. You do this in the main thread before terminating it. Keep record of the threads you spawn (e.g. in a list threads) and then join them at the end of your program (e.g. for t in threads: t.join()).
3) You share the variable self.running between threads. It is fine in this case, as one thread only reads it and another one only writes it. In general, you need to be very careful with shared variables and acquire a lock before changing it.
4) You should catch the KeyboardInterrupt exception in the main thread and find a way to communicate to your other threads to terminate :)
5) Use lowercase method names, so instead of getUserInput call it get_user_input. Use uppercase class names and inherit from object: class Test(object):
This is a running example:
import threading
from time import sleep
def main():
t = Test()
t.go()
try:
join_threads(t.threads)
except KeyboardInterrupt:
print "\nKeyboardInterrupt catched."
print "Terminate main thread."
print "If only daemonic threads are left, terminate whole program."
class Test(object):
def __init__(self):
self.running = True
self.threads = []
def foo(self):
while(self.running):
print '\nHello\n'
sleep(2)
def get_user_input(self):
while True:
x = raw_input("Enter 'e' for exit: ")
if x.lower() == 'e':
self.running = False
break
def go(self):
t1 = threading.Thread(target=self.foo)
t2 = threading.Thread(target=self.get_user_input)
# Make threads daemonic, i.e. terminate them when main thread
# terminates. From: http://stackoverflow.com/a/3788243/145400
t1.daemon = True
t2.daemon = True
t1.start()
t2.start()
self.threads.append(t1)
self.threads.append(t2)
def join_threads(threads):
"""
Join threads in interruptable fashion.
From http://stackoverflow.com/a/9790882/145400
"""
for t in threads:
while t.isAlive():
t.join(5)
if __name__ == "__main__":
main()
When typing e or E, the program ends after a short delay (as intended by you). When pressing ctrl+c, it immediately terminates. Making a program that uses threading responsive to exceptions is a bit trickier than expected. I have included important references in the source above.
This is how it looks like during runtime:
$ python supertest.py
Hello
Enter 'e' for exit:
Hello
Hello
Hello
e
$
I want to know how can I stop my program in console with CTRL+C or smth similar.
The problem is that there are two threads in my program. Thread one crawls the web and extracts some data and thread two displays this data in a readable format for the user. Both parts share same database. I run them like this :
from threading import Thread
import ResultsPresenter
def runSpider():
Thread(target=initSpider).start()
Thread(target=ResultsPresenter.runPresenter).start()
if __name__ == "__main__":
runSpider()
how can I do that?
Ok so I created my own thread class :
import threading
class MyThread(threading.Thread):
"""Thread class with a stop() method. The thread itself has to check
regularly for the stopped() condition."""
def __init__(self):
super(MyThread, self).__init__()
self._stop = threading.Event()
def stop(self):
self._stop.set()
def stopped(self):
return self._stop.isSet()
OK so I will post here snippets of resultPresenter and crawler.
Here is the code of resultPresenter :
# configuration
DEBUG = False
DATABASE = database.__path__[0] + '/database.db'
app = Flask(__name__)
app.config.from_object(__name__)
app.config.from_envvar('CRAWLER_SETTINGS', silent=True)
def runPresenter():
url = "http://127.0.0.1:5000"
webbrowser.open_new(url)
app.run()
There are also two more methods here that I omitted - one of them connects to the database and the second method loads html template to display result. I repeat this until conditions are met or user stops the program ( what I am trying to implement ). There are also two other methods too - one get's initial link from the command line and the second valitated arguments - if arguments are invalid I won't run crawl() method.
Here is short version of crawler :
def crawl(initialLink, maxDepth):
#here I am setting initial values, lists etc
while not(depth >= maxDepth or len(pagesToCrawl) <= 0):
#this is the main loop that stops when certain depth is
#reached or there is nothing to crawl
#Here I am popping urls from url queue, parse them and
#insert interesting data into the database
parser.close()
sock.close()
dataManager.closeConnection()
Here is the init file which starts those modules in threads:
import ResultsPresenter, MyThread, time, threading
def runSpider():
MyThread.MyThread(target=initSpider).start()
MyThread.MyThread(target=ResultsPresenter.runPresenter).start()
def initSpider():
import Crawler
import database.__init__
import schemas.__init__
import static.__init__
import templates.__init__
link, maxDepth = Crawler.getInitialLink()
if link:
Crawler.crawl(link, maxDepth)
killall = False
if __name__ == "__main__":
global killall
runSpider()
while True:
try:
time.sleep(1)
except:
for thread in threading.enumerate():
thread.stop()
killall = True
raise
Killing threads is not a good idea, since (as you already said) they may be performing some crucial operations on database. Thus you may define global flag, which will signal threads that they should finish what they are doing and quit.
killall = False
import time
if __name__ == "__main__":
global killall
runSpider()
while True:
try:
time.sleep(1)
except:
/* send a signal to threads, for example: */
killall = True
raise
and in each thread you check in a similar loop whether killall variable is set to True. If it is close all activity and quit the thread.
EDIT
First of all: the Exception is rather obvious. You are passing target argument to __init__, but you didn't declare it in __init__. Do it like this:
class MyThread(threading.Thread):
def __init__(self, *args, **kwargs):
super(MyThread, self).__init__(*args, **kwargs)
self._stop = threading.Event()
And secondly: you are not using my code. As I said: set the flag and check it in thread. When I say "thread" I actually mean the handler, i.e. ResultsPresenter.runPresenter or initSpide. Show us the code of one of these and I'll try to show you how to handle stopping.
EDIT 2
Assuming that the code of crawl function is in the same file (if it is not, then you have to import killall variable), you can do something like this
def crawl(initialLink, maxDepth):
global killall
# Initialization.
while not killall and not(depth >= maxDepth or len(pagesToCrawl) <= 0):
# note the killall variable in while loop!
# the other code
parser.close()
sock.close()
dataManager.closeConnection()
So basically you just say: "Hey, thread, quit the loop now!". Optionally you can literally break a loop:
while not(depth >= maxDepth or len(pagesToCrawl) <= 0):
# some code
if killall:
break
Of course it will still take some time before it quits (has to finish the loop and close parser, socket, etc.), but it should quit safely. That's the idea at least.
Try this:
ps aux | grep python
copy the id of the process you want to kill and:
kill -3 <process_id>
And in your code (adapted from here):
import signal
import sys
def signal_handler(signal, frame):
print 'You killed me!'
sys.exit(0)
signal.signal(signal.SIGQUIT, signal_handler)
print 'Kill me now'
signal.pause()
I have a threaded python application with a long-running mainloop in the background thread. This background mainloop is actually a call to pyglet.app.run(), which drives a GUI window and also can be configured to call other code periodically. I need a do_stuff(duration) function to be called at will from the main thread to trigger an animation in the GUI, wait for the animation to stop, and then return. The actual animation must be done in the background thread because the GUI library can't handle being driven by separate threads.
I believe I need to do something like this:
import threading
class StuffDoer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.max_n_times = 0
self.total_n_times = 0
self.paused_ev = threading.Event()
def run(self):
# this part is outside of my control
while True:
self._do_stuff()
# do other stuff
def _do_stuff(self):
# this part is under my control
if self.paused_ev.is_set():
if self.max_n_times > self.total_n_times:
self.paused_ev.clear()
else:
if self.total_n_times >= self.max_n_times:
self.paused_ev.set()
if not self.paused_ev.is_set():
# do stuff that must execute in the background thread
self.total_n_times += 1
sd = StuffDoer()
sd.start()
def do_stuff(n_times):
sd.max_n_times += n_times
sd.paused_ev.wait_for_clear() # wait_for_clear() does not exist
sd.paused_ev.wait()
assert (sd.total_n_times == sd.max_n_times)
EDIT: use max_n_times instead of stop_time to clarify why Thread.join(duration) won't do the trick.
From the documentation for threading.Event:
wait([timeout])
Block until the internal flag is true.
If the internal flag is true on entry,
return immediately. Otherwise, block
until another thread calls set() to
set the flag to true, or until the
optional timeout occurs.
I've found I can get the behavior I'm looking for if I have a pair of events, paused_ev and not_paused_ev, and use not_paused_ev.wait(). I could almost just use Thread.join(duration), except it needs to only return precisely when the background thread actually registers that the time is up. Is there some other synchronization object or other strategy I should be using instead?
I'd also be open to arguments that I'm approaching this whole thing the wrong way, provided they're good arguments.
Hoping I get some revision or additional info from my comment, but I'm kind of wondering if you're not overworking things by subclassing Thread. You can do things like this:
class MyWorker(object):
def __init__(self):
t = Thread(target = self._do_work, name "Worker Owned Thread")
t.daemon = True
t.start()
def _do_work(self):
While True:
# Something going on here, forever if necessary. This thread
# will go away if the other non-daemon threads terminate, possibly
# raising an exception depending this function's body.
I find this makes more sense when the method you want to run is something that is more appropriately a member function of some other class than it would be to as the run method on the thread. Additionally, this saves you from having to encapsulate a bunch of business logic inside of a Thread. All IMO, of course.
It appears that your GUI animation thread is using a spin-lock in its while True loop. This can be prevented using thread-safe queues. Based on my reading of your question, this approach would be functionally equivalent and efficient.
I'm omitting some details of your code above which would not change. I'm also assuming here that the run() method which you do not control uses the self.stop_time value to do its work; otherwise there is no need for a threadsafe queue.
from Queue import Queue
from threading import Event
class StuffDoer:
def __init__(self, inq, ready):
self.inq = inq
self.ready = ready
def _do_stuff(self):
self.ready.set()
self.stop_time = self.inq.get()
GUIqueue = Queue()
control = Event()
sd = StuffDoer(GUIqueue, control)
def do_stuff(duration):
control.clear()
GUIqueue.put(time.time() + duration)
control.wait()
I ended up using a Queue similar to what #wberry suggested, and making use of Queue.task_done and Queue.wait:
import Queue
import threading
class StuffDoer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.setDaemon(True)
self.max_n_times = 0
self.total_n_times = 0
self.do_queue = Queue.Queue()
def run(self):
# this part is outside of my control
while True:
self._do_stuff()
# do other stuff
def _do_stuff(self):
# this part is under my control
if self.total_n_times >= self.max_n_times:
try:
self.max_n_times += self.do_queue.get(block=False)
except Queue.Empty, e:
pass
if self.max_n_times > self.total_n_times:
# do stuff that must execute in the background thread
self.total_n_times += 1
if self.total_n_times >= self.max_n_times:
self.do_queue.task_done()
sd = StuffDoer()
sd.start()
def do_stuff(n_times):
sd.do_queue.put(n_times)
sd.do_queue.join()
assert (sd.total_n_times == sd.max_n_times)
I made solution based on #g.d.d.c advice for this question. There is my code:
threads = []
# initializing aux thread(s) in the main thread ...
t = threading.Thread(target=ThreadF, args=(...))
#t.setDaemon(True) # I'm not sure does it really needed
t.start()
threads.append(t.ident)
# Block main thread
while filter(lambda thread: thread.ident in threads, threading.enumerate()):
time.sleep(10)
Also, you can use Thread.join to block the main thread - it is better way.