Develop Reference

How to start, stop, and restart a thread?

Fairly new to Python; working on a Raspberry Pi 4 with Python 3.4.3.
Got a code working to listen for 2 distinct alarms in my lab - one for a -80 freezer getting too warm, and the other for a -20 freezer. Code listens on a microphone, streams data, Fourier-transforms it, detects the peaks I'm interested in, and triggers events when they're found - eventually going to email me and my team if an alarm goes off, but still just testing with Print commands atm. Let's call them Alarm A/EventA and Alarm B/Event B.
I want it to trigger Event A when Alarm A is detected, but then wait 1 hour before triggering Event A again (if Alarm A is still going off/goes off again in an hour).
Meanwhile, though, I also want it to continue listening for Alarm B and trigger Event B if detected - again, only once per hour.
Since I can't just do time.sleep, then, I'm trying to do it with Threads - but am having trouble starting, stopping, and restarting a thread for the 1 hour (currently just 10 second for testing purposes) delay.
I have variables CounterA and CounterB set to 0 to start. When Alarm A is detected I have the program execute EventA and up CounterA to 1; ditto for AlarmB/EventB/CounterB. EventA and EventB are only triggered if CounterA and CounterB are <1.
I'm having a real hard time resetting the counters after a time delay, though. Either I end up stalling the whole program after an event is triggered, or I get the error that threads can only be started once.
Here are the relevant sections of the code:
import time
import threading
CounterA = 0
CounterB = 0
def Aresetter():
time.sleep(10)
global CounterA
CounterA=CounterA-1
thA.join()
def Bresetter():
time.sleep(10)
global CounterB
CounterB=CounterB-1
thB.join()
thA = threading.Thread(target = Aresetter)
thB = threading.Thread(target = Bresetter)
if any(#Alarm A detection) and CounterA<1:
print('Alarm A!')
CounterA=CounterA+1
thA.start()
elif any(#Alarm B detection) and CounterB<1:
print('Alarm B!')
CounterB=CounterB+1
thB.start()
else:
pass
I think the crux of my problem is I can't have the resetter functions join the threads to main once they're finished with their delayed maths - but I also don't know how to do that in the main program without making it wait for the same amount of time and thus stalling everything...

You don't need threads for this at all.
Just keep track of the last time (time.time()) you triggered each alarm, and don't trigger them if less than 60 minutes (or whatever the threshold is) has elapsed since the last time.
Something like (semi pseudocode)...
import time
last_alarm_1 = 0 # a long time ago, so alarm can trigger immediately
# ...
if alarm_1_cond_met():
now = time.time()
if now - last_alarm_1 > 60 * 60: # seconds
send_alarm_1_mail()
last_alarm_1 = now
Repeat for alarm 2 :)

AKX has a better solution to your problem, but you should be aware of what this does when Aresetter() is called by the thA thread:
def Aresetter():
...
thA.join()
The thA.join() method doesn't do anything to the thA thread. All it does is, it waits for the thread to die, and then it returns. But, if it's the thA thread waiting for itself to die, it's going to be waiting for a very long time.

Also, there's this:
How to...restart a thread?
You can't. I don't want to explore why it makes any sense, but you just can't do that. It's not how threads work. If you want your program to do the same task more than one time "in another thread," you have a couple of options:
Create a new thread to do the task each time.
Create a single thread that does the same task again and again, possibly sleep()ing in between, or possibly awaiting some message/signal/trigger before each repetition.
Submit a task to a thread pool* each time you want the thing to be done.
Option (2) could be better than option (1) because creating and destroying threads is a lot of work. With option (2) you're only doing that once.
Option (1) could be better than option (2) because threads use a significant amount of memory. If the thread doesn't exist when it's not needed, then that memory could be used by something else.
Option (3) could be better than the both of them if the same thread pool is also used for other purposes in your program. The marginal cost of throwing a few more tasks at an already-existing thread pool is trivial.
* I don't know that Python has a ready-made, first-class ThreadPool class for you to use. It has this, https://stackoverflow.com/a/64373926/801894 , but I've never used it. It's not that hard though to create your own simple thread pool.

Interrupting a Queue.get

How can I interrupt a blocking Queue.get() in Python 3.X?
In Python 2.X setting a long timeout seems to work but the same cannot be said for Python 3.5.
Running on Windows 7, CPython 3.5.1, 64 bit both machine and Python.
Seems like it does not behave the same on Ubuntu.

The reason it works on Python 2 is that Queue.get with a timeout on Python 2 is implemented incredibly poorly, as a polling loop with increasing sleeps between non-blocking attempts to acquire the underlying lock; Python 2 doesn't actually feature a lock primitive that supports a timed blocking acquire (which is what a Queue internal Condition variable needs, but lacks, so it uses the busy loop). When you're trying this on Python 2, all you're checking is whether the Ctrl-C is processed after one of the (short) time.sleep calls finishes, and the longest sleep in Condition is only 0.05 seconds, which is so short you probably wouldn't notice even if you hit Ctrl-C the instant a new sleep started.
Python 3 has true timed lock acquire support (thanks to narrowing the number of target OSes to those which feature a native timed mutex or semaphore of some sort). As such, you're actually blocking on the lock acquisition for the whole timeout period, not blocking for 0.05s at a time between polling attempts.
It looks like Windows allows for registering handlers for Ctrl-C that mean that Ctrl-C doesn't necessarily generate a true signal, so the lock acquisition isn't interrupted to handle it. Python is informed of the Ctrl-C when the timed lock acquisition eventually fails, so if the timeout is short, you'll eventually see the KeyboardInterrupt, but it won't be seen until the timeout lapses. Since Python 2 Condition is only sleeping 0.05 seconds at a time (or less) the Ctrl-C is always processed quickly, but Python 3 will sleep until the lock is acquired.
Ctrl-Break is guaranteed to behave as a signal, but it also can't be handled by Python properly (it just kills the process) which probably isn't what you want either.
If you want Ctrl-C to work, you're stuck polling to some extent, but at least (unlike Python 2) you can effectively poll for Ctrl-C while live blocking on the queue the rest of the time (so you're alerted to an item becoming free immediately, which is the common case).
import time
import queue
def get_timed_interruptable(q, timeout):
stoploop = time.monotonic() + timeout - 1
while time.monotonic() < stoploop:
try:
return q.get(timeout=1) # Allow check for Ctrl-C every second
except queue.Empty:
pass
# Final wait for last fraction of a second
return q.get(timeout=max(0, stoploop + 1 - time.monotonic()))
This blocks for a second at a time until:
The time remaining is less than a second (it blocks for the remaining time, then allows the Empty to propagate normally)
Ctrl-C was pressed during the one second interval (after the remainder of that second elapses, KeyboardInterrupt is raised)
An item is acquired (if Ctrl-C was pressed, it will raise at this point too)

As mentioned in the comment thread to the great answer #ShadowRanger provided above, here is an alternate simplified form of his function:
import queue
def get_timed_interruptable(in_queue, timeout):
'''
Perform a queue.get() with a short timeout to avoid
blocking SIGINT on Windows.
'''
while True:
try:
# Allow check for Ctrl-C every second
return in_queue.get(timeout=min(1, timeout))
except queue.Empty:
if timeout < 1:
raise
else:
timeout -= 1
And as #Bharel pointed out in the comments, this could run a few milliseconds longer than the absolute timeout, which may be undesirable. As such here is a version with significantly better precision:
import time
import queue
def get_timed_interruptable_precise(in_queue, timeout):
'''
Perform a queue.get() with a short timeout to avoid
blocking SIGINT on Windows. Track the time closely
for high precision on the timeout.
'''
timeout += time.monotonic()
while True:
try:
# Allow check for Ctrl-C every second
return in_queue.get(timeout=max(0, min(1, timeout - time.monotonic())))
except queue.Empty:
if time.monotonic() > timeout:
raise

Just use get_nowait which won't block.
import time
...
while True:
if not q.empty():
q.get_nowait()
break
time.sleep(1) # optional timeout
This is obviously busy waiting, but q.get() does basically the same thing.

Python - How to wake up a sleeping process- multiprocessing?

I need to wake up a sleeping process ?
The time (t) for which it sleeps is calculated as t = D/S . Now since s is varying, can increase or decrease, I need to increase/decrease the sleeping time as well. The speed is received over a UDP procotol. So, how do I change the sleeping time of a process, keeping in mind the following:-
If as per the previous speed `S1`, the time to sleep is `(D/S1)` .
Now the speed is changed, it should now sleep for the new time,ie (D/S2).
Since, it has already slept for D/S1 time, now it should sleep for D/S2 - D/S1.
How would I do it?
As of right now, I'm just assuming that the speed will remain constant all throughout the program, hence not notifying the process. But how would I do that according to the above condition?
def process2():
p = multiprocessing.current_process()
time.sleep(secs1)
# send some packet1 via UDP
time.sleep(secs2)
# send some packet2 via UDP
time.sleep(secs3)
# send some packet3 via UDP
Also, as in threads,
1) threading.activeCount(): Returns the number of thread objects that are active.
2) threading.currentThread(): Returns the number of thread objects in the caller's thread control.
3) threading.enumerate(): Returns a list of all thread objects that are currently active.
What are the similar functions for getting activecount, enumerate in multiprocessing?

Not yet tested but, i think this could work :
Instead of using sleep, create a condition object and use it's wait() method.
Create a Timer object, which call the notify() method of the condition object when timed out.
If you want to change the sleep time, just discard the old Timer (with cancel() method), and create a new Timer.
* UPDATE *
I just tested this and it works.
This is the wait() in the process, don't forge to acquire it first.
def process(condition):
condition.acquire()
condition.wait()
condition.release()
and this is wake_up function, called from main process :
def wake_up(condition):
condition.acquire()
condition.notify()
condition.release()
and create and pass a condition object when creating a process (in your main, or other functions) :
condition=multiprocessing.Condition(multiprocessing.Lock())
p=multiprocessing.Process(target=process, args=(condition,))
p.start()
create a Timer (this timer thread will be created on main process) :
timer=threading.Timer(wake_up_time, wake_up, args(condition,))
start_time=time.time()
timer.start()
if you want to change the time, just stop it and make a new Timer :
timer.cancel()
elapsed_time=time.time-start_time
timer=threading.Timer(new_wake_up_time-elapsed_time, wake_up, args(condition,))
timer.start()

Python Event::wait with timeout gives delay

Running Python 2.6 and 2.7 on Windows 7 and Server 2012
Event::wait causes a delay when used with a timeout that is not triggered because event is set in time. I don't understand why.
Can someone explain?
The following program shows this and gives a possible explanation;
'''Shows that using a timeout in Event::wait (same for Queue::wait) causes a
delay. This is perhaps caused by a polling loop inside the wait implementation.
This polling loop sleeps some time depending on the timeout.
Probably wait timeout > 1ms => sleep = 1ms
A wait with timeout can take at least this sleep time even though the event is
set or queue filled much faster.'''
import threading
event1 = threading.Event()
event2 = threading.Event()
def receiver():
'''wait 4 event2, clear event2 and set event1.'''
while True:
event2.wait()
event2.clear()
event1.set()
receiver_thread = threading.Thread(target = receiver)
receiver_thread.start()
def do_transaction(timeout):
'''Performs a transaction; clear event1, set event2 and wait for thread to set event1.'''
event1.clear()
event2.set()
event1.wait(timeout = timeout)
while True:
# With timeout None this runs fast and CPU bound.
# With timeout set to some value this runs slow and not CPU bound.
do_transaction(timeout = 10.0)

Looking at the source code for wait() method of the threading.Condition class, there are two very different code paths. Without a timeout, we just wait on a lock forever, and when we get the lock, we return immediately.
However, with a timeout you cannot simply wait on the lock forever, and the low-level lock provides no timeout implementation. So the code sleeps for exponentially longer periods of time, after each sleep checking if the lock can be acquired. The relevant comment from the code:
# Balancing act: We can't afford a pure busy loop, so we
# have to sleep; but if we sleep the whole timeout time,
# we'll be unresponsive. The scheme here sleeps very
# little at first, longer as time goes on, but never longer
# than 20 times per second (or the timeout time remaining).
So in an average scenario where the condition/event cannot is not notified within a short period of time, you will see a 25ms delay (a random incoming event will arrive on average with half the max sleep time of 50ms left before the sleep ends).

Python: Pass or Sleep for long running processes?

I am writing an queue processing application which uses threads for waiting on and responding to queue messages to be delivered to the app. For the main part of the application, it just needs to stay active. For a code example like:
while True:
pass
or
while True:
time.sleep(1)
Which one will have the least impact on a system? What is the preferred way to do nothing, but keep a python app running?

I would imagine time.sleep() will have less overhead on the system. Using pass will cause the loop to immediately re-evaluate and peg the CPU, whereas using time.sleep will allow the execution to be temporarily suspended.
EDIT: just to prove the point, if you launch the python interpreter and run this:
>>> while True:
... pass
...
You can watch Python start eating up 90-100% CPU instantly, versus:
>>> import time
>>> while True:
... time.sleep(1)
...
Which barely even registers on the Activity Monitor (using OS X here but it should be the same for every platform).

Why sleep? You don't want to sleep, you want to wait for the threads to finish.
So
# store the threads you start in a your_threads list, then
for a_thread in your_threads:
a_thread.join()
See: thread.join

If you are looking for a short, zero-cpu way to loop forever until a KeyboardInterrupt, you can use:
from threading import Event
Event().wait()
Note: Due to a bug, this only works on Python 3.2+. In addition, it appears to not work on Windows. For this reason, while True: sleep(1) might be the better option.
For some background, Event objects are normally used for waiting for long running background tasks to complete:
def do_task():
sleep(10)
print('Task complete.')
event.set()
event = Event()
Thread(do_task).start()
event.wait()
print('Continuing...')
Which prints:
Task complete.
Continuing...

signal.pause() is another solution, see https://docs.python.org/3/library/signal.html#signal.pause
Cause the process to sleep until a signal is received; the appropriate handler will then be called. Returns nothing. Not on Windows. (See the Unix man page signal(2).)

I've always seen/heard that using sleep is the better way to do it. Using sleep will keep your Python interpreter's CPU usage from going wild.

You don't give much context to what you are really doing, but maybe Queue could be used instead of an explicit busy-wait loop? If not, I would assume sleep would be preferable, as I believe it will consume less CPU (as others have already noted).
[Edited according to additional information in comment below.]
Maybe this is obvious, but anyway, what you could do in a case where you are reading information from blocking sockets is to have one thread read from the socket and post suitably formatted messages into a Queue, and then have the rest of your "worker" threads reading from that queue; the workers will then block on reading from the queue without the need for neither pass, nor sleep.

Running a method as a background thread with sleep in Python:
import threading
import time
class ThreadingExample(object):
""" Threading example class
The run() method will be started and it will run in the background
until the application exits.
"""
def __init__(self, interval=1):
""" Constructor
:type interval: int
:param interval: Check interval, in seconds
"""
self.interval = interval
thread = threading.Thread(target=self.run, args=())
thread.daemon = True # Daemonize thread
thread.start() # Start the execution
def run(self):
""" Method that runs forever """
while True:
# Do something
print('Doing something imporant in the background')
time.sleep(self.interval)
example = ThreadingExample()
time.sleep(3)
print('Checkpoint')
time.sleep(2)
print('Bye')