Develop Reference

How to start, stop, and restart a thread?

Fairly new to Python; working on a Raspberry Pi 4 with Python 3.4.3.
Got a code working to listen for 2 distinct alarms in my lab - one for a -80 freezer getting too warm, and the other for a -20 freezer. Code listens on a microphone, streams data, Fourier-transforms it, detects the peaks I'm interested in, and triggers events when they're found - eventually going to email me and my team if an alarm goes off, but still just testing with Print commands atm. Let's call them Alarm A/EventA and Alarm B/Event B.
I want it to trigger Event A when Alarm A is detected, but then wait 1 hour before triggering Event A again (if Alarm A is still going off/goes off again in an hour).
Meanwhile, though, I also want it to continue listening for Alarm B and trigger Event B if detected - again, only once per hour.
Since I can't just do time.sleep, then, I'm trying to do it with Threads - but am having trouble starting, stopping, and restarting a thread for the 1 hour (currently just 10 second for testing purposes) delay.
I have variables CounterA and CounterB set to 0 to start. When Alarm A is detected I have the program execute EventA and up CounterA to 1; ditto for AlarmB/EventB/CounterB. EventA and EventB are only triggered if CounterA and CounterB are <1.
I'm having a real hard time resetting the counters after a time delay, though. Either I end up stalling the whole program after an event is triggered, or I get the error that threads can only be started once.
Here are the relevant sections of the code:
import time
import threading
CounterA = 0
CounterB = 0
def Aresetter():
time.sleep(10)
global CounterA
CounterA=CounterA-1
thA.join()
def Bresetter():
time.sleep(10)
global CounterB
CounterB=CounterB-1
thB.join()
thA = threading.Thread(target = Aresetter)
thB = threading.Thread(target = Bresetter)
if any(#Alarm A detection) and CounterA<1:
print('Alarm A!')
CounterA=CounterA+1
thA.start()
elif any(#Alarm B detection) and CounterB<1:
print('Alarm B!')
CounterB=CounterB+1
thB.start()
else:
pass
I think the crux of my problem is I can't have the resetter functions join the threads to main once they're finished with their delayed maths - but I also don't know how to do that in the main program without making it wait for the same amount of time and thus stalling everything...

You don't need threads for this at all.
Just keep track of the last time (time.time()) you triggered each alarm, and don't trigger them if less than 60 minutes (or whatever the threshold is) has elapsed since the last time.
Something like (semi pseudocode)...
import time
last_alarm_1 = 0 # a long time ago, so alarm can trigger immediately
# ...
if alarm_1_cond_met():
now = time.time()
if now - last_alarm_1 > 60 * 60: # seconds
send_alarm_1_mail()
last_alarm_1 = now
Repeat for alarm 2 :)

AKX has a better solution to your problem, but you should be aware of what this does when Aresetter() is called by the thA thread:
def Aresetter():
...
thA.join()
The thA.join() method doesn't do anything to the thA thread. All it does is, it waits for the thread to die, and then it returns. But, if it's the thA thread waiting for itself to die, it's going to be waiting for a very long time.

Also, there's this:
How to...restart a thread?
You can't. I don't want to explore why it makes any sense, but you just can't do that. It's not how threads work. If you want your program to do the same task more than one time "in another thread," you have a couple of options:
Create a new thread to do the task each time.
Create a single thread that does the same task again and again, possibly sleep()ing in between, or possibly awaiting some message/signal/trigger before each repetition.
Submit a task to a thread pool* each time you want the thing to be done.
Option (2) could be better than option (1) because creating and destroying threads is a lot of work. With option (2) you're only doing that once.
Option (1) could be better than option (2) because threads use a significant amount of memory. If the thread doesn't exist when it's not needed, then that memory could be used by something else.
Option (3) could be better than the both of them if the same thread pool is also used for other purposes in your program. The marginal cost of throwing a few more tasks at an already-existing thread pool is trivial.
* I don't know that Python has a ready-made, first-class ThreadPool class for you to use. It has this, https://stackoverflow.com/a/64373926/801894 , but I've never used it. It's not that hard though to create your own simple thread pool.

Avoid increased runtime when opening threads in consecutive runs

I'm doing my final thesis and my topic is the creation of a software that will run and control an on-satellite experiment.
For that reason, I had to implement the reading of multiple sensors while the experiment is running. To do that, I wrote the code so that it will create a new thread for each sensor (multiprocessing might not work because I don't yet know which system the software will run on and therefore I can't say if there will be multiple processors available) and these threads run as daemons all the while the software does its thing. It works well, but now I need to test the whole thing and this is where it gets problematic:
To properly test each and every route the software could take, I have multiple variables that need to be set and so there will be a lot of test runs (I calculated around 17.000 but could be wrong). While the first few test runs go over quickly, each run takes longer and longer. I have fiddled around with my code a little bit and it turns out that without threading, each test takes about the same time. Unfortunately, I do not know why and my knowledge of the matter is very limited. The code concerning the threading is as follows:
This sets up the creation of each thread (sensor_list will be populated with multiple sensors in non-test conditions)
sensor_list = [<a single sensor>]
for sensor in sensor_list:
thread = threading.Thread(
target=self.store_sensor_data,
args=[sensor, query_frequency],
daemon=True,
name=f"Thread_{sensor}",
)
self.threads.append(thread)
thread.start()
The function which actually deals with getting and writing the sensor data, self.store_sensor_data, looks like this:
def store_sensor_data(self, sensor, frequency):
"""Get the current reading and result from 'sensor' and store them.
sensor (Sensor) - the sensor whose data shall be stored
frequency (int) - the frequency (in 1/s) at which data shall be stored
"""
value_id = 0
while not self.HALT:
value_id += 1
sensor_reading = sensor.get_reading()
sensor_result = sensor.get_result()
try:
# if there already is a list for that sensor, append the data to it
self.experiment_report.sensor_data_raw[str(sensor)].append(
(value_id, sensor_reading)
)
except KeyError:
# if there is no list, create one containing the current sensor value
self.experiment_report.sensor_data_raw[str(sensor)] = [
(value_id, sensor_reading)
]
# repeat the same for the 'result'
try:
self.experiment_report.sensor_data[str(sensor)].append(
(value_id, sensor_result)
)
except KeyError:
self.experiment_report.sensor_data[str(sensor)] = [
(value_id, sensor_result)
]
time.sleep(1 / frequency)
after the experiment is done, I stop the threads by calling
def interrupt_sensor_data_recording(self):
"""Interrupt the storing of sensor data by ending all daemon threads.
threads (list) - a list of currently running threads
"""
if len(self.threads) > 0:
self.HALT = True
for thread in self.threads:
if thread.is_alive():
logger.debug(f"Stopping thread '{thread.getName()}'")
thread.join()
else:
thread.join()
logger.debug(f"Thread '{thread.getName()}' was already stopped")
Now I am unsure if how I stop the daemon threads is appropriate and this might be the source of my problems. But there also might be some implication that I don't know about yet and in both cases, it would be nice if someone with more knowledge than me could help me out here.
Thanks in advance!

Python: Executing funciton in future time period with continuious postponement?

I have a Django web app which is used by embedded systems to upload regular data, currently every 2 minutes, to the server where Django just pops it into a database.
I'd like to create an alert system where by if there's no data uploaded from the remote system in a time period, say 10 minutes for example, I raise an alarm on the server, via email or something.
In other programming languages/environments I'd create a 10 minute timer to execute a function in 10 minutes, but every time data is uploaded I'd restart the timer. Thus hopefully the timer would never expire and the expiry function would never get called.
I might well have missed something obvious but if there is something I have missed it. This just does not seem possible in Python. Have I missed something?
At present looks like I need an external daemon monitoring the database :-(

You could use the time module for this:
import time
def didEventHappen():
# insert appropriate logic here to check
# for what you want to check for every 10 minutes
value = True # this is just a placeholder so the code runs
return value
def notifyServer():
print("Hello server, the event happened")
start = time.clock()
delay = 10 * 60 # 10 minutes, converted to seconds
while True:
interval = time.clock() - start
eventHappened = False
if interval >= delay:
eventHappened = didEventHappen()
start = time.clock() # reset the timer
if eventHappened:
notifyServer()
else:
print("event did not happen")
Alternatively, you could use the sched module.

Set delay between tasks in group in Celery

I have a python app where user can initiate a certain task.
The whole purpose of a task is too execute a given number of POST/GET requests with a particular interval to a given URL.
So user gives N - number of requests, V - number of requests per second.
How is it better to design such a task taking into account that due to a I/O latency the actual r/s speed could bigger or smaller.
First of all I decided to use Celery with Eventlet because otherwise I would need dozen of works which is not acceptable.
My naive approach:
Client starts a task using task.delay()
Inside task I do something like this:
#task
def task(number_of_requests, time_period):
for _ in range(number_of_requests):
start = time.time()
params_for_concrete_subtask = ...
# .... do some IO with monkey_patched eventlet requests library
elapsed = (time.time() - start)
# If we completed this subtask to fast
if elapsed < time_period / number_of_requests:
eventlet.sleep(time_period / number_of_requests)
A working example is here.
if we are too fast we try to wait to keep the desired speed. If we are too slow it's ok from client's prospective. We do not violate requests/second requirement. But will this resume correctly if I restart Celery?
I think this should work but I thought there is a better way.
In Celery I can define a task with a particular rate limit which will almost match my needs guarantee. So I could use Celery group feature and write:
#task(rate_limit=...)
def task(...):
#
task_executor = task.s(number_of_requests, time_period)
group(task_executor(params_for_concrete_task) for params_for_concrete_task in ...).delay()
But here I hardcode the the rate_limit which is dynamic and I do not see a way of changing it. I saw an example:
task.s(....).set(... params ...)
But I tried to pass rate_limit to the set method it it did not work.
Another maybe a bettre idea was to use Celery's periodic task scheduler. With the default implementation periods and tasks to be executed periodically is fixed.
I need to be able to dynamically create tasks, which run periodically a given number of times with a specific rate limit. Maybe I need to run my own Scheduler which will take tasks from DB? But I do not see any documentation around this.
Another approach was to try to use a chain function, but I could not figure out is there a delay between tasks parameter.

If you want to adjust the rate_limit dynamically you can do it using the following code. It is also creating the chain() at runtime.
Run this you will see that we successfully override the rate_limit of 5/sec to 0.5/sec.
test_tasks.py
from celery import Celery, signature, chain
import datetime as dt
app = Celery('test_tasks')
app.config_from_object('celery_config')
#app.task(bind=True, rate_limit=5)
def test_1(self):
print dt.datetime.now()
app.control.broadcast('rate_limit',
arguments={'task_name': 'test_tasks.test_1',
'rate_limit': 0.5})
test_task = signature('test_tasks.test_1').set(immutable=True)
l = [test_task] * 100
chain = chain(*l)
res = chain()
I also tried to override the attribute from within the class, but IMO the rate_limit is set when the task is registered by the worker, that is why the .set() has no effects. I'm speculating here, one would have to check the source code.
Solution 2
Implement your own waiting mechanism using the end time of the previous call, in the chain the return of the function is passed to the next one.
So it would look like this:
from celery import Celery, signature, chain
import datetime as dt
import time
app = Celery('test_tasks')
app.config_from_object('celery_config')
#app.task(bind=True)
def test_1(self, prev_endtime=dt.datetime.now(), wait_seconds=5):
wait = dt.timedelta(seconds=wait_seconds)
print dt.datetime.now() - prev_endtime
wait = wait - (dt.datetime.now() - prev_endtime)
wait = wait.seconds
print wait
time.sleep(max(0, wait))
now = dt.datetime.now()
print now
return now
#app.control.rate_limit('test_tasks.test_1', '0.5')
test_task = signature('test_tasks.test_1')
l = [test_task] * 100
chain = chain(*l)
res = chain()
I think this is actually more reliable than the broadcast.

Fast and Precise Python Repeating Timer

I need to send repeating messages from a list quickly and precisely. One list needs to send the messages every 100ms, with a +/- 10ms window. I tried using the code below, but the problem is that the timer waits the 100ms, and then all the computation needs to be done, making the timer fall out of the acceptable window.
Simply decreasing the wait is a messy, and unreliable hack. The there is a Lock around the message loop in the event the list gets edited during the loop.
Thoughts on how to get python to send messages consistently around 100ms? Thanks
from threading import Timer
from threading import Lock
class RepeatingTimer(object):
def __init__(self,interval, function, *args, **kwargs):
super(RepeatingTimer, self).__init__()
self.args = args
self.kwargs = kwargs
self.function = function
self.interval = interval
self.start()
def start(self):
self.callback()
def stop(self):
self.interval = False
def callback(self):
if self.interval:
self.function(*self.args, **self.kwargs)
Timer(self.interval, self.callback, ).start()
def loop(messageList):
listLock.acquire()
for m in messageList:
writeFunction(m)
listLock.release()
MESSAGE_LIST = [] #Imagine this is populated with the messages
listLock = Lock()
rt = RepeatingTimer(0.1,loop,MESSAGE_LIST)
#Do other stuff after this
I do understand that the writeFunction will cause some delay, but not more than the 10ms allowed. I essentially need to call the function every 100ms for each message. The messagelist is small, usually less than elements.
The next challenge is to have this work with every 10ms, +/-1ms :P

Yes, the simple waiting is messy and there are better alternatives.
First off, you need a high-precision timer in Python. There are a few alternatives and depending on your OS, you might want to choose the most accurate one.
Second, you must be aware of the basics preemptive multitasking and understand that there is no high-precision sleep function, and that its actual resolution will differ from OS to OS too. For example, if we're talking Windows, the minimal sleep interval might be around 10-13 ms.
And third, remember that it's always possible to wait for a very accurate interval of time (assuming you have a high-resolution timer), but with a trade-off of high CPU load. The technique is called busy waiting:
while(True):
if time.clock() == something:
break
So, the actual solution is to create a hybrid timer. It will use the regular sleep function to wait the main bulk of the interval, and then it'll start probing the high-precision timer in the loop, while doing the sleep(0) trick. Sleep(0) will (depending on the platform) wait the least possible amount of time, releasing the rest of the remaining time slice to other processes and switching the CPU context. Here is a relevant discussion.
The idea is thoroughly described in the Ryan Geiss's Timing in Win32 article. It's in C and for Windows API, but the basic principles apply here as well.

Store the start time. Send the message. Get the end time. Calculate timeTaken=end-start. Convert to FP seconds. Sleep(0.1-timeTaken). Loop back.

try this:
#!/usr/bin/python
import time; # This is required to include time module.
from threading import Timer
def hello(start, interval, count):
ticks = time.time()
t = Timer(interval - (ticks-start-count*interval), hello, [start, interval, count+1])
t.start()
print "Number of ticks since 12:00am, January 1, 1970:", ticks, " #", count
dt = 1.25 # interval in sec
t = Timer(dt, hello, [round(time.time()), dt, 0]) # start over at full second, round only for testing here
t.start()