I would like to get a basic idea working in Python:
Let's say I have a blocking call to a function that reads the current speed of a car (I use python-can for this, and wait for the corresponding message to appear on the bus).
On the other hand, I have an infinite loop that prints as precisely as possible the speed of the car.
What I would like is the value of the speed to be updated directly, so that the infinite loop can print it as soon as it changes.
Right now I had three ideas:
Call my get_speed function in the loop, and block. This works, but only if this is the only value I am willing to print (spoiler alert: it's not). If I want to print precisely the RPM too, I have to wait for the speed to be found, possibly missing multiple RPM value before.
Create another "thread" (subprocess/multiprocessing/thread, whatever you call it) that updates a global variable. I know this works, but meh, I feel like I could have something smarter
From what I saw in javascript, there is a way to ask a function for a result, and keep going until this result is found (this rings some CPU bells right now ahah). I kinda want something like this, which could make me do the following
speed = get_speed()
rpm = get_rpm()
while True:
print("RPM: {0} - Speed: {1}".format(rpm, speed))
This would (in my fairy world) actually display something like this:
RPM: None - Speed: None # Function haven't returned yet, waiting
RPM: None - Speed: None # Same here
RPM: None - Speed: None # Same here
RPM: 300 - Speed: None # get_rpm has returned
RPM: 300 - Speed: None # Nothing happened
RPM: 303 - Speed: 0 # get_rpm and get_speed have returned
RPM: 303 - Speed: 0
RPM: 312 - Speed: 0 # etc.
Right now what I have is something like this, which doesn't work (at all)
#!/usr/bin/env python3
import datetime
import asyncio
import random
from time import sleep
async def _can_get_speed():
# My long function
print ("In can_get_speed")
sleep(4)
print ("Out can_get_speed")
r = random.randint(0, 10)
print(r)
return r
async def can_get_speed():
return await asyncio.gather(_can_get_speed())
if __name__ == "__main__":
loop = asyncio.get_event_loop()
speed = 0
speed = loop.call_soon(can_get_speed, loop)
loop.run_forever()
while True:
print("Speed: {0}".format(speed))
My two questions are:
Is this the most correct way to do it (--> asyncio, I mean), and
if not what is ?
If it is, can somebody help me understanding
asyncio better :)
Thanks in advance all!
I was able to get the expected result like this:
#!/usr/bin/env python3
import datetime
import asyncio
import random
from time import sleep
from multiprocessing import Process, Queue
def can_get_speed(q):
while True:
print("IN get_speed")
r = random.randint(1, 4)
sleep(r)
print("Return {0}".format(r))
q.put(r)
def can_get_rpm(q):
while True:
print("IN get_rpm")
r = random.randint(1, 4)
sleep(r)
print("Return {0}".format(r))
q.put(r)
if __name__ == "__main__":
q = Queue()
q2 = Queue()
p = Process(target=can_get_speed, args=(q,))
p2 = Process(target=can_get_rpm, args=(q2,))
p.start()
p2.start()
speed = None
rpm = None
while True:
if not q.empty():
speed = q.get()
if not q2.empty():
rpm = q2.get()
print("Speed: {0} - RPM: {1} - {2}".format(speed, rpm, datetime.datetime.now()))
Is there a smarter way ?
Related
I Have a counting function that I would like to start and restart while getting the live variables to use in another function my problem is while using threading it seams like even global variables don't seem to work to pass variables around. What I want the code to do is have a counter be triggered as needed or maybe free running I'm not sure yet. To be able to reset the counter and get the value of the counter.
Right now the counter will start and run fine but the print_stuff function keeps telling me that there is no attribute countval.
The count thread gets started at startup but I don't necessarily want it to start up immediately, I would like to trigger it as needed but I cant put count_thread.start() twice or it will through a error so I'm calling the thread at startup and then calling the function again to restart it as needed. Maybe there is a more elegant way of doing that.?
import threading
import time
def count():
global countval
for countval in range(3):
print('looping')
time.sleep(1)
def print_stuff():
global countval
e = input("press enter to start")
count()
while True:
if countval == 3:
print("time out")
count_thread = threading.Thread(target=count)
print_thread = threading.Thread(target=print_stuff)
print_thread.start()
count_thread.start()
print_stuff is getting to the if statement before the count function is able to create the variable. Just do them in the opposite order. Either that, or create a global countval = 0 to start things off.
To solve the no attribute problem you can use Queue,
and if you want to stop your counting thread you can set a global variable or you can pass a function (using lambda or inner function or ...) to do that.
Here is one way to do that:
import threading
import time
from queue import Queue
from typing import Callable
def count(q, stop_counting):
# type: (Queue, Callable[[], bool]) -> None
for countval in range(3):
if stop_counting():
print('stopped')
break
print(f'looping {countval}')
q.put(countval)
time.sleep(1)
def print_stuff(q):
# type: (Queue) -> None
while True:
countval = q.get()
print(f'countval gotten: {countval}')
if countval == 3:
print("time out")
def main():
flag_stop_counting = False
q = Queue()
def stop_counting():
return flag_stop_counting
count_thread = threading.Thread(target=count, args=(q, stop_counting,))
print_thread = threading.Thread(target=print_stuff, args=(q,))
print_thread.start()
count_thread.start()
time.sleep(1.25)
flag_stop_counting = True
if __name__ == '__main__':
main()
In this code:
counter checks if it should stop counting or not
counter puts the value that it made to q
print_stuff get the value from q (Note: he waits until counter puts his value in q)
To check that program works:
after 1.25 seconds we change the value of flag_stop_counting
But if you want your counter to only have a for, i guess it's better to don't make it as a thread and run it whenever you want.
Hope it was helpful.
I'm making a simple client/server program in Python 3 and in the client I would like a clock or printout of the running time. I'm trying to make it in a loop that starts at the beginning of the program, but in a thread so the rest of the code keeps going.
class time_thread():
def run(self):
loop = 0
while (zetime > -1):
print(zetime);
zetime = zetime + 1;
time_thread.start()
zetime = 0
This is what I have so far, but it doesn't work. It says:
time_thread has no attribute start()
I'm new to this and haven't used threads before, so I'm not sure how to go about this. Is there a better way?
I think this is what you're looking for:
import time, sys
zetime = 0
while (zetime > -1):
sys.stdout.write("\r" + str(zetime))
sys.stdout.flush()
time.sleep(1)
zetime = zetime + 1;
First of all , to use Thread module, you have to inherit the class Thread on your class, so you can use their methods like start.
To calculate time, you can use datetime.
from datetime import datetime
from time import sleep
start_time = datetime.now()
sleep(5) # code that you want to calculate.
end_time = datetime.now()
print(end_time - start_time)
Just place this
So let's say you define a function elapsed_time such as:
import time, sys
def elapsed_time(time_start):
''' time_start: a time.time()
Goal: print the elapsed time since time_start '''
# Allow to stop the counting once bool_time = False in the main program
global bool_elapsed_time
# loop while the condition
while bool_elapsed_time == True:
# Note: erase and write below does not work in spyder but in a shell yes
print ("Elapsed time: {} seconds".format(int(time.time() - time_start))),
sys.stdout.flush()
print ("\r"),
time.sleep(1)
# erase the line with elapsed time to clean the shell at the end
sys.stdout.flush()
print ("\r"),
Then in your program:
import threading
bool_elapsed_time = True
t = threading.Thread(name='child procs', target=elapsed_time, args=(time.time(),))
t.start()
## Here add the job you want to do as an example:
time.sleep(10)
bool_elapsed_time = False #to stop the elapsed time printing
Should do the job you want to do.
Note: I used python 2.7 so it might be slightly different with 3.x
First of all, I am new to Python and not familiar with its functionalities. I've been mainly using MATLAB.
PC brief spec.: Windows 10, Intel i7
I am trying to make a timer class for periodic execution of a function such as MATLAB has, which is obviously borrowed from Java timer. The MATLAB timer has an about 1 ms resolution and I've never seen it exceeds 2 ms in any situation. In fact, it is accurate enough for my project.
Recently, I planned to move to Python because of the poor parallel computing and web access features of MATLAB. However, unfortunately, the standard packages of Python offer somewhat low-level of timer (threading.Timer) compared to MATLAB that I had to make my own timer class. First, I referred to the QnA Executing periodic actions in Python [duplicate]. The solution suggested by Michael Anderson gives a simple idea of drift correction. He used time.sleep() to keep the period. The approach is highly accurate and sometimes showed better accuracy over the MATLAB timer. approx. 0.5 ms resolution. However, the timer cannot be interrupted (pause or resume) during being captured in time.sleep(). But I sometimes have to stop immediately regardless of whether it is in sleep() or not.
A solution to the problem I found is to utilize the Event class in threading package. Refer to Python threading.timer - repeat function every 'n' seconds
. Using the timeout feature of Event.wait(), I could make a time gap between executions and it is used to keep the period. That is, the event is usually cleared so that wait(timeout) can act like time.sleep(interval) and I could exit from wait() immediately, when needed, by setting event.
Everything seemed fine then but there is a critical problem in Event.wait(). The time delay varies too largely from 1 ~ 15 ms. I think it comes from the overhead of Event.wait().
I made an example code that shows accuracy comparison between time.sleep() and Event.wait(). This sums total of 1000 iterations of 1 ms sleep() and wait() to see the accumulated time error. The expected result is about 1.000.
import time
from threading import Event
time.sleep(3) # to relax
# time.sleep()
tspan = 1
N = 1000
t1 = time.perf_counter()
for _ in range(N):
time.sleep(tspan/N)
t2 = time.perf_counter()
print(t2-t1)
time.sleep(3) # to relax
# Event.wait()
tspan = 1
event = Event()
t1 = time.perf_counter()
for _ in range(N):
event.wait(tspan/N)
t2 = time.perf_counter()
print(t2-t1)
Result:
1.1379848184879964
15.614547161211096
The result shows that time.sleep() is much better in accuracy. But I cannot purely rely on time.sleep() as previously mentioned.
In summary,
time.sleep(): accurate but not interruptible
threading.Event.wait(): inaccurate but interruptible
I am currently thinking of a compromise: just as in the example, make a loop of tiny time.sleep() (of 0.5 ms interval) and exit the loop using if-statement and break when needed. As far as I know, the method is used in Python 2.x Python time.sleep() vs event.wait().
It was a verbose introduction, but my question can be summarized as follows.
Can I force thread process to break from time.sleep() by an external signal or event? (This seems to be most efficient.???)
To make Event.wait() more accurate or to reduce overhead time.
Are there any better approaches aside of sleep() and Event.wait() approach to improve timing precision.
Thank you very much.
I ran into the same timing issue with Event.wait(). The solution I came up with was to create a class which mimics threading.Event. Internally, it uses a combination of a time.sleep() loop and a busy loop for greatly increased precision. The sleep loop runs in a separate thread so that the blocking wait() call in the main thread can still be immediately interrupted. When the set() method is called, the sleep thread should terminate shortly afterwards. Also, in order to minimize CPU utilization, I made sure that the busy loop will never run for more than 3 milliseconds.
Here is my custom Event class along with a timing demo at the end (the printed execution times from the demo will be in nanoseconds):
import time
import _thread
import datetime
class Event:
__slots__ = (
"_flag", "_lock", "_nl",
"_pc", "_waiters"
)
_lock_type = _thread.LockType
_timedelta = datetime.timedelta
_perf_counter = time.perf_counter
_new_lock = _thread.allocate_lock
class _switch:
__slots__ = ("_on",)
def __call__(self, on: bool = None):
if on is None:
return self._on
self._on = on
def __bool__(self):
return self._on
def __init__(self):
self._on = False
def clear(self):
with self._lock:
self._flag(False)
def is_set(self) -> bool:
return self._flag()
def set(self):
with self._lock:
self._flag(True)
waiters = self._waiters
for waiter in waiters:
waiter.release()
waiters.clear()
def wait(
self,
timeout: float = None
) -> bool:
with self._lock:
return self._wait(self._pc(), timeout)
def _new_waiter(self) -> _lock_type:
waiter = self._nl()
waiter.acquire()
self._waiters.append(waiter)
return waiter
def _wait(
self,
start: float,
timeout: float,
td=_timedelta,
pc=_perf_counter,
end: _timedelta = None,
waiter: _lock_type = None,
new_thread=_thread.start_new_thread,
thread_delay=_timedelta(milliseconds=3)
) -> bool:
flag = self._flag
if flag:
return True
elif timeout is None:
waiter = self._new_waiter()
elif timeout <= 0:
return False
else:
delay = td(seconds=timeout)
end = td(seconds=start) + delay
if delay > thread_delay:
mark = end - thread_delay
waiter = self._new_waiter()
new_thread(
self._wait_thread,
(flag, mark, waiter)
)
lock = self._lock
lock.release()
try:
if waiter:
waiter.acquire()
if end:
while (
not flag and
td(seconds=pc()) < end
):
pass
finally:
lock.acquire()
if waiter and not flag:
self._waiters.remove(waiter)
return flag()
#staticmethod
def _wait_thread(
flag: _switch,
mark: _timedelta,
waiter: _lock_type,
td=_timedelta,
pc=_perf_counter,
sleep=time.sleep
):
while not flag and td(seconds=pc()) < mark:
sleep(0.001)
if waiter.locked():
waiter.release()
def __new__(cls):
_new_lock = cls._new_lock
_self = object.__new__(cls)
_self._waiters = []
_self._nl = _new_lock
_self._lock = _new_lock()
_self._flag = cls._switch()
_self._pc = cls._perf_counter
return _self
if __name__ == "__main__":
def test_wait_time():
wait_time = datetime.timedelta(microseconds=1)
wait_time = wait_time.total_seconds()
def test(
event=Event(),
delay=wait_time,
pc=time.perf_counter
):
pc1 = pc()
event.wait(delay)
pc2 = pc()
pc1, pc2 = [
int(nbr * 1000000000)
for nbr in (pc1, pc2)
]
return pc2 - pc1
lst = [
f"{i}.\t\t{test()}"
for i in range(1, 11)
]
print("\n".join(lst))
test_wait_time()
del test_wait_time
Chris D's custom Event class works impressively well! For practical purposes, I have included it into an installable package (https://github.com/ovinc/oclock, install with pip install oclock) that also includes other timing tools. From version 1.3.0 of oclock and onwards, one can use the custom Event class discussed in Chris D's answer, e.g.
from oclock import Event
event = Event()
event.wait(1)
with the usual set(), clear(), is_set(), wait() methods of the Event class.
The timing accuracy is much better than with threading.Event, in Windows in particular. For example on a Windows machine with 1000 repeated loops, I get a standard deviation in the duration of the loop of 7ms for threading.Event and less than 0.01 ms for oclock.Event. Props to Chris D!
Note: The oclock package is under the GPLv3 license for compatibility with StackOverflow's CC BY-SA 4.0.
Thank you for this topic and all answers. I also experienced some troubles with inaccurate timing (Windows 10 + Python 3.9 + Threading).
The solution is to use oclock package and also change (temporarily) resolution of Windows system timer by wres package. This package utilizes undocumented Windows API function NtSetTimerResolution (warning: resolution is changed system-wide).
Application of oclock package only does not solve the problem.
With both python packages applied, the code below schedules periodic event correctly and precisely enough. If terminated, original timer resolution is restored.
import threading
import datetime
import time
import oclock
import wres
class Job(threading.Thread):
def __init__(self, interval, *args, **kwargs):
threading.Thread.__init__(self)
# use oclock.Event() instead of threading.Event()
self.stopped = oclock.Event()
self.interval = interval.total_seconds()
self.args = args
self.kwargs = kwargs
def stop(self):
self.stopped.set()
self.join()
def run(self):
prevTime = time.time()
while not self.stopped.wait(self.interval):
now = time.time()
print(now - prevTime)
prevTime = now
# Set system timer resolution to 1 ms
# Automatically restore previous resolution when exit with statement
with wres.set_resolution(10000):
# Create thread with periodic task called every 10 ms
job = Job(interval=datetime.timedelta(seconds=0.010))
job.start()
try:
while True:
time.sleep(1)
# Hit Ctrl+C to terminate main loop and spawned thread
except KeyboardInterrupt:
job.stop()
I have written a bit of code to test out multiprocessing before I implement in other code. The main code sends a number to the other program (running on another core). That program increments and returns the number. main code increments and returns the value again. All works well, but the loop in the main program, which is a while time < Timeout form, is never exited. Seems simple enough to me, but it never exists the loop. I wondered if it was hanging when no value is returned (.get() ), but I have tried 'try:', and making the timeout very short and the loop huge. Any suggestions what is going on?
The code is running on Windows7, and will eventually run on a Raspberry Pi 2.
Main program
import multiprocessing as mp
import multi_processing_slave as MPS
from time import perf_counter as TimeIs
if __name__ == "__main__":
print("Hello World")
mp.set_start_method("spawn")
q = mp.Queue()
r = mp.Queue()
p = mp.Process(target = MPS.foo, args = (q, r))
p.start()
ThisVar = 0
Timer = TimeIs() + 2
while TimeIs() < Timer - 1: pass
print("time remaining is", Timer - TimeIs())
while TimeIs() < Timer:
#try:
r.put(ThisVar)
#except: pass
#try:
ThisVar = int(q.get()) + 1
#except:
#r.put(ThisVar)
print("master ThisVar", ThisVar, "and time remaining is", round(Timer - TimeIs(), 4))
#p.join()
#p.close()
p.terminate()
print("at end, ThisVar is", ThisVar, "and", Timer - TimeIs(), "seconds remaining")
Slave program named multi_processing_slave
def foo(q, r):
for i in range(100):
ThisVar2 = r.get() + 1
q.put(ThisVar2)
print("foo value", ThisVar2)
print("foo has finished")
return
After the slave process exits, your master process does one more r.put() and then keeps waiting for q.get() to return. You can solve the problem by providing a timeout value (in seconds) to q.get():
ThisVar = int(q.get(timeout=1)) + 1
Note that timing out on q.get() will raise the Empty exception.
How you could find this problem yourself:
Add debugging print statements that show the progress and state of your program.
Learn to use a debugger.
I am running pool.map on big data array and i want to print report in console every minute.
Is it possible? As i understand, python is synchronous language, it can't do this like nodejs.
Perhaps it can be done by threading.. or how?
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
# I want to call this function every minute
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
results = pool.map(make_job, data)
pool.close()
pool.join()
You can use a permanent threaded timer, like those from this question: Python threading.timer - repeat function every 'n' seconds
from threading import Timer,Event
class perpetualTimer(object):
# give it a cycle time (t) and a callback (hFunction)
def __init__(self,t,hFunction):
self.t=t
self.stop = Event()
self.hFunction = hFunction
self.thread = Timer(self.t,self.handle_function)
def handle_function(self):
self.hFunction()
self.thread = Timer(self.t,self.handle_function)
if not self.stop.is_set():
self.thread.start()
def start(self):
self.stop.clear()
self.thread.start()
def cancel(self):
self.stop.set()
self.thread.cancel()
Basically this is just a wrapper for a Timer object that creates a new Timer object every time your desired function is called. Don't expect millisecond accuracy (or even close) from this, but for your purposes it should be ideal.
Using this your example would become:
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
# set up the monitor to make run the function every minute
monitor = PerpetualTimer(60,display_status)
monitor.start()
results = pool.map(make_job, data)
pool.close()
pool.join()
monitor.cancel()
EDIT:
A cleaner solution may be (thanks to comments below):
from threading import Event,Thread
class RepeatTimer(Thread):
def __init__(self, t, callback, event):
Thread.__init__(self)
self.stop = event
self.wait_time = t
self.callback = callback
self.daemon = True
def run(self):
while not self.stop.wait(self.wait_time):
self.callback()
Then in your code:
def main():
data = [...]
pool = ThreadPool(45)
stop_flag = Event()
RepeatTimer(60,display_status,stop_flag).start()
results = pool.map(make_job, data)
pool.close()
pool.join()
stop_flag.set()
One way to do this, is to use main thread as the monitoring one. Something like below should work:
def main():
data = [...]
results = []
step = 0
pool = ThreadPool(16)
pool.map_async(make_job, data, callback=results.extend)
pool.close()
while True:
if results:
break
step += 1
sleep(1)
if step % 60 == 0:
print "status update" + ...
I've used .map() instead of .map_async() as the former is synchronous one. Also you probably will need to replace results.extend with something more efficient. And finally, due to GIL, speed improvement may be much smaller than expected.
BTW, it is little bit funny that you wrote that Python is synchronous in a question that asks about ThreadPool ;).
Consider using the time module. The time.time() function returns the current UNIX time.
For example, calling time.time() right now returns 1410384038.967499. One second later, it will return 1410384039.967499.
The way I would do this would be to use a while loop in the place of results = pool(...), and on every iteration to run a check like this:
last_time = time.time()
while (...):
new_time = time.time()
if new_time > last_time+60:
print "status update" + ...
last_time = new_time
(your computation here)
So that will check if (at least) a minute has elapsed since your last status update. It should print a status update approximately every sixty seconds.
Sorry that this is an incomplete answer, but I hope this helps or gives you some useful ideas.