Develop Reference

Python multiprocessing - best way to understand progress for each process

I would like to know the progress of my processes. At the moment what I am using is not very effective. This is a mwe:
import time
from multiprocessing import Pool as ProcessPool
import progressbar
import random
def some_random_calculation(n):
with progressbar.ProgressBar(max_value=n) as bar:
for i in range(0,n):
time.sleep(1)
bar.update(i)
if __name__=='__main__':
arguments = [random.randint(4,10) for i in range(4)]
pool = ProcessPool(4)
results = pool.map_async(some_random_calculation, arguments)
print(results.get())
pool.close()
pool.join()
In this case, I am using progressbar2, however, the output is continuously updated on the same line when there is more than 1 process:
You see from the image that the bars are in sorted order just because after the first bar is ended a new one is created by other processes. When there are multiple processes a single bar is updated on the same line.
I am looking for a fix to my problem, it would be cool to have n bars dynamically updated. However, probably there is a smarter way to get a sense of the progress of different processes. Any advice?

So this is by far not perfect, the subject is pretty complex if you want to get everything right. But one thing is sure, you should monitor the progress from outside the subprocesses.
The fastest and probably the easiest way to do it would be to have a call-function that returns the status, and the governor outside can keep the user updated on the progress. That would look something like this:
import os, signal
from threading import Thread, enumerate as t_enumerate
from time import time, sleep
from random import random
clear = lambda: os.system('cls' if os.name=='nt' else 'clear')
def sig_handler(signal, frame):
for t in t_enumerate():
if t.getName() != 'MainThread':
t.stop()
exit(0)
signal.signal(signal.SIGINT, sig_handler)
class worker(Thread):
def __init__(self, init_value=0):
Thread.__init__(self)
self.init_value = init_value
self.progress = 0
self.run_state = True
self.start() # Start ourselves instead of from outside.
def poll(self):
return self.progress
def stop(self):
self.run_state = False
def run(self):
main_thread = None
for t in t_enumerate():
if t.getName() == 'MainThread':
main_thread = t
break
while main_thread and self.run_state and main_thread.isAlive():
for i in range(0, 100):
self.init_value *= i
self.progress = i
sleep(random())
break # Yea kinda unessecary while loop. meh..
workers = [worker(0) for i in range(4)]
while len(t_enumerate()) > 1:
clear()
for index, worker_handle in enumerate(workers):
progress = worker_handle.poll()
print(f'Thread {index} is at {progress}/100.')
sleep(1)
The other approach would be for each thread to acquire a lock on the thread pool before printing. But this adds complexity, for starters, they would all need to sync when it's time to print, so that they don't arbitrarily acquire the lock to print, but you're in some other part of the output process where something else is being printed. Or they would print in the wrong order, or you would need to keep track of which row you should backtrack to re-write..
There's probably going to be a Threading guru here with a better answer, but this is my two cents. Just add a poller function, do a combined status update and live with the very limited processing power it takes to call each thread. Unless you have thousands of them, you won't have any performance impact by calling multiple times.

Running a python function after a certain time passes. Using Threading Timer, however it only runs once and then stops

I am trying to run a specific function in my python file. However, when I run the method with the timer that calls said function, it executes everything that it's supposed to, but then exits the job after the first time. I need it to continue to run the function after the specified time.
This is the function that contains the timer:
def executor(file):
x = datetime.today()
y = x.replace(day=x.day, hour=x.hour, minute=x.minute, second=x.second+10, microsecond=0)
delta_t = y-x
secs = delta_t.seconds+1
t = Timer(secs, parse_file, [file])
t.start()
The function that I am trying to call, is parse_file(file_name).
I am passing in the file_name when calling the executor function.

You haven't given enough detail of what your actual issue is, what code do you want to run more than once? Can you show the code that actually calls this function?
When you call start, the main thread will continue executing from that spot, while the task you scheduled will call the parse_file method at the specified time, and exit once complete. It sounds to me like you don't have anything that is keeping your main thread alive (that is, you don't have any more code after you call the executor).
Here is a small example showing how you can use the Timer to execute tasks while the main thread is still working. You can keep typing in input, and the print statement will show you all the threads that completed since the last time you typed an input.
from threading import Timer
import sys
def scheduled_task(arg):
print("\ntask complete arg %s!\n"%(arg))
def run_scheduled_task(arg):
timer = Timer(10, scheduled_task, [arg])
timer.start()
done = False
while not done:
user_input = input("Give me some input (exit to stop): ")
if user_input == 'exit':
print('Exiting')
done = True
else:
run_scheduled_task(user_input)

Timeout function if it takes too long

Forgive me, I am a newbie. I've surveyed some solution. But it is so hard for me to understand and to modify that. (Or maybe there is no solution in line with my imagination?). And I hope it can work on Ubuntu & Win7.
There is an example like this.
import random,time
def example():
while random.randint(0,10) != 1:
time.sleep(1)
print "down"
example()
And my imagination is...
If, the example() run over 10s, then rerun the example() again. (And maybe there is a place I can code anything else. Like I want to record the timeout event on TXT, and I can code the code at that place.)
Else, do nothing.
Is it possible to do that?

You can run a watch-dog in a separate thread that interrupts the main thread (that runs example) when it exceeds the time limit. Here is a possible implementation, with timeout lowered to 3s for ease of debugging:
import time, threading, thread
def watchdog_timer(state):
time.sleep(3)
if not state['completed']:
thread.interrupt_main()
def run_example():
while True:
state = {'completed': False}
watchdog = threading.Thread(target=watchdog_timer, args=(state,))
watchdog.daemon = True
watchdog.start()
try:
example()
state['completed'] = True
except KeyboardInterrupt:
# this would be the place to log the timeout event
pass
else:
break

I'm not sure if I fully understood what you want to achieve, but as you're constantly looping and only have one short and predictable blocking command, you could simply store the time when the loop started and then compare it to the current time once per loop iteration. If the difference exceeds your limit, do whatever you want:
import random,time
time_limit=10
def example():
time_start = time.time() # store current time (seconds since 1970)
while random.randint(0,10) != 1:
time.sleep(1)
if (time.time() >= time_start + time_limit): # compare with current time
print "canceled!"
break # break the while-loop
print "down"
example()

Python thread holds up the rest of the script

Running Python 3.2 on Windows 7 Pro 64 Bit.
OK I have some very basic code here that's just not behaving like I want it to.
#!/usr/bin/env python
import time
import threading
def shutdown(sleeptime):
time.sleep(sleeptime)
print('I have executed')
threading.Thread(target = shutdown(5)).start()
print('I go first')
The idea being that the script runs, it starts a thread which sleeps for 5 seconds then prints out 'I have executed'. In the meantime the script keeps going and prints out 'I go first'.
What actually happens is that the script starts the thread, everything waits for it to finish and then it continues. Clearly I'm not doing the threading correctly but I'm having trouble finding simple examples of threading with Python 3.

Your statement:
threading.Thread(target = shutdown(5)).start()
can be equivalently written as:
x = shutdown(5)
threading.Thread(target = x).start()
I.e. you are calling shutdown first, then passing the result to the Thread constructor.
You need to pass your function, without calling it, and your argument list, to Thread separately:
threading.Thread(target = shutdown, args = (5,)).start()

Your target is not evaluating to a function, but rather the value of shutdown(5), which is None. You probably want it to be more like:
def shutdown(sleeptime):
def shutter_downer():
time.sleep(sleeptime)
print('I have executed')
return shutter_downer

How can I run a certain function for a specific time in Python?

For example i have function do_something() and I want it to run for exactly 1 second (and not .923 seconds. It won't do. However 0.999 is acceptable.)
However it is very very important that the do_something must exactly run for 1 second. I was thinking of using UNIX time stamp and calculate the seconds. But I am really wondering if Python has a way to do this in a more aesthetic way...
The function do_something() is long-running, and must be interrupted after exactly one second.

I gather from comments that there's a while loop in here somewhere. Here's a class that subclasses Thread, based on the source code for _Timer in the threading module. I know you said you decided against threading, but this is just a timer control thread; do_something executes in the main thread. So this should be clean. (Someone correct me if I'm wrong!):
from threading import Thread, Event
class BoolTimer(Thread):
"""A boolean value that toggles after a specified number of seconds:
bt = BoolTimer(30.0, False)
bt.start()
bt.cancel() # prevent the booltimer from toggling if it is still waiting
"""
def __init__(self, interval, initial_state=True):
Thread.__init__(self)
self.interval = interval
self.state = initial_state
self.finished = Event()
def __nonzero__(self):
return bool(self.state)
def cancel(self):
"""Stop BoolTimer if it hasn't toggled yet"""
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.state = not self.state
self.finished.set()
You could use it like this.
import time
def do_something():
running = BoolTimer(1.0)
running.start()
while running:
print "running" # Do something more useful here.
time.sleep(0.05) # Do it more or less often.
if not running: # If you want to interrupt the loop,
print "broke!" # add breakpoints.
break # You could even put this in a
time.sleep(0.05) # try, finally block.
do_something()

The 'sched' module of Python appears suitable:
http://docs.python.org/library/sched.html
Apart from that: Python is not a real-time language nor does it usually run on a real-time OS. So your requirement is kind of questionable.

This bit of code might work for you. The description sounds like what you want:
http://programming-guides.com/python/timeout-a-function
It relies on the python signal module:
http://docs.python.org/library/signal.html