My program involves a blocking loop. I'd like to use the a python REPL to modify program state while the program is running.
Context:
Often I have a blocking loop in my python code:
# main.py
import itertools
import time
global_state = 123
if __name__ == "__main__":
print("Starting main loop")
m = 0
for n in itertools.count():
time.sleep(1) # "computation"
global_state += 1
m += 10
print(f"{n=}, {m=}, {global_state=}")
When I run this program at the command line, I get something like this:
$ python -i main.py
Starting main loop
n=0, m=10, global_state=124
n=1, m=20, global_state=125
n=2, m=30, global_state=126
...
The loop will run for hours or days. Sometimes, while the loop is running, I desire to interactively modify some program state. For example, I'd like to set global_state = -1000 or m = 75. But the loop function is blocking...
Approaches to interactive modification of program state
Approach 1: Stopping and restarting the loop
I can use a KeyboardInterrupt to stop the loop. Since I have invoked python with the -i interactive flag, I can then modify the global state at the interactive REPL. I then restart the loop. This is a simple approach, but it has drawbacks:
If the loop modifies the global state (or has any side effects), the global state might be left in an inconsistent state when KeyboardInterrupt is raised; the exception could be raised at any point in the loop's control flow, and handling this exception properly can be tricky, especially if the business logic is complex.
When restarting the loop, the counter n will be reset to zero. Other state variables used by the loop (e.g. m) might also be reset or be in an inconsistent state.
Approach 2: Using threading
I can wrap the loop in a function and use threading to run the function in another thread:
def loop():
print("Starting main loop")
m = 0
for n in itertools.count():
time.sleep(1) # "computation"
global_state += 1
m += 10
print(f"{n=}, {m=}, {global_state=}")
import threading
thread = threading.Thread(target=loop)
thread.start()
When I invoke python -i main.py, I can use the REPL to interact with global state as the loop runs in another thread. The loop function still prints to stdout as before. The drawbacks of this approach are:
I can no longer interact with the loop's local state (e.g. I can't modify m) because the local state is wrapped in a function. This is a major drawback.
Using threading makes the program more fragile; threads can crash, and exceptions raised in threads need to be handled carefully. If the thread blocks or crashes or hangs, it can be hard to recover.
Stopping the loop becomes more difficult, because KeyboardInterrupt is no longer an option (instead I need to "cooperate" with the thread, sending it some signal that it should self-terminate).
Approach 3: Drop into REPL from within the loop
I can cooperate with the loop to drop into the repl:
import os
import code
if __name__ == "__main__":
...
for n in itertools.count():
... # business logic
if os.stat("use_repl.txt").st_size > 0: # file nonempty
code.interact(local=locals())
Here we are using a file to signal to the loop when we want to modify state. If the "use_repl.txt" file is nonempty then loop then calls code.interact to create an interactive Console. This has the added benefit of working even if the loop is wrapped in a function; local variables are loaded into the interactive console.
Drawbacks:
Reading a "use_repl.txt" file at every pass through the loop feels a bit clunky. But how else would one signal (cooperatively) to the loop that it should create a REPL?
The loop gets paused when code.interact is called. This differs from the threading-based solution, where the loop runs continuously. This may be an advantage or a disadvantage, depending on your use-case.
The REPL created by code.interact is not as polished as python's native interactive mode. For example, tab-completion no longer works.
An alternative to calling code.interact would be to call the breakpoint built-in function. Another alternative would be to invoke the break keyword, exiting the loop. One could then modify state and restart the loop (as in approach 1), without having to worry that a KeyboardInterrupt has created an inconsistent state.
The Question(s):
Are there any approaches that I've missed? Each of the approaches considered has significant drawbacks... Are there any best-practices regarding achievement of the desired result?
Debug the program, pause it when you want to inspect/modify state, then resume.
Inspired by xaa's answer:
import pdb
import signal
signal.signal(signal.SIGINT, pdb.Pdb().sigint_handler)
This sets a signal handler catching SIGINT (which would otherwise get translated into KeyboardInterrupt), dropping into pdb when CTRL-C is pressed. Thus CTRL-C can be used to interrupt the loop and inspect or modify state at any point, and execution can be resumed by typing continue at the pdb prompt.
Related
In Python's time module, there is a sleep() function, where you can make Python wait x seconds before resuming the program. Is there a way to do this indefinitely until a condition is met? For example:
while True:
time.sleep()
if x:
break
time.unsleep()
I am trying to make a pause function for my PyGame program. Any help is appreciated.
Something like this:
while not x: time.sleep(0.1)
will wait until x is true, sleeping a tenth of a second between checks. This is usually short enough for your script to seem to react instantly (in human terms) when x becomes true. You could use 0.01 instead if this is not quick enough. In my experience, today's computers are fast enough that checking a simple condition even every hundredth of a second doesn't really make a dent in CPU usage.
Of course, x should be something that can actually change, e.g. a function call.
Your code in the question implies that you want some other thread to resume your program. In that case you could use resumed = threading.Event(). You could create it in one thread and pass it into another:
while not resumed.wait(): # wait until resumed
"continue waiting"
Call resumed.set() to resume this code immediately.
I am trying to make a pause function for my PyGame program. Any help is appreciated.
Use pygame.time. Typically, you have the main loop where you update the state of the game and at the end of the loop you call clock.tick(60) # 60 fps. It is enough to use paused flag in this case, to skip the updating.
You could spin off a thread as follows:
import sys
import time
from threading import Thread
prepare_to_stop = 0
def in_background_thread():
while not prepare_to_stop:
# Program code here
print(prepare_to_stop)
time.sleep(0.1)
try:
th = Thread(target=in_background_thread)
th.start()
print("\nProgram will shut down after current operation is complete.\n")
time.sleep(10**8)
except KeyboardInterrupt:
prepare_to_stop = 1
print("Program shutting down...")
I am pretty new to python, and while using a module to print out packets being received I can't execute code while the while loop that reads the packets is being executed. Here is a basic example. Any help would be appreciated.
def foo():
while True:
print("bar")
foo()
print("foobar")
i want it to print foobar once after the while loop has stared, is this possible?
Typically in Python (and most other languages), you start with just one thread of execution.
A while True: ... is an infinite loop – unless code inside the loop breaks out, or something external interrupts it, it never ends. So your code never reaches the call to print('foobar') line.
You could put a special case inside the while loop, for the first pass through, that reports what you want. Or you could look into using multiple threads of execution – an advanced topic.
The program executes sequentially, so the print will never happen because of the infinite loop. So you must use a thread to circumvent this issue, allowing you to simultaneously execute code like so:
threading.Thread(target = foo).start() # new thread instead of foo() in the main thread
I need to run a (series of) infinite loops that must be able to check an externally set condition to terminate. I thought the threading module would allow that, but my efforts so fare have failed. Here is an example of what I am trying to do:
import threading
class Looping(object):
def __init__(self):
self.isRunning = True
def runForever(self):
while self.isRunning == True:
"do stuff here"
l = Looping()
t = threading.Thread(target = l.runForever())
t.start()
l.isRunning = False
I would have expected t.start to run in a separate thread, with l's attributes still accessible. This is not what happens. I tried the snippet above in the python shell (IPython). Execution of t start immediately after instantiation and it blocks any further input.
There is obviously something I am not getting right about the threading module.
Any suggestion on how to solve the problem?
You are calling runForever too early. Use target = l.runForever without parentheses.
A function call is not evaluated until after its arguments are. When you write runforever(), it calls the function right then, before even creating the thread. By just passing runForever, you pass the function object itself, which the threading apparatus can then call when it is ready. The point is that you don't actually want to call runForever; you just want to tell the threading code that runForever is what it should call later.
I have two functions, draw_ascii_spinner and findCluster(companyid).
I would like to:
Run findCluster(companyid) in the backround and while its processing....
Run draw_ascii_spinner until findCluster(companyid) finishes
How do I begin to try to solve for this (Python 2.7)?
Use threads:
import threading, time
def wrapper(func, args, res):
res.append(func(*args))
res = []
t = threading.Thread(target=wrapper, args=(findcluster, (companyid,), res))
t.start()
while t.is_alive():
# print next iteration of ASCII spinner
t.join(0.2)
print res[0]
You can use multiprocessing. Or, if findCluster(companyid) has sensible stopping points, you can turn it into a generator along with draw_ascii_spinner, to do something like this:
for tick in findCluster(companyid):
ascii_spinner.next()
Generally, you will use Threads. Here is a simplistic approach which assumes, that there are only two threads: 1) the main thread executing a task, 2) the spinner thread:
#!/usr/bin/env python
import time
import thread
def spinner():
while True:
print '.'
time.sleep(1)
def task():
time.sleep(5)
if __name__ == '__main__':
thread.start_new_thread(spinner, ())
# as soon as task finishes (and so the program)
# spinner will be gone as well
task()
This can be done with threads. FindCluster runs in a separate thread and when done, it can simply signal another thread that is polling for a reply.
You'll want to do some research on threading, the general form is going to be this
Create a new thread for findCluster and create some way for the program to know the method is running - simplest in Python is just a global boolean
Run draw_ascii_spinner in a while loop conditioned on whether it is still running, you'll probably want to have this thread sleep for a short period of time between iterations
Here's a short tutorial in Python - http://linuxgazette.net/107/pai.html
Run findCluster() in a thread (the Threading module makes this very easy), and then draw_ascii_spinner until some condition is met.
Instead of using sleep() to set the pace of the spinner, you can wait on the thread's wait() with a timeout.
It is possible to have a working example? I am new in Python. I have 6 tasks to run in one python program. These 6 tasks should work in coordinations, meaning that one should start when another finishes. I saw the answers , but I couldn't adopted the codes you shared to my program.
I used "time.sleep" but I know that it is not good because I cannot know how much time it takes each time.
# Sending commands
for i in range(0,len(cmdList)): # port Sending commands
cmd = cmdList[i]
cmdFull = convert(cmd)
port.write(cmd.encode('ascii'))
# s = port.read(10)
print(cmd)
# Terminate the command + close serial port
port.write(cmdFull.encode('ascii'))
print('Termination')
port.close()
# time.sleep(1*60)
I want to execute a function every 60 seconds on Python but I don't want to be blocked meanwhile.
How can I do it asynchronously?
import threading
import time
def f():
print("hello world")
threading.Timer(3, f).start()
if __name__ == '__main__':
f()
time.sleep(20)
With this code, the function f is executed every 3 seconds within the 20 seconds time.time.
At the end it gives an error and I think that it is because the threading.timer has not been canceled.
How can I cancel it?
You could try the threading.Timer class: http://docs.python.org/library/threading.html#timer-objects.
import threading
def f(f_stop):
# do something here ...
if not f_stop.is_set():
# call f() again in 60 seconds
threading.Timer(60, f, [f_stop]).start()
f_stop = threading.Event()
# start calling f now and every 60 sec thereafter
f(f_stop)
# stop the thread when needed
#f_stop.set()
The simplest way is to create a background thread that runs something every 60 seconds. A trivial implementation is:
import time
from threading import Thread
class BackgroundTimer(Thread):
def run(self):
while 1:
time.sleep(60)
# do something
# ... SNIP ...
# Inside your main thread
# ... SNIP ...
timer = BackgroundTimer()
timer.start()
Obviously, if the "do something" takes a long time, then you'll need to accommodate for it in your sleep statement. But, 60 seconds serves as a good approximation.
I googled around and found the Python circuits Framework, which makes it possible to wait
for a particular event.
The .callEvent(self, event, *channels) method of circuits contains a fire and suspend-until-response functionality, the documentation says:
Fire the given event to the specified channels and suspend execution
until it has been dispatched. This method may only be invoked as
argument to a yield on the top execution level of a handler (e.g.
"yield self.callEvent(event)"). It effectively creates and returns
a generator that will be invoked by the main loop until the event has
been dispatched (see :func:circuits.core.handlers.handler).
I hope you find it as useful as I do :)
./regards
It depends on what you actually want to do in the mean time. Threads are the most general and least preferred way of doing it; you should be aware of the issues with threading when you use it: not all (non-Python) code allows access from multiple threads simultaneously, communication between threads should be done using thread-safe datastructures like Queue.Queue, you won't be able to interrupt the thread from outside it, and terminating the program while the thread is still running can lead to a hung interpreter or spurious tracebacks.
Often there's an easier way. If you're doing this in a GUI program, use the GUI library's timer or event functionality. All GUIs have this. Likewise, if you're using another event system, like Twisted or another server-process model, you should be able to hook into the main event loop to cause it to call your function regularly. The non-threading approaches do cause your program to be blocked while the function is pending, but not between functioncalls.
Why dont you create a dedicated thread, in which you put a simple sleeping loop:
#!/usr/bin/env python
import time
while True:
# Your code here
time.sleep(60)
I think the right way to run a thread repeatedly is the next:
import threading
import time
def f():
print("hello world") # your code here
myThread.run()
if __name__ == '__main__':
myThread = threading.Timer(3, f) # timer is set to 3 seconds
myThread.start()
time.sleep(10) # it can be loop or other time consuming code here
if myThread.is_alive():
myThread.cancel()
With this code, the function f is executed every 3 seconds within the 10 seconds time.sleep(10). At the end running of thread is canceled.
If you want to invoke the method "on the clock" (e.g. every hour on the hour), you can integrate the following idea with whichever threading mechanism you choose:
import time
def wait(n):
'''Wait until the next increment of n seconds'''
x = time.time()
time.sleep(n-(x%n))
print(time.asctime())
[snip. removed non async version]
To use asyncing you would use trio. I recommend trio to everyone who asks about async python. It is much easier to work with especially sockets. With sockets I have a nursery with 1 read and 1 write function and the write function writes data from an deque where it is placed by the read function; and waiting to be sent. The following app works by using trio.run(function,parameters) and then opening an nursery where the program functions in loops with an await trio.sleep(60) between each loop to give the rest of the app a chance to run. This will run the program in a single processes but your machine can handle 1500 TCP connections insead of just 255 with the non async method.
I have not yet mastered the cancellation statements but I put at move_on_after(70) which is means the code will wait 10 seconds longer than to execute a 60 second sleep before moving on to the next loop.
import trio
async def execTimer():
'''This function gets executed in a nursery simultaneously with the rest of the program'''
while True:
trio.move_on_after(70):
await trio.sleep(60)
print('60 Second Loop')
async def OneTime_OneMinute():
'''This functions gets run by trio.run to start the entire program'''
with trio.open_nursery() as nursery:
nursery.start_soon(execTimer)
nursery.start_soon(print,'do the rest of the program simultaneously')
def start():
'''You many have only one trio.run in the entire application'''
trio.run(OneTime_OneMinute)
if __name__ == '__main__':
start()
This will run any number of functions simultaneously in the nursery. You can use any of the cancellable statements for checkpoints where the rest of the program gets to continue running. All trio statements are checkpoints so use them a lot. I did not test this app; so if there are any questions just ask.
As you can see trio is the champion of easy-to-use functionality. It is based on using functions instead of objects but you can use objects if you wish.
Read more at:
[1]: https://trio.readthedocs.io/en/stable/reference-core.html