I am pretty new to python, and while using a module to print out packets being received I can't execute code while the while loop that reads the packets is being executed. Here is a basic example. Any help would be appreciated.
def foo():
while True:
print("bar")
foo()
print("foobar")
i want it to print foobar once after the while loop has stared, is this possible?
Typically in Python (and most other languages), you start with just one thread of execution.
A while True: ... is an infinite loop – unless code inside the loop breaks out, or something external interrupts it, it never ends. So your code never reaches the call to print('foobar') line.
You could put a special case inside the while loop, for the first pass through, that reports what you want. Or you could look into using multiple threads of execution – an advanced topic.
The program executes sequentially, so the print will never happen because of the infinite loop. So you must use a thread to circumvent this issue, allowing you to simultaneously execute code like so:
threading.Thread(target = foo).start() # new thread instead of foo() in the main thread
Related
I'm developing a standard python script file (no servers, no async, no multiprocessing, ...) i.e. a classic data science program where I load data, process it as dataframes, and so on. Everything is synchronous.
At some point, I need to call a function of an external library which is totally external to me (I have no control on it, I don't know how it does what it does), like
def inside_my_function(...):
# My code
result = the_function(params)
# Other code
Now, this the_function sometimes never terminates (I don't know why, probably there are bugs or some conditions which makes it stuck, but it's completely random), and when it happens my program gets stuck as well.
Since I have to use it and it cannot be modified, I would like to know if there is a way for example to wrap it in another function which calls the_function, waits for some timeout, and if the_function returns before the timeout the result is returned, otherwise the_function is somehow killed, aborted, skipped, whatever, and retried up to n times.
I realise that in order to execute the_function and check for timeout at the same time for example multithreading will be needed, but I'm not sure if it makes sense and how to implement it correctly without doing bad practices.
How would you proceed?
EDIT: I would avoid multiprocessing because of the great overhead and because I don't want to overcomplicate things with serializability and so on.
Thank you
import time
import random
import threading
def func_that_waits():
t1 = time.time()
while (time.time() - t1) <= 3:
time.sleep(1)
if check_unreliable_func.worked:
break
if not check_unreliable_func.worked:
print("unreliable function has been working for too long, it's killed.")
def check_unreliable_func(func):
check_unreliable_func.worked = False
def inner(*args,**qwargs):
func(*args,**qwargs)
check_unreliable_func.worked = True
return inner
def unreliable_func():
working_time = random.randint(1,6)
time.sleep(working_time)
print(f"unreliable_func has been working for {working_time} seconds")
to_wait = threading.Thread(target=func_that_waits)
main_func = threading.Thread(target=check_unreliable_func(unreliable_func), daemon=True)
main_func.start()
to_wait.start()
Unreliable_func - the function we do not know if it works
check_unreliable_func(func) - decorator the only purpose of which is to make to_wait thread know that unreliable_func returned something and so there is no sense for to_wait to work further
the main thing to understand is that main_func thread is daemon one so it means that when to_wait thread is terminated all daemon threads are terminated automatically and no matter what they've been doing in the moment
Of course it's really far from being best practice, I just show how it can be done. And how it should be done - I myself would be glad to see it too.
This seems like a particularly confusing question based on the other similar answers I found on SO. I have code similar to the following:
def parentFunction():
# Other code
while True:
var1, var2 = anotherFunction1() # Getting client details after listening on open port
threading.Thread(target = anotherFunction2, args=(var1, var2)).start()
childFunction(var1,var2)
print("PRINT #1: Running in Parent Function") # This only prints once for some reason
def childFunction(var1, var2):
threading.Timer(10, childFunction, args=(var1,var2)).start()
print("PRINT #2: Running in child function") # Prints every 10 seconds
# Other code
if (someConditionIsMet):
print("PRINT #3: Exiting")
end_process_and_exit_here()
So basically, when I ran the parentFunction(), I would go into a neverending loop where ever 10 seconds, my console would print "PRINT #2: Running in child function". When the someConditionIsMet was true, my console would print "PRINT #3: Exiting" but then it wouldn't exit. Hence, my loop would carry on forever. I am not sure if it's relevant, but parts of the code has a Threading.Lock as well.
Where I have written end_process_and_exit_here() above, I tried using several methods to kill a thread such as
Raising exceptions and setting flags - These assume that I have started my thread outside of my loop so it's not comparable.
Even this qn about looping threads assumes the thread isnt being looped
Killing using join or stop - stop() was not an option I could access. join() was available but it didn't work i.e. after it was called, the next thread (PRINT #2) continued printing.
Other answers suggesting the use of signals (1) (2), also didn't work.
Using sys.exit() or break in different parts of my code also did not result in the threads stopping.
Is there any method for me to easily exit from such a looping thread?
Note: I need to use threading and not multiprocessing.
You could use python-worker, simply add #worker above you function
pip install python-worker
from worker import worker
#worker
def anotherFunction2(var1,var2):
# your code here
pass
#worker
def parentFunction():
# Other code
while True:
var1, var2 = anotherFunction1() # Getting client details after listening on open port
function2Worker = anotherFunction2(var1,var2) # this will automatically run as thread since you put #worker above your function
childFunction(var1,var2)
print("PRINT #1: Running in Parent Function") # This only prints once for some reason
def childFunction(var1, var2):
parentWorker = parentFunction(var1, var2)
# Other code
if (someConditionIsMet):
parentWorker.abort()
So as an update, I have managed to resolve this issue. The problem with the other answer stated by me (shown below) is that just .cancel() by itself only seemed to work for one timer thread. But as can be seen in the problem, childFunction() itself calls childFunction() and can also be called by the parentFunction, meaning that there may be multiple timer threads.
What worked for my specific case was naming my threads as below:
t1 = threading.Timer(10, childFunction, args=(var1,var2,number))
t1.name = t1.name + "_timer" + str(number)
t1.start()
Thereafter, I could cancel all timer threads that were created from this process by:
for timerthread in threading.enumerate():
if timerthread.name.endswith('timer' + str(number)):
timerthread.cancel()
Below is the ORIGINAL METHOD I USED WHICH CAUSED MANY ISSUES:
I'm not certain if this is a bad practice (in fact I feel it may be based on the answers linked in the question saying that we should never 'kill a thread'). I'm sure there are reasons why this is not good and I'd appreciate anyone telling me why. However, the solution that ultimately worked for me was to use .cancel().
So first change would be to assign your thread Timer to a variable instead of calling it directly. So instead of threading.Timer(10, childFunction, args=(var1,var2)).start(), it should be
t = threading.Timer(10, childFunction, args=(var1,var2))
t.start()
Following that, instead of end_process_and_exit_here(), you should use t.cancel(). This seems to work and stops all threads mid-process. However, the bad thing is that it doesn't seem to carry on with other parts of the program.
I'm trying to make a program that does several timer-related things in python and I need to make it so Asyncio creates a task (without waiting for it) by calling another function with asyncio.get_event_loop().create_task(timer_function(my_parameters)), I've used this before in another project and it worked just fine, however, in this case, it ends up not calling timer_function() like it should and I suspect that it happens because it's inside loops or something related to the project structure. I could not find anything that worked as for now, only using await managed to call the function, but that ended up not making it run in parallel. The project structure is as follows:
async def timer_function(my_parameters):
print('Timer_Function called')
# Do stuff with the parameters
asyncio.sleep(time_based_on_those_parameters)
# Finish doing some other things
# Note: final() doesn't need to be async, I only made it so
# to try and test some fixes
async def final(parameters):
# Do stuff
while True: # This part loops forever every minute
# Do stuff
for i in range(my_range):
if some_condition_a:
asyncio.get_event_loop().create_task(timer_function(my_parameters))
print('Condition A met')
if some_condition_b:
asyncio.get_event_loop().create_task(timer_function(some_different_parameters)
print('Condition B met')
# Do some other stuff
sleep(60)
Once I run the code, all that gets printed when those conditions are met is
>>> Condition met
but what I expected to see is both
>>> Condition met
>>> Timer function called
I then put await before the create_task part all that gets printed at the time is
>>> Timer function called
And then only when the timer runs out and does what it needs to do is when >>> Condition met
gets printed. Is there a way to change this structure to accomodate Asyncio or something else I could try?
EDIT: I found a workaround using threading instead of asyncio. The code is now like this:
def timer_function(my_parameters): # Sync method now
print('Timer_Function called')
# Do stuff with the parameters
sleep(time_based_on_those_parameters) # No longer asyncio.sleep()
# Finish doing some other things
def final(parameters):
# Do stuff
threads = []
while True: # This part loops forever every minute
# Do stuff
for i in range(my_range):
if some_condition_a:
t = threading.Thread(target=timer_function, args=(my_parameters))
threads.append(t)
t.start()
print('Condition A met')
if some_condition_b:
t = threading.Thread(target=timer_function, args=(my_parameters))
threads.append(t)
t.start()
print('Condition B met')
# Do some other stuff
sleep(60)
This now works as intended, so for me I no longer need to fix this issue, however if anyone knows why Asyncio doesn't do that when in this structure please let me know, as someone may have this same issue in the future. (I checked on another project I made and
asyncio.get_event_loop().create_task(timer_function(my_parameters))
can be called without awaiting, the difference is that in this case it's inside a while True and a for loop, and on that case that worked it was simply called once on an event listener)
I am only starting to work with asyncio but my guess is that create_task is not running anything, you could try run_until_complete instead
My program involves a blocking loop. I'd like to use the a python REPL to modify program state while the program is running.
Context:
Often I have a blocking loop in my python code:
# main.py
import itertools
import time
global_state = 123
if __name__ == "__main__":
print("Starting main loop")
m = 0
for n in itertools.count():
time.sleep(1) # "computation"
global_state += 1
m += 10
print(f"{n=}, {m=}, {global_state=}")
When I run this program at the command line, I get something like this:
$ python -i main.py
Starting main loop
n=0, m=10, global_state=124
n=1, m=20, global_state=125
n=2, m=30, global_state=126
...
The loop will run for hours or days. Sometimes, while the loop is running, I desire to interactively modify some program state. For example, I'd like to set global_state = -1000 or m = 75. But the loop function is blocking...
Approaches to interactive modification of program state
Approach 1: Stopping and restarting the loop
I can use a KeyboardInterrupt to stop the loop. Since I have invoked python with the -i interactive flag, I can then modify the global state at the interactive REPL. I then restart the loop. This is a simple approach, but it has drawbacks:
If the loop modifies the global state (or has any side effects), the global state might be left in an inconsistent state when KeyboardInterrupt is raised; the exception could be raised at any point in the loop's control flow, and handling this exception properly can be tricky, especially if the business logic is complex.
When restarting the loop, the counter n will be reset to zero. Other state variables used by the loop (e.g. m) might also be reset or be in an inconsistent state.
Approach 2: Using threading
I can wrap the loop in a function and use threading to run the function in another thread:
def loop():
print("Starting main loop")
m = 0
for n in itertools.count():
time.sleep(1) # "computation"
global_state += 1
m += 10
print(f"{n=}, {m=}, {global_state=}")
import threading
thread = threading.Thread(target=loop)
thread.start()
When I invoke python -i main.py, I can use the REPL to interact with global state as the loop runs in another thread. The loop function still prints to stdout as before. The drawbacks of this approach are:
I can no longer interact with the loop's local state (e.g. I can't modify m) because the local state is wrapped in a function. This is a major drawback.
Using threading makes the program more fragile; threads can crash, and exceptions raised in threads need to be handled carefully. If the thread blocks or crashes or hangs, it can be hard to recover.
Stopping the loop becomes more difficult, because KeyboardInterrupt is no longer an option (instead I need to "cooperate" with the thread, sending it some signal that it should self-terminate).
Approach 3: Drop into REPL from within the loop
I can cooperate with the loop to drop into the repl:
import os
import code
if __name__ == "__main__":
...
for n in itertools.count():
... # business logic
if os.stat("use_repl.txt").st_size > 0: # file nonempty
code.interact(local=locals())
Here we are using a file to signal to the loop when we want to modify state. If the "use_repl.txt" file is nonempty then loop then calls code.interact to create an interactive Console. This has the added benefit of working even if the loop is wrapped in a function; local variables are loaded into the interactive console.
Drawbacks:
Reading a "use_repl.txt" file at every pass through the loop feels a bit clunky. But how else would one signal (cooperatively) to the loop that it should create a REPL?
The loop gets paused when code.interact is called. This differs from the threading-based solution, where the loop runs continuously. This may be an advantage or a disadvantage, depending on your use-case.
The REPL created by code.interact is not as polished as python's native interactive mode. For example, tab-completion no longer works.
An alternative to calling code.interact would be to call the breakpoint built-in function. Another alternative would be to invoke the break keyword, exiting the loop. One could then modify state and restart the loop (as in approach 1), without having to worry that a KeyboardInterrupt has created an inconsistent state.
The Question(s):
Are there any approaches that I've missed? Each of the approaches considered has significant drawbacks... Are there any best-practices regarding achievement of the desired result?
Debug the program, pause it when you want to inspect/modify state, then resume.
Inspired by xaa's answer:
import pdb
import signal
signal.signal(signal.SIGINT, pdb.Pdb().sigint_handler)
This sets a signal handler catching SIGINT (which would otherwise get translated into KeyboardInterrupt), dropping into pdb when CTRL-C is pressed. Thus CTRL-C can be used to interrupt the loop and inspect or modify state at any point, and execution can be resumed by typing continue at the pdb prompt.
I need to run a (series of) infinite loops that must be able to check an externally set condition to terminate. I thought the threading module would allow that, but my efforts so fare have failed. Here is an example of what I am trying to do:
import threading
class Looping(object):
def __init__(self):
self.isRunning = True
def runForever(self):
while self.isRunning == True:
"do stuff here"
l = Looping()
t = threading.Thread(target = l.runForever())
t.start()
l.isRunning = False
I would have expected t.start to run in a separate thread, with l's attributes still accessible. This is not what happens. I tried the snippet above in the python shell (IPython). Execution of t start immediately after instantiation and it blocks any further input.
There is obviously something I am not getting right about the threading module.
Any suggestion on how to solve the problem?
You are calling runForever too early. Use target = l.runForever without parentheses.
A function call is not evaluated until after its arguments are. When you write runforever(), it calls the function right then, before even creating the thread. By just passing runForever, you pass the function object itself, which the threading apparatus can then call when it is ready. The point is that you don't actually want to call runForever; you just want to tell the threading code that runForever is what it should call later.