I'm trying to make a program that does several timer-related things in python and I need to make it so Asyncio creates a task (without waiting for it) by calling another function with asyncio.get_event_loop().create_task(timer_function(my_parameters)), I've used this before in another project and it worked just fine, however, in this case, it ends up not calling timer_function() like it should and I suspect that it happens because it's inside loops or something related to the project structure. I could not find anything that worked as for now, only using await managed to call the function, but that ended up not making it run in parallel. The project structure is as follows:
async def timer_function(my_parameters):
print('Timer_Function called')
# Do stuff with the parameters
asyncio.sleep(time_based_on_those_parameters)
# Finish doing some other things
# Note: final() doesn't need to be async, I only made it so
# to try and test some fixes
async def final(parameters):
# Do stuff
while True: # This part loops forever every minute
# Do stuff
for i in range(my_range):
if some_condition_a:
asyncio.get_event_loop().create_task(timer_function(my_parameters))
print('Condition A met')
if some_condition_b:
asyncio.get_event_loop().create_task(timer_function(some_different_parameters)
print('Condition B met')
# Do some other stuff
sleep(60)
Once I run the code, all that gets printed when those conditions are met is
>>> Condition met
but what I expected to see is both
>>> Condition met
>>> Timer function called
I then put await before the create_task part all that gets printed at the time is
>>> Timer function called
And then only when the timer runs out and does what it needs to do is when >>> Condition met
gets printed. Is there a way to change this structure to accomodate Asyncio or something else I could try?
EDIT: I found a workaround using threading instead of asyncio. The code is now like this:
def timer_function(my_parameters): # Sync method now
print('Timer_Function called')
# Do stuff with the parameters
sleep(time_based_on_those_parameters) # No longer asyncio.sleep()
# Finish doing some other things
def final(parameters):
# Do stuff
threads = []
while True: # This part loops forever every minute
# Do stuff
for i in range(my_range):
if some_condition_a:
t = threading.Thread(target=timer_function, args=(my_parameters))
threads.append(t)
t.start()
print('Condition A met')
if some_condition_b:
t = threading.Thread(target=timer_function, args=(my_parameters))
threads.append(t)
t.start()
print('Condition B met')
# Do some other stuff
sleep(60)
This now works as intended, so for me I no longer need to fix this issue, however if anyone knows why Asyncio doesn't do that when in this structure please let me know, as someone may have this same issue in the future. (I checked on another project I made and
asyncio.get_event_loop().create_task(timer_function(my_parameters))
can be called without awaiting, the difference is that in this case it's inside a while True and a for loop, and on that case that worked it was simply called once on an event listener)
I am only starting to work with asyncio but my guess is that create_task is not running anything, you could try run_until_complete instead
Related
My project requires me to run a blocking code (from another library), whilst continuing my asyncio while: true loop. The code looks something like this:
async def main():
while True:
session_timeout = aiohttp.ClientTimeout()
async with aiohttp.ClientSession() as session:
// Do async stuffs like session.get and so on
# At a certain point, I have a blocking code that I need to execute
// Blocking_code() starts here. The blocking code needs time to get the return value.
Running blocking_code() is the last thing to do in my main() function.
# My objective is to run the blocking code separately.
# Such that whilst the blocking_code() runs, I would like my loop to start from the beginning again,
# and not having to wait until blocking_code() completes and returns.
# In other words, go back to the top of the while loop.
# Separately, the blocking_code() will continue to run independently, which would eventually complete
# and returns. When it returns, nothing in main() will need the return value. Rather the returned
# result continue to be used in blocking_code()
asyncio.run(main())
I have tried using pool = ThreadPool(processes=1) and thread = pool.apply_async(blocking_code, params). It sort of works if there are things that needs to be done after blocking_code() within main(); but blocking_code() is the last thing in main(), and it would cause the whole while loop to pause, until blocking_code() completes, before starting back from the top.
I don't know if this is possible, and if it is, how it's done; but the ideal scenario is this.
Run main(), then run blocking_code() in its own instance. As if executing another .py file. So once the loops reaches blocking_code() in main(), it triggers the blocking_code.py file, and whilst blocking_code.py script runs, the while loops continues from the top again.
If by the time on the 2nd pass in the while loop, it reaches blocking_code() again and the previous run has not complete; another instance of blocking_code() will run on its own instance, independently.
Does what I say make sense? Is it possible to achieve the desired outcome?
Thank you!
This is possible with threads. So you don't block your main loop, you'll need to wrap your thread in an asyncio task. You can wait for return values once your loop is finished if you need to. You can do this with a combination of asyncio.create_task and asyncio.to_thread
import aiohttp
import asyncio
import time
def blocking_code():
print('Starting blocking code.')
time.sleep(5)
print('Finished blocking code.')
async def main():
blocking_code_tasks = []
while True:
session_timeout = aiohttp.ClientTimeout()
async with aiohttp.ClientSession() as session:
print('Executing GET.')
result = await session.get('https://www.example.com')
blocking_code_task = asyncio.create_task(asyncio.to_thread(blocking_code))
blocking_code_tasks.append(blocking_code_task)
#do something with blocking_code_tasks, wait for them to finish, extract errors, etc.
asyncio.run(main())
The above code runes blocking code in a thread and then puts that into an asyncio task. We then add this to the blocking_code_tasks list to keep track of all the currently running tasks. Later on, you can get the values or errors out with something like asyncio.gather
This seems like a particularly confusing question based on the other similar answers I found on SO. I have code similar to the following:
def parentFunction():
# Other code
while True:
var1, var2 = anotherFunction1() # Getting client details after listening on open port
threading.Thread(target = anotherFunction2, args=(var1, var2)).start()
childFunction(var1,var2)
print("PRINT #1: Running in Parent Function") # This only prints once for some reason
def childFunction(var1, var2):
threading.Timer(10, childFunction, args=(var1,var2)).start()
print("PRINT #2: Running in child function") # Prints every 10 seconds
# Other code
if (someConditionIsMet):
print("PRINT #3: Exiting")
end_process_and_exit_here()
So basically, when I ran the parentFunction(), I would go into a neverending loop where ever 10 seconds, my console would print "PRINT #2: Running in child function". When the someConditionIsMet was true, my console would print "PRINT #3: Exiting" but then it wouldn't exit. Hence, my loop would carry on forever. I am not sure if it's relevant, but parts of the code has a Threading.Lock as well.
Where I have written end_process_and_exit_here() above, I tried using several methods to kill a thread such as
Raising exceptions and setting flags - These assume that I have started my thread outside of my loop so it's not comparable.
Even this qn about looping threads assumes the thread isnt being looped
Killing using join or stop - stop() was not an option I could access. join() was available but it didn't work i.e. after it was called, the next thread (PRINT #2) continued printing.
Other answers suggesting the use of signals (1) (2), also didn't work.
Using sys.exit() or break in different parts of my code also did not result in the threads stopping.
Is there any method for me to easily exit from such a looping thread?
Note: I need to use threading and not multiprocessing.
You could use python-worker, simply add #worker above you function
pip install python-worker
from worker import worker
#worker
def anotherFunction2(var1,var2):
# your code here
pass
#worker
def parentFunction():
# Other code
while True:
var1, var2 = anotherFunction1() # Getting client details after listening on open port
function2Worker = anotherFunction2(var1,var2) # this will automatically run as thread since you put #worker above your function
childFunction(var1,var2)
print("PRINT #1: Running in Parent Function") # This only prints once for some reason
def childFunction(var1, var2):
parentWorker = parentFunction(var1, var2)
# Other code
if (someConditionIsMet):
parentWorker.abort()
So as an update, I have managed to resolve this issue. The problem with the other answer stated by me (shown below) is that just .cancel() by itself only seemed to work for one timer thread. But as can be seen in the problem, childFunction() itself calls childFunction() and can also be called by the parentFunction, meaning that there may be multiple timer threads.
What worked for my specific case was naming my threads as below:
t1 = threading.Timer(10, childFunction, args=(var1,var2,number))
t1.name = t1.name + "_timer" + str(number)
t1.start()
Thereafter, I could cancel all timer threads that were created from this process by:
for timerthread in threading.enumerate():
if timerthread.name.endswith('timer' + str(number)):
timerthread.cancel()
Below is the ORIGINAL METHOD I USED WHICH CAUSED MANY ISSUES:
I'm not certain if this is a bad practice (in fact I feel it may be based on the answers linked in the question saying that we should never 'kill a thread'). I'm sure there are reasons why this is not good and I'd appreciate anyone telling me why. However, the solution that ultimately worked for me was to use .cancel().
So first change would be to assign your thread Timer to a variable instead of calling it directly. So instead of threading.Timer(10, childFunction, args=(var1,var2)).start(), it should be
t = threading.Timer(10, childFunction, args=(var1,var2))
t.start()
Following that, instead of end_process_and_exit_here(), you should use t.cancel(). This seems to work and stops all threads mid-process. However, the bad thing is that it doesn't seem to carry on with other parts of the program.
I am pretty new to python, and while using a module to print out packets being received I can't execute code while the while loop that reads the packets is being executed. Here is a basic example. Any help would be appreciated.
def foo():
while True:
print("bar")
foo()
print("foobar")
i want it to print foobar once after the while loop has stared, is this possible?
Typically in Python (and most other languages), you start with just one thread of execution.
A while True: ... is an infinite loop – unless code inside the loop breaks out, or something external interrupts it, it never ends. So your code never reaches the call to print('foobar') line.
You could put a special case inside the while loop, for the first pass through, that reports what you want. Or you could look into using multiple threads of execution – an advanced topic.
The program executes sequentially, so the print will never happen because of the infinite loop. So you must use a thread to circumvent this issue, allowing you to simultaneously execute code like so:
threading.Thread(target = foo).start() # new thread instead of foo() in the main thread
I need to run a (series of) infinite loops that must be able to check an externally set condition to terminate. I thought the threading module would allow that, but my efforts so fare have failed. Here is an example of what I am trying to do:
import threading
class Looping(object):
def __init__(self):
self.isRunning = True
def runForever(self):
while self.isRunning == True:
"do stuff here"
l = Looping()
t = threading.Thread(target = l.runForever())
t.start()
l.isRunning = False
I would have expected t.start to run in a separate thread, with l's attributes still accessible. This is not what happens. I tried the snippet above in the python shell (IPython). Execution of t start immediately after instantiation and it blocks any further input.
There is obviously something I am not getting right about the threading module.
Any suggestion on how to solve the problem?
You are calling runForever too early. Use target = l.runForever without parentheses.
A function call is not evaluated until after its arguments are. When you write runforever(), it calls the function right then, before even creating the thread. By just passing runForever, you pass the function object itself, which the threading apparatus can then call when it is ready. The point is that you don't actually want to call runForever; you just want to tell the threading code that runForever is what it should call later.
I want to execute a function every 60 seconds on Python but I don't want to be blocked meanwhile.
How can I do it asynchronously?
import threading
import time
def f():
print("hello world")
threading.Timer(3, f).start()
if __name__ == '__main__':
f()
time.sleep(20)
With this code, the function f is executed every 3 seconds within the 20 seconds time.time.
At the end it gives an error and I think that it is because the threading.timer has not been canceled.
How can I cancel it?
You could try the threading.Timer class: http://docs.python.org/library/threading.html#timer-objects.
import threading
def f(f_stop):
# do something here ...
if not f_stop.is_set():
# call f() again in 60 seconds
threading.Timer(60, f, [f_stop]).start()
f_stop = threading.Event()
# start calling f now and every 60 sec thereafter
f(f_stop)
# stop the thread when needed
#f_stop.set()
The simplest way is to create a background thread that runs something every 60 seconds. A trivial implementation is:
import time
from threading import Thread
class BackgroundTimer(Thread):
def run(self):
while 1:
time.sleep(60)
# do something
# ... SNIP ...
# Inside your main thread
# ... SNIP ...
timer = BackgroundTimer()
timer.start()
Obviously, if the "do something" takes a long time, then you'll need to accommodate for it in your sleep statement. But, 60 seconds serves as a good approximation.
I googled around and found the Python circuits Framework, which makes it possible to wait
for a particular event.
The .callEvent(self, event, *channels) method of circuits contains a fire and suspend-until-response functionality, the documentation says:
Fire the given event to the specified channels and suspend execution
until it has been dispatched. This method may only be invoked as
argument to a yield on the top execution level of a handler (e.g.
"yield self.callEvent(event)"). It effectively creates and returns
a generator that will be invoked by the main loop until the event has
been dispatched (see :func:circuits.core.handlers.handler).
I hope you find it as useful as I do :)
./regards
It depends on what you actually want to do in the mean time. Threads are the most general and least preferred way of doing it; you should be aware of the issues with threading when you use it: not all (non-Python) code allows access from multiple threads simultaneously, communication between threads should be done using thread-safe datastructures like Queue.Queue, you won't be able to interrupt the thread from outside it, and terminating the program while the thread is still running can lead to a hung interpreter or spurious tracebacks.
Often there's an easier way. If you're doing this in a GUI program, use the GUI library's timer or event functionality. All GUIs have this. Likewise, if you're using another event system, like Twisted or another server-process model, you should be able to hook into the main event loop to cause it to call your function regularly. The non-threading approaches do cause your program to be blocked while the function is pending, but not between functioncalls.
Why dont you create a dedicated thread, in which you put a simple sleeping loop:
#!/usr/bin/env python
import time
while True:
# Your code here
time.sleep(60)
I think the right way to run a thread repeatedly is the next:
import threading
import time
def f():
print("hello world") # your code here
myThread.run()
if __name__ == '__main__':
myThread = threading.Timer(3, f) # timer is set to 3 seconds
myThread.start()
time.sleep(10) # it can be loop or other time consuming code here
if myThread.is_alive():
myThread.cancel()
With this code, the function f is executed every 3 seconds within the 10 seconds time.sleep(10). At the end running of thread is canceled.
If you want to invoke the method "on the clock" (e.g. every hour on the hour), you can integrate the following idea with whichever threading mechanism you choose:
import time
def wait(n):
'''Wait until the next increment of n seconds'''
x = time.time()
time.sleep(n-(x%n))
print(time.asctime())
[snip. removed non async version]
To use asyncing you would use trio. I recommend trio to everyone who asks about async python. It is much easier to work with especially sockets. With sockets I have a nursery with 1 read and 1 write function and the write function writes data from an deque where it is placed by the read function; and waiting to be sent. The following app works by using trio.run(function,parameters) and then opening an nursery where the program functions in loops with an await trio.sleep(60) between each loop to give the rest of the app a chance to run. This will run the program in a single processes but your machine can handle 1500 TCP connections insead of just 255 with the non async method.
I have not yet mastered the cancellation statements but I put at move_on_after(70) which is means the code will wait 10 seconds longer than to execute a 60 second sleep before moving on to the next loop.
import trio
async def execTimer():
'''This function gets executed in a nursery simultaneously with the rest of the program'''
while True:
trio.move_on_after(70):
await trio.sleep(60)
print('60 Second Loop')
async def OneTime_OneMinute():
'''This functions gets run by trio.run to start the entire program'''
with trio.open_nursery() as nursery:
nursery.start_soon(execTimer)
nursery.start_soon(print,'do the rest of the program simultaneously')
def start():
'''You many have only one trio.run in the entire application'''
trio.run(OneTime_OneMinute)
if __name__ == '__main__':
start()
This will run any number of functions simultaneously in the nursery. You can use any of the cancellable statements for checkpoints where the rest of the program gets to continue running. All trio statements are checkpoints so use them a lot. I did not test this app; so if there are any questions just ask.
As you can see trio is the champion of easy-to-use functionality. It is based on using functions instead of objects but you can use objects if you wish.
Read more at:
[1]: https://trio.readthedocs.io/en/stable/reference-core.html