I have a list of functions (1..N) and function i needs to be called every X_i seconds (X_i would be large such as 1000+ s). Each X_i doesn't have to be unique, i.e. it is possible that X_i == X_j.
Provided, I generate a list of (function_i, X_i) how can I simply execute these functions at their appropriate times in the future and sleep between calls? I have used ApScheduler before but it runs tasks in parallel and I need functions to be run one after the other.
I can write my own iterator which returns the current function that needs to be executed and blocks until the next one but I'd rather use a library if one exists?
EDIT: N is about 200 at the moment.
threading module
The threading module lets you start a new thread, which will not be affected by other threads' sleep statements. This requires N threads, so if N is extremely huge, let me know and I will try to think of an alternative solution.
You can create N threads and set each one on a timed loop, like so:
import threading, time
def looper(function, delay): # Creates a function that will loop that function
def inner(): # Will keep looping once invoked
while True:
function() # Call the function; you can optionally add args
time.sleep(delay) # Swap this line and the one before it to wait before running rather than after
return inner # The function that should be called to start the loop is returned
def start(functions, delays): # Call this with the two lists to start the loops
for function, delay in zip(functions, delays): # Goes through the respective pairs
thread = threading.Thread(target = looper(function, delay)) # This thread will start the looper
thread.start()
start([lambda: print("hi"), lambda: print("bye")], [0.2, 0.3])
You can try it online here; just hit run and then hit run again when you want to kill it (Thanks to #DennisMitchell for the online interpreter)
Related
In Python, I am making a cube game (like Minecraft pre-classic) that renders chunk by chunk (16x16 blocks). It only renders blocks that are not exposed (not covered on all sides). Even though this method is fast when I have little height (like 16x16x2, which is 512 blocks in total), once I make the terrain higher (like 16x16x64, which is 16384 blocks in total), rendering each chunk takes roughly 0.03 seconds, meaning that when I render multiple chunks at once the game freezes for about a quarter of a second. I want to render the chunks "asynchronously", meaning that the program will keep on drawing frames and calling the chunk render function multiple times, no matter how long it takes. Let me show you some pictures to help:
I tried to make another program in order to test it:
import threading
def run():
n=1
for i in range(10000000):
n += 1
print(n)
print("Start")
threading.Thread(target=run()).start()
print("End")
I know that creating such a lot of threads is not the best solution, but nothing else worked.
Threading, however, didn't work, as this is what the output looked like:
>>> Start
>>> 10000001
>>> End
It also took about a quarter of a second to complete, which is about how long the multiple chunk rendering takes.
Then I tried to use async:
import asyncio
async def run():
n = 1
for i in range(10000000):
n += 1
print(n)
print("Start")
asyncio.run(run())
print("End")
It did the exact same thing.
My questions are:
Can I run a function without stopping/pausing the program execution until it's complete?
Did I use the above correctly?
Yes. No. The answer is complicated.
First, your example has at least one error on it:
print("Start")
threading.Thread(target=run).start() #notice the missing parenthesis after run
print("End")
You can use multithreading for your game of course, but it can come at a disadvantage of code complexity because of synchronization and you might not gain any performance because of GIL.
asyncio is probably not for this job either, since you don't need to highly parallelize many tasks and it has the same problems with GIL as multithreading.
The usual solution for this kind of problem is to divide your work into small batches and only process the next batch if you have time to do so on the same frame, kind of like so:
def runBatch(range):
for x in range:
print(x)
batches = [range (x, x+200) for x in range(0, 10000, 200)]
while (true): # main loop
while (timeToNextFrame() > 15):
runBatch(batch.pop())
renderFrame() #or whatever
However, in this instance, optimizing the algorithm itself could be even better than any other option. One thing that Minecraft does is it subdivides chunks into subchunks (you can mostly ignore subchunks that are full of blocks). Another is that it only considers the visible surfaces of the blocks (renders only those sides of the block that could be visible, not the whole block).
asyncio only works asynchronously only when your function is waiting on I/O task like network call or wait on disk I/O etc.
for non I/O tasks to execute asynchronously multi-threading is the only option so create all your threads and wait for the threads to complete their tasks using thread join method
from threading import Thread
import time
def draw_pixels(arg):
time.sleep(arg)
print(arg)
threads = []
args = [1,2,3,4,5]
for arg in args:
t = Thread(target=draw_pixels, args=(arg, ))
t.start()
threads.append(t)
# join all threads
for t in threads:
t.join()
I have a function in thread A which needs to wait until a function in thread B is called.
The function in thread B is called periodically, so it just needs to wait until the next time it is called. This allows me to sync up with it.
How would I do this?
(Sorry if this is trivial.)
It may be a principle of computer science that no multithreading question is trivial.
There are various ways to do this, but one of the simplest involves the use of a threading.Event object. Events are the simplest of the so-called synchronization primitives. See the manual section on the threading module for more ideas. Here is a working example:
#! python3.8
import threading
import time
t0 = time.time()
def elapsed_time():
return time.time() - t0
class StopMe:
def __init__(self):
self.running = True
def main():
ev1 = threading.Event()
stop = StopMe()
th1 = threading.Thread(target=thread1, args=(ev1, stop))
th1.start()
for _ in range(10):
ev1.wait()
print("The function was just called", elapsed_time())
ev1.clear()
stop.running = False
th1.join()
print("Exit", elapsed_time())
def thread1(event, stop):
def a_function():
event.set()
print("I am the function", elapsed_time())
while stop.running:
time.sleep(1.0)
a_function()
main()
Output:
I am the function 1.0116908550262451
The function was just called 1.0116908550262451
I am the function 2.0219264030456543
The function was just called 2.0219264030456543
I am the function 3.0322916507720947
The function was just called 3.0322916507720947
I am the function 4.033170938491821
The function was just called 4.033170938491821
I am the function 5.043376445770264
The function was just called 5.043376445770264
I am the function 6.043909788131714
The function was just called 6.043909788131714
I am the function 7.054021596908569
The function was just called 7.054021596908569
I am the function 8.06399941444397
The function was just called 8.06399941444397
I am the function 9.064924716949463
The function was just called 9.064924716949463
I am the function 10.066757678985596
The function was just called 10.066757678985596
I am the function 11.076870918273926
Exit 11.076870918273926
Some things to note here:
Once you put a synchronization primitive into your code, you need to give some thought about how to terminate the thread gracefully, and how to terminate the application as a whole. In this example, the threads communicate through the little "StopMe" object, and through the Event object. Note that the main thread may have to wait one second until the secondary thread finishes its sleep function. That occurs if thread1 begins its time delay before the main thread calls the join function. That didn't happen in my test run but it might, depending on how CPU time slices are given to the different threads. If that's not acceptable to you, you have to write more code to get around it.
Also note that the function call ev1.wait() will block the main thread until the event is set from the secondary thread. In a GUI application that is not what you want.
I ran this with Python3.8 but the program doesn't use any version-specific features, so it should work the same with any reasonably recent version of Python.
So I have a Python code running with one very expensive function that gets executed at times on demand, but it's result is not needed straight away (it can be delayed by a few cycles).
def heavy_function(arguments):
return calc_obtained_from_arguments
def main():
a = None
if some_condition:
a = heavy_function(x)
else:
do_something_with(a)
The thing is that whenever I calculate the heavy_function, the rest of the program hangs. However, I need it to run with empty a value, or better make it know that a is being processed separately and thus should not be accessed. How can I move the heavy_function to separate process and keep calling the main function all the time until heavy_function is done executing, then read the obtained a value and use it in main function?
You could use a simple queue.
Put your heavy_function inside a separate process that idles as long as there is no input in the input queue. Use Queue.get(block=True) to do so. Put the result of the computation in another queue.
Run your normal process with the empty a-value and check emptiness of the output queue from time to time. Maybe use while Queue.empty(): here.
If an item becomes available, because your heavy_functionhas finished, switch to a calculation with the value a from your output queue.
I have a situation, where at some point in my code I want to trigger a number of timers, the code will keep running, but at some point these functions will trigger and remove an item from a given list. Similar though not exactly like the code below. The problem is, I want these functions to wait a certain amount of time, the only way I know how is to use sleep, but that stops all of the code, when I need the first function to keep running. So how can I set a function aside with out making everything wait for it? If the answer involves threading, please know that I have very little experience with it and like explanations with pictures and small words.
from time import sleep
from datetime import datetime
def func():
x = 1
for i in range(20):
if i % 4 == 0:
func2()
print("START", datetime.now())
x += 1
else:
print("continue")
def func2():
print("go")
sleep(10)
print("func 2--------------------------------------", datetime.now())
func()
You need to use threading. http://docs.python.org/2/library/threading.html
You can start functions in their own threads.
I used background function. It will run in the background, even if going to another page.
You need to import threading, also time to use time.sleep():
import threading
import time
I had a function where I wanted to sleep code in the background, here is an example:
# This is the one that will sleep, but since you used args on the Thread, it will not make the mainFunction to sleep.
def backgroundFunction(obj):
theObj = obj
time.sleep(120)
# updates the Food to 5 in 2 minutes
obj["Food"] = 5
return
def mainFunction():
obj = {"Food": 4, "Water": 3}
# Make sure there are a comma in the args().
t1 = threading.Thread(target=backgroundFunction, args=(obj,))
t1.start()
return
If you used t1 = threading.Thread(target=backgroundFunction(obj)) it will not be in the background so don't use this, unless you want mainFunction to sleep also.
Depending on the situation, another option might be an event queue based system. That avoids threads, so it can be simpler.
The idea is that instead of using sleep(20), you calculate when the event should fire, using datetime.now() + timedelta(seconds=20). You then put that in a sorted list.
Regularly, perhaps each time through the main loop of your program, you check the first element in the list; if the time has passed, you remove it and call the relevant function.
To add an event:
pending_events.append((datetime.now() + timedelta(seconds=20), e))
pending_events.sort()
Then, as part of your main loop:
for ... # your main loop
# handle timed events:
while pending_events[0][0] < datetime.now():
the_time, e = pending_events.pop(0)
handle_event(e, the_time)
... # rest of your main loop
This relies on your main loop regularly calling the event-handling code, and on the event-handling code not taking much time to handle the event. Depending on what the main loop and the events are doing, this may come naturally or it may be some effort or it may rule out this method...
Notes:
You only need to check the first element in the list, because the list is sorted in time order; checking the first element checks the earliest one and you don't need to check the others until that one has passed.
Instead of a sorted list, you can use a heapq, which is more complicated but faster; in practice, you'd need a lot of pending events to notice any difference.
If the event is to be "every 20s" rather than "after 20s", use the_time + timedelta(seconds=20) to schedule each subsequent event; that way, the delay in getting to and processing the event won't be added.
I have two different functions f, and g that compute the same result with different algorithms. Sometimes one or the other takes a long time while the other terminates quickly. I want to create a new function that runs each simultaneously and then returns the result from the first that finishes.
I want to create that function with a higher order function
h = firstresult(f, g)
What is the best way to accomplish this in Python?
I suspect that the solution involves threading. I'd like to avoid discussion of the GIL.
I would simply use a Queue for this. Start the threads and the first one which has a result ready writes to the queue.
Code
from threading import Thread
from time import sleep
from Queue import Queue
def firstresult(*functions):
queue = Queue()
threads = []
for f in functions:
def thread_main():
queue.put(f())
thread = Thread(target=thread_main)
threads.append(thread)
thread.start()
result = queue.get()
return result
def slow():
sleep(1)
return 42
def fast():
return 0
if __name__ == '__main__':
print firstresult(slow, fast)
Live demo
http://ideone.com/jzzZX2
Notes
Stopping the threads is an entirely different topic. For this you need to add some state variable to the threads which needs to be checked in regular intervals. As I want to keep this example short I simply assumed that part and assumed that all workers get the time to finish their work even though the result is never read.
Skipping the discussion about the Gil as requested by the questioner. ;-)
Now - unlike my suggestion on the other answer, this piece of code does exactly what you are requesting:
from multiprocessing import Process, Queue
import random
import time
def firstresult(func1, func2):
queue = Queue()
proc1 = Process(target=func1,args=(queue,))
proc2 = Process(target=func2, args=(queue,))
proc1.start();proc2.start()
result = queue.get()
proc1.terminate(); proc2.terminate()
return result
def algo1(queue):
time.sleep(random.uniform(0,1))
queue.put("algo 1")
def algo2(queue):
time.sleep(random.uniform(0,1))
queue.put("algo 2")
print firstresult(algo1, algo2)
Run each function in a new worker thread, the 2 worker threads send the result back to the main thread in a 1 item queue or something similar. When the main thread receives the result from the winner, it kills (do python threads support kill yet? lol.) both worker threads to avoid wasting time (one function may take hours while the other only takes a second).
Replace the word thread with process if you want.
You will need to run each function in another process (with multiprocessing) or in a different thread.
If both are CPU bound, multithread won help much - exactly due to the GIL -
so multiprocessing is the way.
If the return value is a pickleable (serializable) object, I have this decorator I created that simply runs the function in background, in another process:
https://bitbucket.org/jsbueno/lelo/src
It is not exactly what you want - as both are non-blocking and start executing right away. The tirck with this decorator is that it blocks (and waits for the function to complete) as when you try to use the return value.
But on the other hand - it is just a decorator that does all the work.