I want to run a timer in a tornado based web app such that the it runs in background and is non blocking.
Once the timer finishes, a specific task has to be called so it is very important that the timer completes exactly on time.
What should be the ideal way to do it ?
I read Tornado IOLoop.spawn_callback in the documentation but I am not very clear that it would behave correctly.
I don't quite understand the statement in the doc
Unlike all other callback-related methods on IOLoop, spawn_callback does not associate the callback with its caller’s stack_context
If you want to run a function after a specific time, you can use IOLoop.call_later. Use it like this:
IOLoop.current().call_later(5, my_func) # will call my_func 5 seconds later
def my_func():
# do something
IOLoop.spawn_callback is used for running a callback/function in the next iteration of the IOLoop, that is - almost instantly. You can't add a time out to spawn_callback. Since you want to schedule a callback after a timeout, IOLoop.call_later is what you need.
In your comment you asked
Why according to you IOLoop.spawn_callback is not to be used?
Well, I never said to not use it. You can use it if you need it. In this case, you don't.
So, when do you need it? When you'll need to run a callback almost instantly, without a timeout, that's when you can use spawn_callback. But even then, there's IOLoop.add_callback which is used much more widely than spawn_callback.
Related
I have a python app that calls a recursive method which runs forever. It is a loop that scrapes a webpage and looks for a number and once it finds it, it prints out a message, increments the number, and calls the method again with the incremented number. This goes on forever because the webpage updates itself about once a week and my method prints out the message when that update is caught.
I want to make a mobile app that notifies users when the method prints out a message (ideally within a minute or two of the change). What is the best way to create an api that would allow me to do this? If there is another way, how can i do it?
Using recursive method for infinite loop is a big mistake because every time you call method again the last method goes to stack and if you do it infinite time finally you get stack overflow error. best way for infinite jobs are thread with a simple "while True":
import threading
SenderThread = threading.Thread(target=sender)
SenderThread.start()
def sender():
while True:
# do your job here
edit:
according to this:
Python threads are used in cases where the execution of a task involves some waiting. One example would be interaction with a service hosted on another computer, such as a webserver. Threading allows python to execute other code while waiting; this is easily simulated with the sleep function.
The reason i used thread is for the main program to do its job or respose to inputs or any thing else that you need.
How would you be able to move to the next iteration of a for loop if a given iteration takes more than a certain amount of time? The code should look something like this.
for i in range(0, max_iterations):
timer function
call to api
The timer function will serve the purpose of forcing the for loop to continue onto the next iteration if the api has not finished. It should work in 120 seconds for that iteration. How would the timer function be written? Thank you in advance!
This is only truly possible with a non-blocking API call or an API call with a timeout. For example, if you are using the socket library, you could use socket.setblocking(0)to make the socket API calls non-blocking.
In your case, you have said you are using the Yandex API. This appears to be JSON over https, so you may wish to try urllib2.urlopen(). This method accepts a timeout. This is even easier than using a non-blocking call as urlopen() will simply give up and return an error after the timeout has expired.
Using threads as suggested in some of the comments will give you a partial solution. Since there is no ability to stop a thread started with the threading module, all of the API calls you initiate that do not complete will stay in a blocked state for the life of the python interpreter and those threads will never exit.
If you do use the threading module to solve this problem, you should make all of the threads that run API calls daemon threads thread.setDaemon(True) so that when your main thread exits, the interpreter stops. Otherwise the interpreter will not exit until all of the API calls have completed and returned.
I've added a module to mopidy core that uses gobject.timeout_add() for a repeating function. It works fine when running normally, however when running tests it appears that the handler function never gets called.
The module under test has a method that starts a regular process that emits events roughly every half a second. The test calls the method, then call time.sleep(2). The events from the timer function don't occur, and neither does some debug logging in the timer function. Other events and debug logging (outside the timer function) work fine.
What do I need to do to get gobject.timeout_add() to work in nose tests? Or do I need to use something other than time.sleep() in the test in order to allow the other code to run? It's calling gobject.threads_init() in the test setup, is there anything else I need to do?
You need to be running an event loop. As the documentation for g_timeout_add explains, that function (and other similar functions in glib) will create a new timeout source which is then attached to the event loop (well, the GMainContext, but you don't really need to worry about that). It is not the equivalent of spawning a new thread and having it sleep for whatever you specified as the duration of the timeout, which seems to be the behavior you're expecting—using the main loop allows everything to happen in a single thread.
So I have this library that I use and within one of my functions I call a function from that library, which happens to take a really long time. Now, at the same time I have another thread running where I check for different conditions, what I want is that if a condition is met, I want to cancel the execution of the library function.
Right now I'm checking the conditions at the start of the function, but if the conditions happen to change while the library function is running, I don't need its results, and want to return from it.
Basically this is what I have now.
def my_function():
if condition_checker.condition_met():
return
library.long_running_function()
Is there a way to run the condition check every second or so and return from my_function when the condition is met?
I've thought about decorators, coroutines, I'm using 2.7 but if this can only be done in 3.x I'd consider switching, it's just that I can't figure out how.
You cannot terminate a thread. Either the library supports cancellation by design, where it internally would have to check for a condition every once in a while to abort if requested, or you have to wait for it to finish.
What you can do is call the library in a subprocess rather than a thread, since processes can be terminated through signals. Python's multiprocessing module provides a threading-like API for spawning forks and handling IPC, including synchronization.
Or spawn a separate subprocess via subprocess.Popen if forking is too heavy on your resources (e.g. memory footprint through copying of the parent process).
I can't think of any other way, unfortunately.
Generally, I think you want to run your long_running_function in a separate thread, and have it occasionally report its information to the main thread.
This post gives a similar example within a wxpython program.
Presuming you are doing this outside of wxpython, you should be able to replace the wx.CallAfter and wx.Publisher with threading.Thread and PubSub.
It would look something like this:
import threading
import time
def myfunction():
# subscribe to the long_running_function
while True:
# subscribe to the long_running_function and get the published data
if condition_met:
# publish a stop command
break
time.sleep(1)
def long_running_function():
for loop in loops:
# subscribe to main thread and check for stop command, if so, break
# do an iteration
# publish some data
threading.Thread(group=None, target=long_running_function, args=()) # launches your long_running_function but doesn't block flow
myfunction()
I haven't used pubsub a ton so I can't quickly whip up the code but it should get you there.
As an alternative, do you know the stop criteria before you launch the long_running_function? If so, you can just pass it as an argument and check whether it is met internally.
First method:
import threading
import time
def keepalive():
while True:
print 'Alive.'
time.sleep(200)
threading.Thread(target=keepalive).start()
Second method:
import threading
def keepalive():
print 'Alive.'
threading.Timer(200, keepalive).start()
threading.Timer(200, keepalive).start()
Which method takes up more RAM? And in the second method, does the thread end after being activated? or does it remain in the memory and start a new thread? (multiple threads)
Timer creates a new thread object for each started timer, so it certainly needs more resources when creating and garbage collecting these objects.
As each thread exits immediately after it spawned another active_count stays constant, but there are constantly new Threads created and destroyed, which causes overhead. I'd say the first method is definitely better.
Altough you won't realy see much difference, only if the interval is very small.
Here's an example of how to test this yourself:
And in the second method, does the thread end after being activated? or does it remain in the memory and start a new thread? (multiple threads)
import threading
def keepalive():
print 'Alive.'
threading.Timer(200, keepalive).start()
print threading.active_count()
threading.Timer(200, keepalive).start()
I also changed the 200 to .2 so it wouldn't take as long.
The thread count was 3 forever.
Then I did this:
top -pid 24767
The #TH column never changed.
So, there's your answer: We don't have enough info to know whether Python maintains a single timer thread for all of the timers, or ends and cleans up the thread as soon as the timer runs, but we can be sure the threads doesn't stick around and pile up. (If you do want to know which of the former is happening, you can, e.g., print the thread ids.)
An alternative way to find out is to look at the source. As the documentation says, "Timer is a subclass of Thread and as such also functions as an example of creating custom threads". The fact that it's a subclass of Thread already tells you that each Timer is a Thread. And the fact that it "functions as an example" implies that it ought to be easy to read. If you click the link form the documentation to the source, you can see how trivial it is. Most of the work is done by Event, but that's in the same source file, and it's almost as simple. Effectively, it just creates a condition variable, waits on it (so it blocks until it times out, or you notify the condition by calling cancel), then quits.
The reason I'm answering one sub-question and explaining how I did it, rather than answering each sub-question, is because I think it would be more useful for you to walk through the same steps.
On further reflection, this probably isn't a question to be decided by optimization in the first place:
If you have a simple, synchronous program that needs to do nothing for 200 seconds, make a blocking call to sleep. Or, even simpler, just do the job and quit, and pick an external tool to schedule your script to run every 200s.
On the other hand, if your program is inherently asynchronous—especially if you've already got thread, signal handlers, and/or an event loop—there's just no way you're going to get sleep to work. If Timer is too inefficient, go to PyPI or ActiveState and find a better timer that lets you schedule repeatable timers (or even multiple timers) with a single instance and thread. (Or, if you're using signals, use signal.alarm or setitimer, and if you're using an event loop, build the timer into your main loop.)
I can't think of any use case where sleep and Timer would both be serious contenders.