Here's some code that illustrates my problem:
def blocking1():
while True:
yield 'first blocking function example'
def blocking2():
while True:
yield 'second blocking function example'
for i in blocking1():
print 'this will be shown'
for i in blocking2():
print 'this will not be shown'
I have two functions which contain while True loops. These will yield data which I will then log somewhere (most likely, to an sqlite database).
I've been playing around with threading and have gotten it working. However, I don't really like it... What I would like to do is make my blocking functions asynchronous. Something like:
def blocking1(callback):
while True:
callback('first blocking function example')
def blocking2(callback):
while True:
callback('second blocking function example')
def log(data):
print data
blocking1(log)
blocking2(log)
How can I achieve this in Python? I've seen the standard library comes with asyncore and the big name in this game is Twisted but both of these seem to be used for socket IO.
How can I async my non-socket related, blocking functions?
A blocking function is a function which doesn't return, but still leaves your process idle - unable to complete more work.
You're asking us to make your blocking functions non-blocking. However – unless you're writing an operating system – you don't have any blocking functions. You might have functions which block because they make calls to blocking system calls, or you might have functions which "block" because they do a lot of computation.
Making the former type of function non-blocking is impossible without making the underlying system call non-blocking. Depending on what that system call is, it may be difficult to make it non-blocking without also adding an event loop to your program; you don't just need to make the call and have it not block, you also have to make another call to determine that the result of that call will be delivered somewhere you could associate it.
The answer to this question is a very long python program and a lot of explanations of different OS interfaces and how they work, but luckily I already wrote that answer on a different site; I called it Twisted. If your particular task is already supported by a Twisted reactor, then you're in luck. Otherwise, as long as your task maps to some existing operating system concept, you can extend a reactor to support it. Practically speaking there are only 2 of these mechanisms: file descriptors on every sensible operating system ever, and I/O Completion Ports on Windows.
In the other case, if your functions are consuming a lot of CPU, and therefore not returning, they're not really blocking; your process is still chugging along and getting work done. There are three ways to deal with that:
separate threads
separate processes
if you have an event loop, write a task that periodically yields, by writing the task in such a way that it does some work, then asks the event loop to resume it in the near future in order to allow other tasks to run.
In Twisted this last technique can be accomplished in various ways, but here's a syntactically convenient trick that makes it easy:
from twisted.internet import reactor
from twisted.internet.task import deferLater
from twisted.internet.defer import inlineCallbacks, returnValue
#inlineCallbacks
def slowButSteady():
result = SomeResult()
for something in somethingElse:
result.workHardForAMoment(something)
yield deferLater(reactor, 0, lambda : None)
returnValue(result)
You can use generators for cooperative multitasking, but you have to write your own main loop that passes control between them.
Here's a (very simple) example using your example above:
def blocking1():
while True:
yield 'first blocking function example'
def blocking2():
while True:
yield 'second blocking function example'
tasks = [blocking1(), blocking2()]
# Repeat until all tasks have stopped
while tasks:
# Iterate through all current tasks. Use
# tasks[:] to copy the list because we
# might mutate it.
for t in tasks[:]:
try:
print t.next()
except StopIteration:
# If the generator stops, remove it from the task list
tasks.remove(t)
You could further improve it by allowing the generators to yield new generators, which then could be added to tasks, but hopefully this simplified example will give the general idea.
The twisted framework is not just sockets. It has asynchronous adapters for many scenarios, including interacting with subprocesses. I recommend taking a closer look at that. It does what you are trying to do.
If you don't want to use full OS threading, you might try Stackless, which is a variant of Python that adds many interesting features, including "microthreads". There are a number of good examples that you will find helpful.
Your code isn’t blocking. blocking1() and it’s brother return iterators immediately (not blocking), and neither does a single iteration block (in your case).
If you want to “eat” from both iterators one-by-one, don’t make your program try to eat up “blocking1()” entirely, before continuing...
for b1, b2 in zip(blocking1(), blocking2()):
print 'this will be shown', b1, 'and this, too', b2
Related
Suppose there's a synchronous function in a twisted-powered Python program that takes a long time to execute, doing that in a lot of reasonable-sized pieces of work. If the function could return deferreds, this would be a no-brainer, however the function happens to be deep inside some synchronous code, so that yielding deferreds to continue is impossible.
Is there a way to let twisted handle outstanding events without leaving that function? I.e. what I want to do is something along the lines of
def my_func():
results = []
for item in a_lot_of_items():
results.append(do_computation(item))
reactor.process_outstanding_events()
return results
Of course, this imposes reentrancy requirements on the code, but still, there's QCoreApplication.processEvents for that in Qt, is there anything in twisted?
The solution taken by some event-loop-based systems (essentially the solution you're referencing via Qt's QCoreApplication.processEvents API) is to make the main loop re-entrant. In Twisted terms, this would mean something like (not working code):
def my_expensive_task_that_cannot_be_asynchronous():
#inlineCallbacks
def do_work(units):
for unit in units:
yield do_one_work_asynchronously(unit)
work = do_work(some_work_units())
work.addBoth(lambda ignored: reactor.stop())
reactor.run()
def main():
# Whatever your setup is...
# Then, hypothetical event source triggering your
# expensive function:
reactor.callLater(
30,
my_expensive_task_that_cannot_be_asynchronous,
)
reactor.run()
Notice how there are two reactor.run calls in this program. If Twisted had a re-entrant event loop, this second call would start spinning the reactor again and not return until a matching reactor.stop call is encountered. The reactor would process all events it knows about, not just the ones generated by do_work, and so you would have the behavior you desire.
This requires a re-entrant event loop because my_expensive_task_... is already being called by the reactor loop. The reactor loop is on the call stack. Then, reactor.run is called and the reactor loop is now on the call stack again. So the usual issues apply: the event loop cannot have left over state in its frame (otherwise it may be invalid by the time the nested call is complete), it cannot leave its instance state inconsistent during any calls out to other code, etc.
Twisted does not have a re-entrant event loop. This is a feature that has been considered and, at least in the past, explicitly rejected. Supporting this features brings a huge amount of additional complexity (described above) to the implementation and the application. If the event loop is re-entrant then it becomes very difficult to avoid requiring all application code to be re-entrant safe as well. This negates one of the major benefits of the cooperative multitasking approach Twisted takes to concurrency (that you are guaranteed your functions will not be re-entered).
So, when using Twisted, this solution is out.
I'm not aware of another solution which would allow you to continue to run this code in the reactor thread. You mentioned that the code in question is nested deeply within some other synchronous code. The other options that come to mind are:
make the synchronous code capable of dealing with asynchronous things
factor the expensive parts out and compute them first, then pass the result in to the rest of the code
run all of that code, not just the computationally expensive part, in another thread
You could use deferToThread.
http://twistedmatrix.com/documents/13.2.0/core/howto/threading.html
That method runs your calculation in a separate thread and returns a deferred that is called back when the calculation is actually finished.
The issue is if do_heavy_computation() is code that blocks then execution won't go to the next function. In this case use deferToThread or blockingCallFromThread for heavy calculations. Also if you don't care for the results of the calculation then you can use callInThread. Take a look at documentation on threads
This should do:
for item in items:
reactor.callLater(0, heavy_func, item)
reactor.callLater should bring you back into the event loop.
So I have this library that I use and within one of my functions I call a function from that library, which happens to take a really long time. Now, at the same time I have another thread running where I check for different conditions, what I want is that if a condition is met, I want to cancel the execution of the library function.
Right now I'm checking the conditions at the start of the function, but if the conditions happen to change while the library function is running, I don't need its results, and want to return from it.
Basically this is what I have now.
def my_function():
if condition_checker.condition_met():
return
library.long_running_function()
Is there a way to run the condition check every second or so and return from my_function when the condition is met?
I've thought about decorators, coroutines, I'm using 2.7 but if this can only be done in 3.x I'd consider switching, it's just that I can't figure out how.
You cannot terminate a thread. Either the library supports cancellation by design, where it internally would have to check for a condition every once in a while to abort if requested, or you have to wait for it to finish.
What you can do is call the library in a subprocess rather than a thread, since processes can be terminated through signals. Python's multiprocessing module provides a threading-like API for spawning forks and handling IPC, including synchronization.
Or spawn a separate subprocess via subprocess.Popen if forking is too heavy on your resources (e.g. memory footprint through copying of the parent process).
I can't think of any other way, unfortunately.
Generally, I think you want to run your long_running_function in a separate thread, and have it occasionally report its information to the main thread.
This post gives a similar example within a wxpython program.
Presuming you are doing this outside of wxpython, you should be able to replace the wx.CallAfter and wx.Publisher with threading.Thread and PubSub.
It would look something like this:
import threading
import time
def myfunction():
# subscribe to the long_running_function
while True:
# subscribe to the long_running_function and get the published data
if condition_met:
# publish a stop command
break
time.sleep(1)
def long_running_function():
for loop in loops:
# subscribe to main thread and check for stop command, if so, break
# do an iteration
# publish some data
threading.Thread(group=None, target=long_running_function, args=()) # launches your long_running_function but doesn't block flow
myfunction()
I haven't used pubsub a ton so I can't quickly whip up the code but it should get you there.
As an alternative, do you know the stop criteria before you launch the long_running_function? If so, you can just pass it as an argument and check whether it is met internally.
I was reading a gevent tutorial and saw this interesting snippet:
import gevent
def foo():
print('Running in foo')
gevent.sleep(0)
print('Explicit context switch to foo again')
def bar():
print('Explicit context to bar')
gevent.sleep(0)
print('Implicit context switch back to bar')
gevent.joinall([
gevent.spawn(foo),
gevent.spawn(bar),
])
In which the flow of execution goes like this foo -> bar -> foo -> bar .
Is it not possible to do the same without the gevent module but with yield statements?
I've been trying to do this with 'yield' but for some reason I can't get it to work... :(
Generators used for this purpose are often called tasks (among many other terms), and I'll use that term here for clarity. Yes, it is possible. There are, in fact, several approaches that work and make sense in some contexts. However, none (that I'm aware of) work without an equivalent for at least one of gevent.spawn and gevent.joinall. The more powerful and well-designed ones require an equivalent for both.
The fundamental problem is this: Generators can be suspended (when they hit yield), but that's it. To kick them off again, you need some other code calling next() on them. In fact, you even need to call next() on a freshly-created generator for it to do anything to begin with.
Similarly, the generator itself isn't the best place to decide what should run next. So you need what is a loop that initiates each tasks's time slice (runs them to the next yield) and switches between them, indefinitely. This is usually called a scheduler. They tend to become really hairy really quickly, so I won't attempt to write a full scheduler in one answer. There are however some core concepts I can try to explain:
One usually treats yield as giving control back to the sheduler (in effect similar to gevent.sleep(0) in your code). That means, the generator does whatever it wants to do, and when it's in a place where a context switch is convenient and possibly useful, it yields.
In Python 3.3+, yield from is a very useful tool to delegate to another generator. If you can't use it, you have to make the scheduler emulate a call stack and route return values to the right place, and do things like result = yield subtasks() in your tasks. This is slower, more complex to implement, and unlikely to yield useful stack traces (yield from does this for free). But until recently, it was the best we had.
Depending on your use case, you may need a wide range of tools to manage tasks. Common examples are spawning more tasks, waiting for a task to complete, waiting for any one of several tasks to complete, detecting failure (uncaught exception) of other tasks, etc. These are usually handled by the scheduler and the tasks are given an API to communicate with the scheduler. A neat (but not always perfect) way to do this communication is yielding special values.
One rather important difference between generator-based tasks and gevent (and similar libraries) is that context switches in the latter are implicit, while tasks make it trivial to identify context switches: Only things that yield [from] can possibly run scheduler code. For example, you can make sure whether a piece of code is atomic (w.r.t. other tasks; if you add threads to the mix, you have to worry about them independently) just by looking at the code, without inspecting anything it calls.
Finally, you may be interested in Greg Ewing's tutorial on creating such a scheduler. (This came up on python-ideas while brainstorming over what now is PEP 3156. These mail threads may also be of interest to your, though the web-based archive is not really suited to reading hundreds of mails in dozens of threads written half a year ago.)
The key is to realise that you will have to provide your own driving loop—I have provided a simple demo below. I was lazy and used a Queue object to provide a FIFO, I haven't used python for a significant project for a while.
#!/usr/bin/python
import Queue
def foo():
print('Constructing foo')
yield
print('Running in foo')
yield
print('Explicit context switch to foo again')
def bar():
print('Constructing bar')
yield
print('Explicit context to bar')
yield
print('Implicit context switch back to bar')
def trampoline(taskq):
while not taskq.empty():
task = taskq.get()
try:
task.next()
taskq.put(task)
except StopIteration:
pass
tasks = Queue.Queue()
tasks.put(foo())
tasks.put(bar())
trampoline(tasks)
print('Finished')
And when run:
$ ./coroutines.py
Constructing foo
Constructing bar
Running in foo
Explicit context to bar
Explicit context switch to foo again
Implicit context switch back to bar
Finished
I have a list in my python program that gets new items on certain occasions (It's a message-queue consumer). Then I have a thread that every few minutes checks to see if there's anything in the list, and if there is then I want to do an action on each item and then empty the list.
Now my problem: should I use locks to ensure that the action in the subthread is atomic, and does this ensure that the main thread can't alter the list while I'm going through the list?
Or should I instead use some kind of flag?
Pseudocode to make my problem clearer.
Subthread:
def run(self):
while 1:
if get_main_thread_list() is not empty:
do_operations()
empty_the_list()
sleep(30)
Main thread:
list = []
def on_event(item):
list.add(item)
def main():
start_thread()
start_listening_to_events()
I hope this makes my problem clearer, and any links to resources or comments are obviously welcome!
PS: I'm well aware that I just might not grasp threaded programming well enough for this question, if you believe so could you please take some time explaining whats wrong with my reasoning if you have the time.
should I use locks to ensure that the action in the subthread is atomic, and does this ensure that the main thread can't alter the list while I'm going through the list?
Yes. If you implement it correctly yes.
Or should I instead use some kind of flag?
"some kind of flag" == lock, so you'd better use threading locks.
Important: It looks to me like you're trying to reimplement the queue module from the stdlib, you might want to take a look at it.
Other than having a bunch of interesting features is also thread safe:
The queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics.
I'm a bit puzzled about how to write asynchronous code in python/twisted. Suppose (for arguments sake) I am exposing a function to the world that will take a number and return True/False if it is prime/non-prime, so it looks vaguely like this:
def IsPrime(numberin):
for n in range(2,numberin):
if numberin % n == 0: return(False)
return(True)
(just to illustrate).
Now lets say there is a webserver which needs to call IsPrime based on a submitted value. This will take a long time for large numberin.
If in the meantime another user asks for the primality of a small number, is there a way to run the two function calls asynchronously using the reactor/deferreds architecture so that the result of the short calc gets returned before the result of the long calc?
I understand how to do this if the IsPrime functionality came from some other webserver to which my webserver would do a deferred getPage, but what if it's just a local function?
i.e., can Twisted somehow time-share between the two calls to IsPrime, or would that require an explicit invocation of a new thread?
Or, would the IsPrime loop need to be chunked into a series of smaller loops so that control can be passed back to the reactor rapidly?
Or something else?
I think your current understanding is basically correct. Twisted is just a Python library and the Python code you write to use it executes normally as you would expect Python code to: if you have only a single thread (and a single process), then only one thing happens at a time. Almost no APIs provided by Twisted create new threads or processes, so in the normal course of things your code runs sequentially; isPrime cannot execute a second time until after it has finished executing the first time.
Still considering just a single thread (and a single process), all of the "concurrency" or "parallelism" of Twisted comes from the fact that instead of doing blocking network I/O (and certain other blocking operations), Twisted provides tools for performing the operation in a non-blocking way. This lets your program continue on to perform other work when it might otherwise have been stuck doing nothing waiting for a blocking I/O operation (such as reading from or writing to a socket) to complete.
It is possible to make things "asynchronous" by splitting them into small chunks and letting event handlers run in between these chunks. This is sometimes a useful approach, if the transformation doesn't make the code too much more difficult to understand and maintain. Twisted provides a helper for scheduling these chunks of work, cooperate. It is beneficial to use this helper since it can make scheduling decisions based on all of the different sources of work and ensure that there is time left over to service event sources without significant additional latency (in other words, the more jobs you add to it, the less time each job will get, so that the reactor can keep doing its job).
Twisted does also provide several APIs for dealing with threads and processes. These can be useful if it is not obvious how to break a job into chunks. You can use deferToThread to run a (thread-safe!) function in a thread pool. Conveniently, this API returns a Deferred which will eventually fire with the return value of the function (or with a Failure if the function raises an exception). These Deferreds look like any other, and as far as the code using them is concerned, it could just as well come back from a call like getPage - a function that uses no extra threads, just non-blocking I/O and event handlers.
Since Python isn't ideally suited for running multiple CPU-bound threads in a single process, Twisted also provides a non-blocking API for launching and communicating with child processes. You can offload calculations to such processes to take advantage of additional CPUs or cores without worrying about the GIL slowing you down, something that neither the chunking strategy nor the threading approach offers. The lowest level API for dealing with such processes is reactor.spawnProcess. There is also Ampoule, a package which will manage a process pool for you and provides an analog to deferToThread for processes, deferToAMPProcess.