I want to integrate some asyncore.dispatcher instances into GLib's default main context.
I figure I can create a custom GSource that's able to detect event readiness on the various sockets in asyncore.socket_map. From C I believe this is done by creating the necessary GSourceFuncs which could involve cheap and non-blocking calls to select, and then handling them using asyncore.read, .write and friends.
How do I actually create a GSource from Python? The class glib.Source is undocumented, and attempts to use the class interactively have been in vain.
Is there some other method that allows me to handled socket events in the asyncore module without resorting to timeouts (or anything that endangers potential throughput and CPU usage)?
glib.io_add_watch does it (add a watch over a fd to the main loop):
http://library.gnome.org/devel/pygobject/2.26/glib-functions.html#function-glib--io-add-watch
An easy way to integrate a dispatcher with a glib loop is shown, very simplified, below:
from asyncore import dispatcher, loop
from glib import MainLoop, idle_add
def loop_one():
loop(timeout=0, count=1)
return True
idle_add(loop_one)
Related
I'm trying to figure out how Gevent works with respect to other asynchronous frameworks in python, like Twisted.
The key difference between Gevent and Twisted is that Gevent uses greenlets and monkey patching the standard library for an implicit behavior and a synchronous programming model whereas Twisted requires specific libraries and callbacks for an explicit behavior. The event loop in Gevent is libev/libevent, which is written in C, and the event loop in Twisted is the reactor, which is written in python.
Is there anything special about libev/libevent that allows for this implicit behavior? Why not use an event loop written in Python? Conversely, why isn't Twisted using libev/libevent? Is there any particular reason? Maybe it was simply a design choice and could have gone either way...
Theoretically, can Gevent's libev be replaced with another event loop, written in python, like Twisted's reactor? And can Twisted's reactor be replaced with libev?
Short answer: Twisted is a network framework. Gevent tries to act as a library without requiring from the programmer to change the way he programs. That's their focus.. and not so much how that is achieved under the hood.
Long answer:
All asyncio libraries (Gevent, Asyncio, etc.) work pretty much the same:
Have a main loop running endlessly on a single thread.
When an event occurs, it's captured by the main loop.
The main loop decides based on different rules (scheduling) if it should continue checking for events or switch temporarily and give control to any subscriber functions to the event.
greenlet is a different library. It's very simple in that it just changes the order that Python code is run and lets you change jumping back and forth between functions. Gevent uses it under the hood to implement its async features.
asyncio which comes with Python3 is like gevent. The big difference is the interface again. It requires the programmer to mark functions with async and allow him to explicitly wait for a subscribed function in the main loop with await.
Gevent is like asyncio. But instead of the keywords it patches existing code where appropriate. It uses greenlet under the hood to switch between main loop and subscribed functions and make it all work seamlessly.
Twisted as mentioned feels more like a framework than a library. It requires the programmer to follow very specific ways to achieve concurrency. Again though it has a main loop under the hood called reactor like everything else.
Back to your initial question: You can in theory replace the reactor with any loop (including gevent). But that would defeat the purpose. Probably Twisted's team decided to use their own version of a main loop for optimisation reasons. All these libraries use different scheduling in their main loops to meet their needs.
For example - I have some class that internally uses Gevent for concurrency, and it takes a callback, what if user of my class will use Python threading or multiprocessing modules? Will this lead the program to disaster?
UPDATE
Some more details:
My class is a custom protocol implementation, so it involves lots of reading/writing with sockets. Protocol includes such feature as option to send multiple requests (within single TCP connection) in order to load multiple cores on a server, and then wait for responses. That's why I need concurrency. This class is a library which will be part of a large project, and it can be used in every possible crazy way. I'm also new to Gevent and I barely aware about monkey-patching feature.
Well, the initial thing to my mind was how to make sure if pydispatcher or pubsub is thread-safe or not. pubsub might be a little tricky or complex to figure out but pydispatcher seems simple to realize. Then I started to wonder how to figure out if a python module thread-safe or not. Any heuristics?
For determining if a library or application is thread safe, without author input, I would look for mechanisms for synchronizing threads: http://effbot.org/zone/thread-synchronization.htm
or that it contains threading methods: http://docs.python.org/library/threading.html
However, none of that will tell you how to use the API in a thread safe manner. Practically anything can be stuffed inside a thread object and communicated to using thread synchronization objects.
For something like pubsub one could create a class that wraps the API and communicates over Queues exclusively. If pubsub lived in the same thread as wx for example, then an API could be created to inject messages into the Queue using a threading API for sending messages. Then a pubsub loop or timer could be monitoring the Queue. It would then send out messages. One of the issues with wrapping something like pubsub is that somewhere it will require polling. It could be made transparent if the polling were done by timers. Each thread would have to allocate a timer to receive messages if pubsub did not reside in that thread. There might be more elegant approaches to this, but I am not aware of them.
From a theoretic point of view: There is no algorithm which does this for an arbitrary program. It is like the halting problem.
You can inspect the used modules and check if these are granted to be thread safe. But there is no general way to check the byte code of a module for thread safety.
Is it possible to integrate asyncore with dbus through the same main loop?
Usually, DBus integration is done through glib main loop: is it possible to have either asyncore integrate this main loop or have dbus use asyncore's ?
asyncore sucks. glib already provides async stuff, so just use glib's mainloop to do everything.
I wrote a trivial GSource wrapper for one of my own projects called AsyncoreGSource
Just attach it to an appropriate MainContext:
source = AsyncoreGSource([socket_map])
source.attach([main_context])
Naturally the defaults are asyncore.socket_map and the default MainContext respectively.
You can also try monkey-patching asyncore.socket_map, which would have been my solution had I not poked through the GLib python bindings source code for GSource.
Although you got what is probably a perfectly reasonable answer, there is another approach - you don't need to use asyncore's loop per se. Just call asyncore.loop with a zero timeout and a count of 1, which stops it iterating (and thus makes the function name completely misleading) and polls the sockets just once. Call this as often as you need.
I don't know anything about glib's async support but if it requires threads you might still get better performance by using asyncore in this way since it will use select or poll and won't need to spawn additional threads.
I need to allow other Python applications to register callback functions for events in my application. These need to have priorities associated with them (so a callback with a priority of 10 runs before a callback with a priority of 1) and callback functions need to be able to signal errors to the dispatcher.
These are all lightweight callbacks running in the same process, so I don't need to send signals across process boundaries.
Is there a good Python library to handle this, or do I need to write my own?
Are these other applications running in another address space? If so, you'll need to use an interprocess communication library like D-BUS.
If you're just sending signals in the same process, try PyDispatcher
What platform are you running under? GObject is the basis of the GTK GUI that's widely-used under Linux, and it supports event loops with prioritizable events like this.
Try Twisted for anything network-related. Its perspective broker is quite nice to use.
Try python-callbacks - http://code.google.com/p/python-callbacks/.