Is it possible to integrate asyncore with dbus through the same main loop?
Usually, DBus integration is done through glib main loop: is it possible to have either asyncore integrate this main loop or have dbus use asyncore's ?
asyncore sucks. glib already provides async stuff, so just use glib's mainloop to do everything.
I wrote a trivial GSource wrapper for one of my own projects called AsyncoreGSource
Just attach it to an appropriate MainContext:
source = AsyncoreGSource([socket_map])
source.attach([main_context])
Naturally the defaults are asyncore.socket_map and the default MainContext respectively.
You can also try monkey-patching asyncore.socket_map, which would have been my solution had I not poked through the GLib python bindings source code for GSource.
Although you got what is probably a perfectly reasonable answer, there is another approach - you don't need to use asyncore's loop per se. Just call asyncore.loop with a zero timeout and a count of 1, which stops it iterating (and thus makes the function name completely misleading) and polls the sockets just once. Call this as often as you need.
I don't know anything about glib's async support but if it requires threads you might still get better performance by using asyncore in this way since it will use select or poll and won't need to spawn additional threads.
Related
I'm trying to figure out how Gevent works with respect to other asynchronous frameworks in python, like Twisted.
The key difference between Gevent and Twisted is that Gevent uses greenlets and monkey patching the standard library for an implicit behavior and a synchronous programming model whereas Twisted requires specific libraries and callbacks for an explicit behavior. The event loop in Gevent is libev/libevent, which is written in C, and the event loop in Twisted is the reactor, which is written in python.
Is there anything special about libev/libevent that allows for this implicit behavior? Why not use an event loop written in Python? Conversely, why isn't Twisted using libev/libevent? Is there any particular reason? Maybe it was simply a design choice and could have gone either way...
Theoretically, can Gevent's libev be replaced with another event loop, written in python, like Twisted's reactor? And can Twisted's reactor be replaced with libev?
Short answer: Twisted is a network framework. Gevent tries to act as a library without requiring from the programmer to change the way he programs. That's their focus.. and not so much how that is achieved under the hood.
Long answer:
All asyncio libraries (Gevent, Asyncio, etc.) work pretty much the same:
Have a main loop running endlessly on a single thread.
When an event occurs, it's captured by the main loop.
The main loop decides based on different rules (scheduling) if it should continue checking for events or switch temporarily and give control to any subscriber functions to the event.
greenlet is a different library. It's very simple in that it just changes the order that Python code is run and lets you change jumping back and forth between functions. Gevent uses it under the hood to implement its async features.
asyncio which comes with Python3 is like gevent. The big difference is the interface again. It requires the programmer to mark functions with async and allow him to explicitly wait for a subscribed function in the main loop with await.
Gevent is like asyncio. But instead of the keywords it patches existing code where appropriate. It uses greenlet under the hood to switch between main loop and subscribed functions and make it all work seamlessly.
Twisted as mentioned feels more like a framework than a library. It requires the programmer to follow very specific ways to achieve concurrency. Again though it has a main loop under the hood called reactor like everything else.
Back to your initial question: You can in theory replace the reactor with any loop (including gevent). But that would defeat the purpose. Probably Twisted's team decided to use their own version of a main loop for optimisation reasons. All these libraries use different scheduling in their main loops to meet their needs.
I have implemented a server program using Twisted. I am using twisted.protocols.basic.LineReceiver along with twisted.internet.protocol.ServerFactory.
I would like to have each client that connects to the server run a set of functions in parallel (I'm thinking of multi-threading for this).
I have some confusion with using twisted.internet.threads.deferToThread for this problem.
Should I call deferToThread in the ServerFactory for this purpose?
Are twisted threads, thread-safe with respect to race conditions?
Previously, I tried using multiprocessing in my server program but it seemed not to work in combination with the Twisted reactor, while deferToThread did the job.
I'm wondering how are Twisted threads implemented? Don't they utilize multiprocessing?
Previously, I tried using multiprocessing in my server program but it seemed not to work in combination with the Twisted reactor, while deferToThread did the job. I'm wondering how are Twisted threads implemented? Don't they utilize multiprocessing?
You didn't say whether you used the multi-threaded version of multiprocessing or the multi-process version of multiprocessing.
You can read about mixing Twisted and multiprocessing on Stack Overflow, though:
Mix Python Twisted with multiprocessing?
Twisted network client with multiprocessing workers?
is twisted incompatible with multiprocessing events and queues?
(And more)
To answer the shorter part of this question - no, Twisted does not use the stdlib multiprocessing package to implement its threading APIs. It uses the stdlib threading module.
Are twisted threads, thread-safe with respect to race conditions?
The answer to this is implied by the above answer: no. "Twisted threads" aren't really a thing. Twisted's threading APIs are just a layer on top of the stdlib threading module (which is really just a Python API for POSIX threads (or something kind of similar but different on Windows). Twisted's threading APIs don't magically eliminate the possibility of race conditions (if there is any magic in Twisted, it is the ability to do certain things concurrently without using threads at all - which helps reduce the number of race conditions in your program, though it doesn't entirely eliminate the possibility of creating them).
Should I call deferToThread in the ServerFactory for this purpose?
I'm not quite sure what the point of this question is. Are you wondering if a method on your ServerFactory subclass is the best place to put your calls to deferToThread? That probably depends on the details of your implementation approach. It probably doesn't make a huge difference overall, though. If you like the pattern of having the factory provide services to protocol instances - go for it.
I want to integrate some asyncore.dispatcher instances into GLib's default main context.
I figure I can create a custom GSource that's able to detect event readiness on the various sockets in asyncore.socket_map. From C I believe this is done by creating the necessary GSourceFuncs which could involve cheap and non-blocking calls to select, and then handling them using asyncore.read, .write and friends.
How do I actually create a GSource from Python? The class glib.Source is undocumented, and attempts to use the class interactively have been in vain.
Is there some other method that allows me to handled socket events in the asyncore module without resorting to timeouts (or anything that endangers potential throughput and CPU usage)?
glib.io_add_watch does it (add a watch over a fd to the main loop):
http://library.gnome.org/devel/pygobject/2.26/glib-functions.html#function-glib--io-add-watch
An easy way to integrate a dispatcher with a glib loop is shown, very simplified, below:
from asyncore import dispatcher, loop
from glib import MainLoop, idle_add
def loop_one():
loop(timeout=0, count=1)
return True
idle_add(loop_one)
I'm trying to write a Python lib that will implement the client side of a certain chat protocol.
After I connect to the server,
I start the main loop where I read from the server and handle received commands and here I need to call a callback function (like on_message or on file_received, etc).
How should I go about implementing this?
Should a start a new thread for each callback function? As maybe some callbacks will take some time to return and I will timeout.
Also,
If the main loop where I read from the server is in a thread can I write to the socket from another thread(send messages to the server)?
Or is there a better approach?
Thanks.
For a python app doing this, I wouldn't use threads. I would use a framework like Twisted.
The docs have examples; here's a chat example.
I would use the select module, or alternately twisted, however select is a bit more portable, and to my mind somewhat more pythonic.
Threads are just an unnecessary complication here and will lead to obscure bugs if you're not familiar with how to use them correctly. asyncore or asynchat are simple routes to the same goal, however.
I need to allow other Python applications to register callback functions for events in my application. These need to have priorities associated with them (so a callback with a priority of 10 runs before a callback with a priority of 1) and callback functions need to be able to signal errors to the dispatcher.
These are all lightweight callbacks running in the same process, so I don't need to send signals across process boundaries.
Is there a good Python library to handle this, or do I need to write my own?
Are these other applications running in another address space? If so, you'll need to use an interprocess communication library like D-BUS.
If you're just sending signals in the same process, try PyDispatcher
What platform are you running under? GObject is the basis of the GTK GUI that's widely-used under Linux, and it supports event loops with prioritizable events like this.
Try Twisted for anything network-related. Its perspective broker is quite nice to use.
Try python-callbacks - http://code.google.com/p/python-callbacks/.