Django and Flask make use of signals — the latter uses the Blinker library. In the context of Python, Blinker and the Python pubsub library, how do signals and pubsub compare? When would I use one or the other?
Blinker docs and PubSub docs.
As far as Blinker and PubSub go, they are the same thing. The difference is in how they go about it:
With Blinker when you subscribe to a signal you give the name of the signal, and when you activate the signal you pass the activating object.
With PubSub when you subscribe to a listener you give the name (same as Blinker), but when you notify the listener you pass the data directly as keyword arguments. Because of the keyword argument method of passing data, it is possible to have many more safety checks using PubSub.
Personally, I would go with Blinker as it matches my way of thinking better, but PubSub certainly has a place also.
This might clear up exactly how Pubsub relates to signals: http://pubsub.sourceforge.net/apidocs/concepts.html
Pubsub facilitates the decoupling of components (callables, modules, packages) within an application. It does this by:
Allowing parts of the application to send messages to “the rest of the application” without having to know
if the messages will be handled:
perhaps the message will be ignored completely,
or handled by a many different parts of the application;
how the messages will be handled:
what will be done with the message and its contents;
in what order any given message will be sent to the rest of the application;
Allowing parts of the application to receive and handle messages from “the rest of the application” without having to know who sent the messages.
A listener is “a part of the application that wants to receive messages”. A listener subscribes to one or more topics. A sender is any part of the application that asks Pubsub to send a message of a given topic. The sender provides data, if any. Pubsub will send the message, including any data, to all listeners of the message’s topic.
Related
I need to interact with a the Apache Qpid C++ broker using Qpid Proton in Python. The objective is to generate a receiver that hooks to a topic exchange and receives only messages marked with a topic. I am aware that this can be easily done with Qpid API, but I needs to use Qpid Proton.
I have a horrible hack that inspects any message that arrives and check if topic (subject) of the message match some pattern, in a similar way as AMPQ 0-10 exchange binding do with routing keys. My idea is to move this to a Filter in the receiver, but after hours googling I have found no way to proceed.
My current implementation is using blocking connections, both to sender and receiver.
Any suggestion?
I am pulling pubsub messages through a subscription and need to acknowledge these before processing as I am doing multiprocessing and that throws an error of SSL corruption on account of the grpc module.
I want to ack all messages beforehand and unack in case there was an error, I am aware that we can do this for an asynchronous pull but is there a way where we can implement unack in synchronous pull as well?
I am using the official python module to pull from subscription
I suppose that unack you mean nack explained in Python API reference:
In Pub/Sub, the term ack stands for “acknowledge”.
...
It is also possible to nack a message, which is the opposite...
The same documentation contain part Pulling a Subscription Synchronously
in which it is explained how to nack with modify_ack_deadline():
If you want to nack some of the received messages (...), you can use the modify_ack_deadline() method and set their
acknowledge deadlines to zero. This will cause them to be dropped by
this client and the backend will try to re-deliver them.
I am a newbie to Kafka and PyKafka.I know that a producer and a consumer are made in PyKafka via the below code.
from pykafka import KafkaClient
client = KafkaClient("localhost:9092")
topic = client.topics["topicname"]
producer = topic.get_producer()
consumer = topic.get_simple_consumer()
I want to know what is KafkaClient, and how it is helping in creating producer and consumer.
I have read we can create cluster and broker also using client.cluster and client.broker, but I can't understand the use of client here.
To make terms simpler, replace Kafka with "server".
You interact with servers with clients.
To interact with Kafka, in particular, you send messages to topics via producers, and get messages with consumers.
I don't know this library off-hand, but .broker and .cluster aren't actually "making a Kafka broker / cluster", only establishing a connection to an existing one, from which you can issue later commands.
You need the client. on those function calls because the client is a wrapper around both
To know why it is structured in this way, you'd have to ask the developers themselves
pykafka.KafkaClient is the root object of the PyKafka API, providing an interface to Kafka brokers as well as the ability to instantiate consumer and producer instances. The KafkaClient can be thought of as a representation of the totality of one Python process' interaction with a given Kafka cluster. There is no direct comparison between KafkaClient and any of the concepts mentioned in the official Kafka documentation.
It's totally possible in theory to design a python Kafka client library that doesn't have a "client" class like KafkaClient. We decided not to since in our opinion a single root class provides a cleaner, more learnable interface than a bag of various classes.
I am using mysqldb for my database currently, and I need to integrate a messaging feature that is in real-time. The chat demo that Tornado provides does not implement a database, (whereas the blog does.)
This messaging service also will also double as an email in the future (like how Facebook's message service works. The chat platform is also email.) Regardless, I would like to make sure that my current, first chat version will be able to be expanded to function as email, and overall, I need to store messages in a database.
Is something like this as simple as: for every chat message sent, query the database and display the message on the users' screen. Or, is this method prone to suffer from high server load and poor optimization? How exactly should I structure the "infrastructure" to make this work?
(I apologize for some of the inherent subjectivity in this question; however, I prefer to "measure twice, code once.")
Input, examples, and resources appreciated.
Regards.
Tornado is a single threaded non blocking server.
What this means is that if you make any blocking calls on the main thread you will eventually kill performance. You might not notice this at first because each database call might only block for 20ms. But once you are making more than 200 database calls per seconds your application will effectively be locked up.
However that's quite a few DB calls. In your case that would be 200 people hitting send on their chat message in the same second.
What you probably want to do is use a queue with a non blocking API. So Tornado receives a chat message. You put it on the queue to be saved to the database by another process, then you send the chat message back out to the other chat members.
When someone connects to a chat session you also need to send off a request to the queue for all the previous messages, when the queue responds you send those off to the newly connected user.
That's how I would approach the problem anyway.
Also see this question and answer: Any suggestion for using non-blocking MySQL api on Tornado in Python3?
Just remember, Tornado is single threaded. It's amazing. And can handle thousands of simultaneous connections. But if code in one of those connections blocks for 1 second then NOTHING else will be done for any other connection during that second.
I am looking for a RPC library in Java or Python (Python is preferred) that uses TCP. It should support:
Asynchronous
Bidirectional
RPC
Some sort of event loop (with callbacks or similar)
Any recommendations? I have looked a things like bjsonrpc which seemed to be the right sort of thing however it didn't seem possible for the server to identify which connections; so if a user has identified himself/herself and a request comes in from another user to send a message to that user it doesn't expose that users connection so we can send the message.
You should definitely check out Twisted. It's an event-based Python networking framework that has an implementation of an event loop (called the "reactor") supporting select, poll, epoll, kqueue and I/O completion ports, and mediates asynchronous calls with objects called Deferreds
As for your RPC requirement, perhaps you should check out Twisted's PB library or AMP.
I'm not entirely sure what you meanאt by "Event loop", but you should check out RPyC (Python)
RPyC Project page
i'm the author of bjsonrpc. I'm sure it's possible to do what you want with it.
Some things maybe are poorly documented or maybe some examples are needed.
But, in short, Handlers can store internal states (like authenticated or not, or maybe username). From any handler you can access the "Connection" class, which has the socket itself.
Seems you want something like a chat as an example. I did something similar in the past. I'll try to add a chat example for a new release.
Internal states are explained here:
http://packages.python.org/bjsonrpc/tutorial1/index.html#stateful-server
They should be used for authentication (but no standard auth method is provided yet).
On how to reach the connection class from the handler, that isn't documented yet (sorry), but it is used sometimes in the examples inside the source code. For example, example1-server.py contains this public function:
def gettotal(self):
self._conn.notify.notify("total")
return self.value_total
BaseHandler._conn represents the connection for that user. And is exactly the same class you get when you connect:
conn = bjsonrpc.connect(host=host,port=port,handler_factory=MyHandler)
So, you can store the connections for logged users in a global variable, and later call any client method you want to.
I am involved in developing Versile Python (VPy) which provides the capabilities you are requesting. It is currently available as development releases intended primarily for testing, however you may want to check it out.
Regarding identifying users you can configure remote methods to receive a context object which enables the method to receive information about an authenticated user, using a syntax similar to this draft code.
from versile.quick import *
#doc
class MessageBox(VExternal):
"""Dispatches IM messages."""
#publish(show=True, doc=True, ctx=True)
def send_message(self, msg, ctx=None):
"""Sends a message to the message box"""
if ctx.identity is None:
raise VException('No authenticated user')
else:
# do something ...
pass