I'm working in a project with Python, Twisted and Redis. So the team decided to use txredisapi for the communication between the Python modules and Redis. This project does a lot of different things and we need to subscribe to several channels for listen the messages sent by Redis without the other functionalities stops (asynchronously).
Can one execution handle all the work and listen the messages sent by Redis at the same time or must we separate and execute the code in differents flows?
We use the following code for listen the messages:
import txredisapi as redis
class RedisListenerProtocol(redis.SubscriberProtocol):
def connectionMade(self):
self.subscribe("channelName")
def messageReceived(self, pattern, channel, message):
print "pattern=%s, channel=%s message=%s" %(pattern, channel, message)
def connectionLost(self, reason):
print "lost connection:", reason
class RedisListenerFactory(redis.SubscriberFactory):
maxDelay = 120
continueTrying = True
protocol = RedisListenerProtocol
We try to listen the messages with:
self.connRedisChannels = yield redis.ConnectionPool()
I'm interested to know how can I specify that the Connection must use the "RedisListenerFactory", then I guess that the function "messageReceived" will be fired when a message arrives.
Any suggestions, example or correction will be apreciated.
Thanks!
The following code solves the problem:
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(HOST, PORT)
Thanks to Philippe T. for the help.
If you want to use directly the redis.Connection() may be you can do this before:
redis.SubscriberFactory.protocol = RedisListenerProtocol
the package make internal call to is factory for connection.
other way is to rewrite *Connection class and make*Connection factory to use your factory.
to make the connection on other part of your code you can do something like this :
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
# some where :
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(__HOST__, __PORT__)
# the defer will have your client when the connection is done
Related
I have a python based page which recieves data by POST, which is then forwarded to the Crossbar server using Autobahn (Wamp). It works well the first 1-2 times but when it's called again after that it throws ReactorNotRestartable.
Now, I need this to work whichever way possible, either by reusing this "Reactor" based on a conditional check or by stopping it properly after every run. (The first one would be preferable because it might reduce the execution time)
Thanks for your help!
Edit:
This is in a webpage (Django View) so it needs to run as many times as the page is loaded/data is sent to it via POST.
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.application.internet import ClientService
from autobahn.wamp.types import ComponentConfig
from autobahn.twisted.wamp import ApplicationSession, WampWebSocketClientFactory
class MyAppSession(ApplicationSession):
def __init__(self, config):
ApplicationSession.__init__(self, config)
def onConnect(self):
self.join(self.config.realm)
def onChallenge(self, challenge):
pass
#inlineCallbacks
def onJoin(self, details):
yield self.call('receive_data', data=message)
yield self.leave()
def onLeave(self, details):
self.disconnect()
def onDisconnect(self):
reactor.stop()
message = "data from POST[]"
session = MyAppSession(ComponentConfig('realm_1', {}))
transport = WampWebSocketClientFactory(session, url='ws://127.0.0.1:8080')
endpoint = TCP4ClientEndpoint(reactor, '127.0.0.1', 8080)
service = ClientService(endpoint, transport)
service.startService()
reactor.run()
I figured out a probably hacky-and-not-so-good way by using multiprocessing and putting reactor.stop() inside onJoin() right after the function call. This way I don't have to bother with the "twisted running in the main thread" thing because its process gets killed as soon as my work is done.
Is there a better way?
I have about 200 ip controlled power-bars that I need to connect to and control over TCP sockets. So far I can connect to a single power-bar and control it without issue. Where my issue lies is in how to connect to, play ping pong, as well as send and receive commands for all 200 in one client.
I have researched, hopefully exhaustively, and at most all I can find is a pointer towards select, or twisted - but only for multiple clients connecting to a single server (whereas I need the reverse). All I really need is a prod in the right direction. I can create the sockets for all 200, but I cannot for the life of me figure out how to connect to each device using the IP and Port (60000) and send and receive the proper messages in a non-blocking manner.
Any pointers in the general direction would be greatly appreciated. Hopefully this answer will help someone else with a similar issue to solve. Thanks.
You're learning about non-blocking patterns in Python at the right time ;) There's a plethora of ways to do it so I'm not really surprised you're confused. You've named twisted, which is the most mature framework, and there's also asyncio, which is built into Python 3+. Pick which ever one is easiest for you to learn. As you can see, they're very similar in style.
asyncio_client.py
import asyncio
from uuid import uuid4
class Echo(asyncio.Protocol):
def __init__(self):
self.identity = uuid4().hex
def connection_made(self, transport):
message = '{}: hello world'.format(self.identity)
transport.write(message.encode())
def data_received(self, data):
print(data.decode())
def echo_factory():
return Echo()
async def connect_to_server(loop):
await loop.create_connection(echo_factory, host='127.0.0.1', port=6000)
def main():
loop = asyncio.get_event_loop()
loop.create_task(connect_to_server(loop))
loop.create_task(connect_to_server(loop))
loop.create_task(connect_to_server(loop))
loop.run_forever()
main()
twisted_client.py
from uuid import uuid4
from twisted.internet import endpoints, protocol, reactor
class Echo(protocol.Protocol):
def __init__(self):
self.identity = uuid4().hex
def connectionMade(self):
message = '{}: hello world'.format(self.identity)
self.transport.write(message.encode())
def dataReceived(self, data):
print(data.decode())
def connect_to_server(factory):
return endpoints.clientFromString(reactor, 'tcp:6000:host=127.0.0.1').connect(factory)
def main():
factory = protocol.ClientFactory.forProtocol(Echo)
connect_to_server(factory)
connect_to_server(factory)
connect_to_server(factory)
reactor.run()
main()
I am trying to understand the examples given here: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/wamp/basic/pubsub/basic
I built this script which is supposed to handle multiple pub/sub websocket connections and also open a tcp port ( 8123 ) for incoming control messages. When a message comes on the 8123 port, the application should broadcast to all the connected subscribers the message received on port 8123. How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message.
Another thing that i do not understand is the url. The client javascript connects to the url http://:8080/ws . Where does the "ws" come from ?
Also can someone explain the purpose of RouterFactory, RouterSessionFactory and this bit:
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
my code is below:
import sys, time
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.defer import inlineCallbacks
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.util import sleep
class NotificationProtocol(Protocol):
def __init__(self, factory):
self.factory = factory
def dataReceived(self, data):
print "received new data"
class NotificationFactory(Factory):
protocol = NotificationProtocol
class WsNotificationComponent(ApplicationSession):
#inlineCallbacks
def onJoin(self, details):
counter = 0
while True:
self.publish("com.myapp.topic1", "test %d" % counter )
counter += 1
yield sleep(1)
## we use an Autobahn utility to install the "best" available Twisted reactor
##
from autobahn.twisted.choosereactor import install_reactor
reactor = install_reactor()
## create a WAMP router factory
##
from autobahn.wamp.router import RouterFactory
router_factory = RouterFactory()
## create a WAMP router session factory
##
from autobahn.twisted.wamp import RouterSessionFactory
session_factory = RouterSessionFactory(router_factory)
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
from autobahn.twisted.websocket import WampWebSocketServerFactory
transport_factory = WampWebSocketServerFactory(session_factory)
transport_factory.setProtocolOptions(failByDrop = False)
from twisted.internet.endpoints import serverFromString
## start the server from an endpoint
##
server = serverFromString(reactor, "tcp:8080")
server.listen(transport_factory)
notificationFactory = NotificationFactory()
reactor.listenTCP(8123, notificationFactory)
reactor.run()
"How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message":
Check out one of my other answers on SO: Persistent connection in twisted. Jump down to the example code and model your websocket logic like the "IO" logic and you'll have a good fit (You might also want to see the follow-on answer about the newer endpoint calls from one of the twisted core-team too)
"Where does the "ws" come from ?"
Websockets are implemented by retasking http connections, which by their nature have to have a specific path on the request. That "ws" path typically would map to a special http handler that autobahn is building for you to process websockets (or at least that's what your javascript is expecting...). Assuming thing are setup right you can actually point your web-browswer at that url and it should print back an error about the websocket handshake (Expected WebSocket Headers in my case, but I'm using cyclones websockets not autobahn).
P.S. one of the cool side-effects from "websockets must have a specific path" is that you can actually mix websockets and normal http content on the same handler/listen/port, this gets really handy when your trying to run them all on the same SSL port because your trying to avoid the requirement of a proxy front-ending your code.
I am running an HTTP server using the twisted framework. Is there any way I can "manually" ask it to process some payload? For example, if I've constructed some Ethernet frame can I ask twisted's reactor to handle it just as if it had just arrived on my network card?
You can do something like this:
from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, ClientFactory
class SomeWebThing(Resource):
def render_GET(self, request):
return "hello\n"
class SomeClient(Protocol):
def dataReceived(self, data):
p = self.factory.site.buildProtocol(self.transport.addr)
p.transport = self.transport
p.dataReceived(data)
class SomeClientFactory(ClientFactory):
protocol = SomeClient
def __init__(self, site):
self.site = site
if __name__ == '__main__':
root = Resource()
root.putChild('thing', SomeWebThing())
site = server.Site(root)
reactor.listenTCP(8000, site)
factory = SomeClientFactory(site)
reactor.connectTCP('localhost', 9000, factory)
reactor.run()
and save it as simpleinjecter.py, if you then do (from the commandline):
echo -e "GET /thing HTTP/1.1\r\n\r\n" | nc -l 9000 # runs a server, ready to send req to first client connection
python simpleinjecter.py
it should work as expected, with the request from the nc server on port 9000 getting funneled as the payload into the twisted web server, and the response coming back as expected.
The key lines are in SomeClient.dataRecieved(). You'll need a transport object with the right methods -- in the example above, I just steal the object from the client connection. If you aren't going to do that, I imagine you'll have to make one up, as the stack will want to do things like call getPeer() on it.
What is the use-case?
Perhaps you want to create your own Datagram Protocol
At the base, the place where you
actually implement the protocol
parsing and handling, is the
DatagramProtocol class. This class
will usually be decended from twisted.internet.protocol.DatagramProtocol.
Most protocol handlers inherit either
from this class or from one of its
convenience children. The
DatagramProtocol class receives
datagrams, and can send them out over
the network. Received datagrams
include the address they were sent
from, and when sending datagrams the
address to send to must be specified.
If you want to see wire-level transmissions rather than inject them, install and run WireShark, the fantastic, free packet sniffer.
I am looking for a way to periodically send some data over all clients connected to a TCP port. I am looking at twisted python and I am aware of reactor.callLater. But how do I use it to send some data to all connected clients periodically ? The data sending logic is in Protocol class and it is instantiated by the reactor as needed. I don't know how to tie it from reactor to all protocol instances...
You would probably want to do this in the Factory for the connections. The Factory is not automatically notified of every time a connection is made and lost, so you can notify it from the Protocol.
Here is a complete example of how to use twisted.internet.task.LoopingCall in conjunction with a customised basic Factory and Protocol to announce that '10 seconds has passed' to every connection every 10 seconds.
from twisted.internet import reactor, protocol, task
class MyProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.clientConnectionMade(self)
def connectionLost(self, reason):
self.factory.clientConnectionLost(self)
class MyFactory(protocol.Factory):
protocol = MyProtocol
def __init__(self):
self.clients = []
self.lc = task.LoopingCall(self.announce)
self.lc.start(10)
def announce(self):
for client in self.clients:
client.transport.write("10 seconds has passed\n")
def clientConnectionMade(self, client):
self.clients.append(client)
def clientConnectionLost(self, client):
self.clients.remove(client)
myfactory = MyFactory()
reactor.listenTCP(9000, myfactory)
reactor.run()
I'd imagine the easiest way to do that is to manage a list of clients in the protocol with connectionMade and connectionLost in the client and then use a LoopingCall to ask each client to send data.
That feels a little invasive, but I don't think you'd want to do it without the protocol having some control over the transmission/reception. Of course, I'd have to see your code to really know how it'd fit in well. Got a github link? :)