I have an activemq set up with a ton of api calls to Zendesk. I need to retrieve those calls and then remove them from the queue entirely. conn.ack doesn't seem to be working!
I'm using python3 and the most recent version of stomp. I used this to make the initial connection script: https://github.com/jasonrbriggs/stomp.py/wiki/Simple-Example
https://jasonrbriggs.github.io/stomp.py/api.html
In this doc, it looks like you have to label the "id" tag of the .subscribe method. You call conn.ack with that id, but you also add the message id as an argument. I found out that the headers of the message are retrieved with the listener function. I printed those out and they look like this:
ID:[my workstation id].local-49557-1560302581785-5:58:-1:1:61592
I tried regex'ing out the whole string after ID:, and then I tried regexing out just the number on the very end of the string (it looks like possibly a unique number), but when I do conn.ack (matchObj.group(1), 4), the queue count doesn't change and I get no feedback as to why it doesn't.
The connection works absolutely fine so far-- I just can't send those acks.
import stomp
import time
import re
class SampleListener(object):
def on_message(self, headers, msg):
regex = r"ID:.*1:1:(.*)"
print(msg)
print(headers['message-id'])
matchObj = re.match ( regex, headers['message-id'], re.M|re.I)
print (matchObj.group(1))
conn.ack (matchObj.group(1), 4)
conn = stomp.Connection10()
conn.set_listener('SampleListener', SampleListener())
conn.start()
conn.connect()
conn.subscribe('zendeskqueue', id=4, ack='client')
time.sleep(1) # secs
conn.disconnect()
The code above has no error, it just exists without output.
Acknowledging messages with stomp.py and ActiveMQ works using ack id headers['ack'].
And you can get more verbose output by implementing on_error on your listener, and enabling debug logging.
With both, your code will look like that:
import stomp
import time
import logging
class SampleListener(object):
def on_message(self, headers, msg):
conn.ack(headers['ack'])
def on_error(self, headers, body):
print("An error occurred: headers=" + str(headers) + " ; body=" + str(body))
logging.basicConfig(level=logging.INFO)
logging.getLogger('stomp').setLevel(logging.DEBUG)
conn = stomp.Connection12()
conn.set_listener('SampleListener', SampleListener())
conn.start()
conn.connect()
conn.subscribe('zendeskqueue', id=4, ack='client')
time.sleep(1) # secs
conn.disconnect()
Related
I'm using ActiveMQ classic v5.16.3 and experimenting with NACK. My expectation is that if the client sends a NACK then the message will remain on the queue and be available for another client. My code is below. I set a prefetch of 1, and ack mode of client-individual.
If I omit the conn.nack() call then I see my print statements, and the message remains on the queue - hence I believe that ActiveMQ is looking for an ACK or NACK.
When I include the conn.nack() call then I again see my print statements, and the message is removed from the queue.
Is this expected behaviour? I think a client should be able to reject malformed messages by NACK-ing and that eventually ActiveMQ should put them to a dead letter queue.
import time
import sys
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, frame):
print('received an error "%s"' % frame.body)
def on_message(self, frame):
# experiment with and without the following line
conn.nack(id=frame.headers['message-id'], subscription=frame.headers["subscription"])
print('received a message "%s"' % frame.body)
print('headers "%s"' % frame.headers)
print('Connecting ...')
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.connect('admin', 'admin', wait=True)
print('Connected')
conn.subscribe(destination='/queue/audit', id=1, ack='client-individual', headers={'activemq.prefetchSize': 1})
As suggested by Tim Bish, I needed to configure ActiveMQ to retry. I made the following changes to activemq.xml
Added scheduler support to the broker:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}"
schedulerSupport="true" >
Specified the redelivery plugin:
<plugins>
<redeliveryPlugin fallbackToDeadLetter="true"
sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<defaultEntry>
<redeliveryPolicy maximumRedeliveries="4"
initialRedeliveryDelay="5000"
redeliveryDelay="10000"/>
</defaultEntry>
</redeliveryPolicyMap>
</redeliveryPolicyMap>
</redeliveryPlugin>
</plugins>
And then for my chosen destination, specify that poison messages be sent to a specific queue - default is to publish to a Topic.
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue="audit" prioritizedMessages="true" >
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."
useQueueForQueueMessages="true"/>
</deadLetterStrategy>
</policyEntry>
How to write a simple Python3 websockets client ?
Given https://github.com/Pithikos/python-websocket-server (a multi-client server implementation) which has been tested with the authors provided web client (ie. javascript), I like to write a simple python3 client using standard python websockets module. I expect the client to send a text message and get an answer back (sort of like an echo server). Nothing fancy on the client, no concurrency, just connect, send, recv and close (or exit).
#!/usr/bin/python3
import sys
import websockets
req = 'joe'
if sys.argv[1:]:
req = sys.argv[1]
with websockets.connect("ws://localhost:9001") as ws:
print("req: %s" % (req) )
ws.send(req)
res = ws.recv()
print("res: %s" % (res) )
And when I run it ...
medi#medi:~/proto/python/d4> ./client 'hello Joe'
Traceback (most recent call last):
File "./client", line 10, in <module>
with websockets.connect("ws://localhost:9001") as ws:
AttributeError: __enter__
I was expecting to be able to send() and recv(). I am having difficulty finding python based code (have seen lots of JS clients) sample.
This is an async websocket client that supports asyncio. You need to use it like this:
import asyncio
import websockets
async def main():
async with websockets.connect(...) as websocket:
...
asyncio.run(main())
I am currently writing two scripts to subscribe to a message server using the stomp client library, write.py to write data and read.py to get data.
If I start read.py first and then run write.py, write.py receives the messages correctly.
However, if I run write.py first and then run read.py, read.py does not retrieve any messages previously sent to the server.
Below are relevant parts of the scripts.
How can I achieve that messages put into the queue by write.py are being retained until read.py subscribes and retrieves them?
write.py
def writeMQ(msg):
queue = '/topic/test'
conn = stomp.Connection(host_and_ports=[(MQ_SERVER, MQ_PORT)])
try:
conn.start()
conn.connect(MQ_USER, MQ_PASSWD, wait=True)
conn.send(body=msg, destination=queue, persistent=True)
except:
traceback.print_exc()
finally:
conn.disconnect()
return
read.py
class MyListener(stomp.ConnectionListener):
def on_error(self, headers, message):
print ('received an error {0}'.format(message))
def on_message(self, headers, message):
print ('received an message {0}'.format(message))
def readMQ():
queue = '/topic/test'
conn = stomp.Connection(host_and_ports=[(MQ_SERVER, MQ_PORT)])
try:
conn.set_listener("", MyListener())
conn.start()
conn.connect(MQ_USER, MQ_PASSWD, wait=True)
conn.subscribe(destination=queue, ack="auto", id=1)
stop = raw_input()
except:
traceback.print_exc()
finally:
conn.disconnect()
return
The problem is that the messages are being sent to a topic.
The Apollo Documentation describes the difference between topics and queues as follows:
Queues hold on to unconsumed messages even when there are no subscriptions attached, while a topic will drop messages when there are no connected subscriptions.
Thus, when read.py is startet first and listening, the topic recognizes the subscription and forwards the message. But when write.py is startet first the message is dropped because there is no subscribed client.
So you can use a queue instead of a topic. If the server is able to create a queue silently simply set
queue = '/queue/test' .
I don't know which version of stomp is being used, but I cannot find the parameter
send(..., persistent=True) .
Anyway persisting is not the right way to go since it still does not allow for messages to simply be retained for a later connection, but saves the messages in case of a server failure.
You can use the
retain:set
header for topic messages instead.
I am attempting to write a simple TCP server in twisted which has to perform the following operations in sequence:
A client connects to the server and the KEEPALIVE flag for this connection is set to 1.
The server receives data from the client.
It then computes the response which is a list.
The server then sends each item of the list one by one while waiting for explicit ACKs from the client in between, i.e., after sending a single item from the list, the server waits for an ACK packet from the client and only after receiving the ACK does it proceed to send the rest of the items in the same manner.
The following is the code:
class MyFactory(ServerFactory):
protocol = MyProtocol
def __init__(self, service):
self.service = service
class MyProtocol(Protocol):
def connectionMade(self):
try:
self.transport.setTcpKeepAlive(1)
except AttributeError:
pass
self.deferred = Deferred()
self.deferred.addCallback(self.factory.service.compute_response)
self.deferred.addCallback(self.send_response)
def dataReceived(self, data):
self.fire(data)
def fire(self, data):
if self.deferred is not None:
d, self.deferred = self.deferred, None
d.callback(data)
def send_response(self, data):
for item in data:
d = Deferred()
d.addCallback(self.transport.write)
d.addCallback(self.wait_for_ack)
d.callback(item)
return
def wait_for_ack(self, dummy):
try:
self.transport.socket.recv(1024)
except socket.error as e:
print e
return
Upon running the server and the client I get the following exception:
Resource temporarily unavailable
I understand the reason for this exception - I'm trying to call a blocking method on non blocking socket.
Please help me in finding a solution to this problem.
There are some problems with your example:
You don't define compute_response anywhere (among other things) so I can't run your example. Consider making it an http://sscce.org
You should never call either send or recv on a socket underlying a Twisted transport; let Twisted call those methods for you. In the case of recv it will deliver the results of recv to dataReceived.
You can't rely upon dataReceived to receive whole messages; packets may always be arbitrarily fragmented in transit so you need to have a framing protocol for encapsulating your messages.
However, since my other answer was so badly botched, I owe you a more thorough explanation of how to set up what you want to do.
As stipulated in your question, your protocol is not completely defined enough to give an answer; you cannot do requests and responses with raw TCP fragments, because your application can't know where they start and end (see point 3 above). So, I've invented a little protocol to serve for this example: it's a line-delimited protocol where the client sends "request foo\n" and the server immediately sends "thinking...\n", computes a response, then sends "response foo\n" and waits for the client to send "ok"; in response, the server will either send the next "response ..." line, or a "done\n" line indicating that it's finished sending responses.
With that as our protocol, I believe the key element of your question is that you cannot "wait for acknowledgement", or for that matter, anything else, in Twisted. What you need to do is implement something along the lines of "when an acknowledgement is received...".
Therefore, when a message is received, we need to identify the type of the message: acknowledgement or request?
if it's a request, we need to compute a response; when the response is finished being computed, we need to enqueue all the elements of the response and send the first one.
if it's an acknowledgement, we need to examine the outgoing queue of responses, and if it has any contents, send the first element of it; otherwise, send "done".
Here's a full, runnable example that implements the protocol I described in that way:
from twisted.internet.protocol import ServerFactory
from twisted.internet.task import deferLater
from twisted.internet import reactor
from twisted.internet.interfaces import ITCPTransport
from twisted.protocols.basic import LineReceiver
class MyProtocol(LineReceiver):
delimiter = "\n"
def connectionMade(self):
if ITCPTransport.providedBy(self.transport):
self.transport.setTcpKeepAlive(1)
self.pendingResponses = []
def lineReceived(self, line):
split = line.rstrip("\r").split(None, 1)
command = split[0]
if command == b"request":
# requesting a computed response
payload = split[1]
self.sendLine("thinking...")
(self.factory.service.computeResponse(payload)
.addCallback(self.sendResponses))
elif command == b"ok":
# acknowledging a response; send the next response
if self.pendingResponses:
self.sendOneResponse()
else:
self.sendLine(b"done")
def sendOneResponse(self):
self.sendLine(b"response " + self.pendingResponses.pop(0))
def sendResponses(self, listOfResponses):
self.pendingResponses.extend(listOfResponses)
self.sendOneResponse()
class MyFactory(ServerFactory):
protocol = MyProtocol
def __init__(self, service):
self.service = service
class MyService(object):
def computeResponse(self, request):
return deferLater(
reactor, 1.0,
lambda: [request + b" 1", request + b" 2", request + b" 3"]
)
from twisted.internet.endpoints import StandardIOEndpoint
endpoint = StandardIOEndpoint(reactor)
endpoint.listen(MyFactory(MyService()))
reactor.run()
I've made this runnable on standard I/O so that you can just run it and type into it to get a feel how it works; if you want to run it on an actual network port, just substitute that with a different type of endpoint. Hopefully this answers your question.
I have the simple code of XMPP bot by python and http://xmpppy.sourceforge.net/
#!/usr/bin/python
# -*- coding: utf-8 -*-
import xmpp
import urllib2
import ConfigParser
config = ConfigParser.ConfigParser()
config.read('ipbot.conf')
##########################
user= (config.get('account', 'login'))
password=(config.get('account', 'password'))
presence=(config.get('presence','presence'))
##########################
jid=xmpp.protocol.JID(user)
client=xmpp.Client(jid.getDomain())
client.connect()
client.auth(jid.getNode(),password)
################Parse IP##################
strURL='http://api.wipmania.com/'
f = urllib2.urlopen(urllib2.Request(strURL))
response = f.read()
ipget= response.split("<br>")
f.close()
#############################################
def status(xstatus):
status=xmpp.Presence(status=xstatus,show=presence,priority='1')
client.send(msging)
def message(conn,mess):
global client
if ( mess.getBody() == "ip" ):
client.send(xmpp.protocol.Message(mess.getFrom(),ipget[1]+" => "+ipget[0]))#Send IP
client.RegisterHandler('message',message)
client.sendInitPresence()
while True:
client.Process(1)
Please, tell me, how to translate this code to use http://wokkel.ik.nu/ and twistedmatrix.com/
Thanks a lot.
The following code should do it. A few notes:
Wokkel uses so-called subprotocol handlers to cover support for specific
subprotocols, usually split up by conceptual feature, by namespace or per
XEP.
XMPPClient is a so-called stream manager that establishes connections,
and takes care of authentication with the server. It works with the
hooked-up subprotocol handlers to process traffic with the XML stream it
manages. It automatically reconnects if the connection has been lost.
This example defines one new handler to process incoming messages.
Unlike the original code, here the request to retrieve the IP address is
done for each incoming message with ip in the body.
In the original code, status was never called. I now the use
PresenceProtocol subprotocol handler to send out the presence each time
a connection has been established and authentication has taken place.
The example is a so-called Twisted Application, to be started using twistd, as mentioned in the docstring. This will daemonize the process and logs go to twisted.log. If you specify -n (before -y), it will not detach and log to the console instead.
#!/usr/bin/python
"""
XMPP example client that replies with its IP address upon request.
Usage:
twistd -y ipbot.tac
"""
import ConfigParser
from twisted.application import service
from twisted.python import log
from twisted.web.client import getPage
from twisted.words.protocols.jabber.jid import JID
from twisted.words.protocols.jabber.xmlstream import toResponse
from wokkel.client import XMPPClient
from wokkel.xmppim import PresenceProtocol, MessageProtocol
class IPHandler(MessageProtocol):
"""
Message handler that sends presence and returns its IP upon request.
#ivar presenceHandler: Presence subprotocol handler.
#type presenceHandler: L{PresenceProtocol}
#ivar show: Presence show value to send upon connecting.
#type show: C{unicode} or C{NoneType}
"""
def __init__(self, presenceHandler, show=None):
self.presenceHandler = presenceHandler
self.show = show
def connectionInitialized(self):
"""
Connection established and authenticated.
Use the given presence handler to send presence.
"""
MessageProtocol.connectionInitialized(self)
self.presenceHandler.available(show=self.show, priority=1)
def onMessage(self, message):
"""
A message has been received.
If the body of the incoming message equals C{"ip"}, retrieve our
IP address and format the response message in the callback.
"""
def onPage(page):
address, location = page.split(u"<br>")
body = u"%s => %s" % (location, address)
response = toResponse(message, stanzaType=message['type'])
response.addElement("body", content=body)
self.send(response)
if unicode(message.body) != u"ip":
return
d = getPage("http://api.wipmania.com")
d.addCallback(onPage)
d.addErrback(log.err)
# Read the configuration file
config = ConfigParser.ConfigParser()
config.read('ipbot.conf')
user = config.get('account', 'login')
password = config.get('account', 'password')
presence = config.get('presence','presence')
# Set up a Twisted application.
application = service.Application('XMPP client')
# Set up an XMPP Client.
jid = JID(user)
client = XMPPClient(jid, password)
client.logTraffic = True
client.setServiceParent(application)
# Add a presence handler.
presenceHandler = PresenceProtocol()
presenceHandler.setHandlerParent(client)
# Add our custom handler
ipHandler = IPHandler(presenceHandler, presence)
ipHandler.setHandlerParent(client)