I'm connecting to a remote mq queue using pymqi. I'm using put method as normal but I would like to know if there are some way to set something symilar as JMSReplyTo on JMS system to get a synchronous response of my message.
The request queue and the response are created in the remote mq where I'm connecting.
Yes you can set ReplyToQ (JMSReplyTo) and No, there is no one-call function for getting synchronous response.
Look example page or example in source
Is short:
# Prepare a Message Descriptor for the request message.
md = pymqi.MD()
md.ReplyToQ = dyn_queue_name
# Send the message.
queue = pymqi.Queue(qmgr, request_queue)
queue.put(message, md)
Related
I am working with multiple applications that communicate asynchronously using Kafka. These applications are managed by several departments and contract testing is appropriate to ensure that the messages used during communication follow the expected schema and will evolve according to the contract specification.
It sounded like the pact library for python is a good fit because it helps creating contract tests for HTTP and message integrations.
What I wanted to do is to send an HTTP request and to listen from the appropriate and dedicated Kafka topic immediately after. But it seems that the test is forcing me specify an HTTP code even if what I am expecting is a message from a queue without an HTTP status code. Furthermore, it seems that the HTTP request is being sent before the consumer is listening. Here is some sample code.
from pact.consumer import Consumer as p_Consumer
from pact.provider import Provider as p_Provider
from confluent_kafka import Consumer as k_Consumer
pact = p_Consumer('Consumer').has_pact_with(p_Provider('Provider'))
pact.start_service()
atexit.register(pact.stop_service)
config = {'bootstrap.servers':'server', 'group.id':0, 'auto.offset.reset':'latest'}
consumer = k_consumer(config)
consumer.subscribe(['usertopic'])
def user():
while True:
msg = consumer.poll(timeout=1)
if msg is None:
continue
else:
return msg.value().decode()
class ConstractTesting(unittest.TestCase):
expected = {
'username': 'UserA',
'id':123,
'groups':['Editors']
}
pact.given('UserA exists and is not an administrator')
.upon_receiving('a request for UserA')
.with_request(method='GET',path='/user/')
.will_respond_with(200, body=expected)
with pact:
result = user()
self.assertEqual(result,expected)
How would I carry out contract testing in Python using Kafka? It feels like I am going through a lot of hoops to carry out this test.
With Pact message it's a different API you write tests against. You don't use the standard HTTP one, in fact the transport itself is ignored altogether and it's just the payload - the message - we're interested in capturing and verifying. This allows us to test any queue without having to build specific interfaces for each
See this example: https://github.com/pact-foundation/pact-python/blob/02643d4fb89ff7baad63e6436f6a929256c6bf12/examples/message/tests/consumer/test_message_consumer.py#L65
You can read more about message pact testing here: https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
And finally here are some Kafka examples for other languages that may be helpful: https://docs.pactflow.io/docs/examples/kafka/js/consumer
I use the python socket.io client and I would like to know if it is possible to retrieve the session cookie, if so, how?
Another question, is it possible to get the answer from an .emit without going through an #sio.event or #sio.on()? As with the websockets library:
websocket.send('Hi')
response = websocket.rcv()
print(response) -> "Hi :)"
If not, is it possible to create an event/on that retrieves messages from the server that do not contain an event name?
For example I send from the client: sio.emit("GetNbrPlayers") but the server answers me [{"data": "5"}] without an event before the data (it is a server/api that uses socket.io too), I would like to get this message but impossible with an event/on because there is no event name in the answer.
Thank you in advance!
it is possible to retrieve the session cookie,
What do you mean by session cookie? If you mean something like the Flask session cookie, then no, there is no session cookie. The user session is stored in the server, not in a cookie.
If you mean the sid cookie that contains the session id assigned to the client, you can obtain this id from the sid attribute of your client object, for example as sio.sid.
is it possible to get the answer from an .emit without going through an #sio.event or #sio.on()?
Yes, you can use the "ack" feature of the Socket.IO protocol for this. If the server is a Python server, you can just return the response from your handler function. For example:
#sio.event
def GetNbrPlayers():
return [{"data": "5"}]
In the Python client you have two ways to receive this information. You can use a callback function:
def get_response(data):
print(data)
sio.emit("GetNbrPlayers", callback=get_response)
Or you can use call() instead of emit() to combine the emit and the callback handling into a single function call:
data = sio.call("GetNbrPlayers")
So we have a single thread flask server running where we receive requests from a python app client. In this flask server we use rabbitMQ with pika library to distribute messages to other clients.
What is happening is that in the get function the program is crashing with the error:
pika.exceptions.ConnectionClosed: (505, 'UNEXPECTED_FRAME - expected
content header for class 60, got non content header frame instead')
I've searched a lot of topics about this in stack overflow and others but they all address problems with multi threading which is not the case. Flask should only serve with one thread unless it is called in app.run(threaded=yes).
The program normally crashes when multiple messages are sent in a short interval (e.g. 5 per second) and it's also important to note that messages are being received every second with a request to this function:
#app.route('/api/users/getMessages', methods=['POST'])
def get_Messages():
data = json.loads(request.data)
token = data['token']
payload = jwt.decode(token, 'SECRET', algorithms=['HS256'])
istid = payload['istid']
print('istid: '+istid)
messages = []
queue = channel.queue_declare(queue=istid)
for i in range(queue.method.message_count):
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
if method_frame:
#print(method_frame, header_frame, body)
messages.append(body)
else:
print('No message returned')
res = {'messages':messages, 'error':0}
return jsonify(res)
In this code it crashes normally in the line:
queue = channel.queue_declare(queue=istid)
But we also tried to change the code to use a while instead of a for where it ends when the body is None and it crashes in the line:
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
in that case.
Also important, the crashes are random and it can work a few times and then randomly crashes after a get request while messages are being sent. If anyone knows anything related to this we would appreciate any help.
Another note, we thought about using basic_consume with callback instead of basic_get but we didn't find a way in which this would work since we have to send the messages back and have several user making requests to this same function.
EDIT #1:
In the rabbitMQ docs rabbitmq if you search for the function "def basic_get" you will notice there are some TODO comments and also a reference to this
Due to implementation details, this cannot be called a second time
until the callback is executed.
So I suspected that this could be what was happening but even if it is I don't know how could it be solved.
For anyone interested in the solution, as it is in the other comments, the program was not thread safe since flask as of version 1.0 uses threaded = True as default.
The solution is either:
1) running flask with app.run(threaded = False)
2) Making the program thread safe by implementing locks whenever accessing the channel /connection with pika.
I'm trying to build client-server app in Python.
My client use requests module to connect to the server and upload json and files.
Server use tornado framework. When server receive data from client, they start processing and send result to client by parts.
Example of my post handler:
class PostAd(tornado.web.RequestHandler):
def post(self):
jdata = self.get_body_arguments('json', False)[0]
jdata = json.loads(jdata)
id = self.insert_ad(jdata)
fpath_list = self.save_files(self.request.files.values(), id)
self.insert_file_path(id, fpath_list)
self.write("Successfully posted into SQL with sql id: {0}".format(id))
self.flush()
self.write("Are u there?")
self.finish()
in the client requests used to post data
r=agent.post("http://localhost:8888/api/v1/add-ad", data={"json": thread_data}, files=files)
in this way I cannot receive data by pieces because r=agent.post will wait until server will close connection but I need to check returned values every time when tornado server will send me data with self.flush() command (in my example I expect to get two answers, first one: "Successfully posted into SQL with sql id: 100" and second: "Are u there?").
is it possible to do it with requests module or I need to use something else here?
I don't know what agent.post() is, but you can do this with tornado's HTTP client and the streaming_callback option. You'll have to format the request body yourself, though, since Tornado doesn't have built-in client-side support for multipart file uploads.
await AsyncHTTPClient().fetch(url, body=encoded_body, streaming_callback=print)
There is no guarantee that the chunks observed by streaming_callback will align with the calls to flush, so you should format the data so that the client can determine where messages begin or end.
I have the following scenario I would like to implement:
User surfs to our website
User enters a bitcoin address.
A websocket is created to the server, passing the address.
The server registers a callback with Blocktrail
When the callback is triggered (a payment was seen by Blocktrail) we send a message back to the browser.
The page the user is browsing is updated to show the message recieved
I'm using webhooks from the Blocktrail API to "listen" to an event, being the reception of coins on an address.
Now, when the event happens, the API does a POST to my URL. This should send a message to the browser that is connected to my server with socket.io (such as 'payment seen on blockchain')
So the question is,
How can I send a message from a route to a socket using flask-socketio
Pseudo code:
#app.route('/callback/<address>')
def callback(id):
socketio.send('payment seen on blockchain')
#socketio.on('address',address)
def socketlisten(address):
registerCallback(address)
I'm going to describe how to solve this using Flask-SocketIO beta version 1.0b1. You can also do this with the 0.6 release, but it is a bit more complicated, the 1.0 release makes addressing individual clients easier.
Each client of a socket connection gets assigned a session id that uniquely identifies it, the so called sid. Within a socket function handler, you can access it as request.sid. Also, upon connection, each client is assigned to a private room, named with the session id.
I assume the metadata that you receive with the callback allows you to identify the user. What you need is to obtain the sid of that user. Once you have it, you can send your alert to the corresponding room.
Example (with some hand-waving regarding how you attach a sid to an address):
#app.route('/callback/<address>')
def callback(address):
sid = get_sid_from_address(address)
socketio.send('payment seen on blockchain', room=sid)
#socketio.on('address')
def socketlisten(address):
associate_address_with_sid(address, request.sid)