So we have a single thread flask server running where we receive requests from a python app client. In this flask server we use rabbitMQ with pika library to distribute messages to other clients.
What is happening is that in the get function the program is crashing with the error:
pika.exceptions.ConnectionClosed: (505, 'UNEXPECTED_FRAME - expected
content header for class 60, got non content header frame instead')
I've searched a lot of topics about this in stack overflow and others but they all address problems with multi threading which is not the case. Flask should only serve with one thread unless it is called in app.run(threaded=yes).
The program normally crashes when multiple messages are sent in a short interval (e.g. 5 per second) and it's also important to note that messages are being received every second with a request to this function:
#app.route('/api/users/getMessages', methods=['POST'])
def get_Messages():
data = json.loads(request.data)
token = data['token']
payload = jwt.decode(token, 'SECRET', algorithms=['HS256'])
istid = payload['istid']
print('istid: '+istid)
messages = []
queue = channel.queue_declare(queue=istid)
for i in range(queue.method.message_count):
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
if method_frame:
#print(method_frame, header_frame, body)
messages.append(body)
else:
print('No message returned')
res = {'messages':messages, 'error':0}
return jsonify(res)
In this code it crashes normally in the line:
queue = channel.queue_declare(queue=istid)
But we also tried to change the code to use a while instead of a for where it ends when the body is None and it crashes in the line:
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
in that case.
Also important, the crashes are random and it can work a few times and then randomly crashes after a get request while messages are being sent. If anyone knows anything related to this we would appreciate any help.
Another note, we thought about using basic_consume with callback instead of basic_get but we didn't find a way in which this would work since we have to send the messages back and have several user making requests to this same function.
EDIT #1:
In the rabbitMQ docs rabbitmq if you search for the function "def basic_get" you will notice there are some TODO comments and also a reference to this
Due to implementation details, this cannot be called a second time
until the callback is executed.
So I suspected that this could be what was happening but even if it is I don't know how could it be solved.
For anyone interested in the solution, as it is in the other comments, the program was not thread safe since flask as of version 1.0 uses threaded = True as default.
The solution is either:
1) running flask with app.run(threaded = False)
2) Making the program thread safe by implementing locks whenever accessing the channel /connection with pika.
Related
I am working with multiple applications that communicate asynchronously using Kafka. These applications are managed by several departments and contract testing is appropriate to ensure that the messages used during communication follow the expected schema and will evolve according to the contract specification.
It sounded like the pact library for python is a good fit because it helps creating contract tests for HTTP and message integrations.
What I wanted to do is to send an HTTP request and to listen from the appropriate and dedicated Kafka topic immediately after. But it seems that the test is forcing me specify an HTTP code even if what I am expecting is a message from a queue without an HTTP status code. Furthermore, it seems that the HTTP request is being sent before the consumer is listening. Here is some sample code.
from pact.consumer import Consumer as p_Consumer
from pact.provider import Provider as p_Provider
from confluent_kafka import Consumer as k_Consumer
pact = p_Consumer('Consumer').has_pact_with(p_Provider('Provider'))
pact.start_service()
atexit.register(pact.stop_service)
config = {'bootstrap.servers':'server', 'group.id':0, 'auto.offset.reset':'latest'}
consumer = k_consumer(config)
consumer.subscribe(['usertopic'])
def user():
while True:
msg = consumer.poll(timeout=1)
if msg is None:
continue
else:
return msg.value().decode()
class ConstractTesting(unittest.TestCase):
expected = {
'username': 'UserA',
'id':123,
'groups':['Editors']
}
pact.given('UserA exists and is not an administrator')
.upon_receiving('a request for UserA')
.with_request(method='GET',path='/user/')
.will_respond_with(200, body=expected)
with pact:
result = user()
self.assertEqual(result,expected)
How would I carry out contract testing in Python using Kafka? It feels like I am going through a lot of hoops to carry out this test.
With Pact message it's a different API you write tests against. You don't use the standard HTTP one, in fact the transport itself is ignored altogether and it's just the payload - the message - we're interested in capturing and verifying. This allows us to test any queue without having to build specific interfaces for each
See this example: https://github.com/pact-foundation/pact-python/blob/02643d4fb89ff7baad63e6436f6a929256c6bf12/examples/message/tests/consumer/test_message_consumer.py#L65
You can read more about message pact testing here: https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
And finally here are some Kafka examples for other languages that may be helpful: https://docs.pactflow.io/docs/examples/kafka/js/consumer
I'm looking for some advice, or a relevant tutorial regarding the following:
My task is to set up a flask route that POSTs to API endpoint X, receives a new endpoint Y in X's response, then GETs from endpoint Y repeatedly until it receives a certain status message in the body of Y's response, and then returns Y's response.
The code below (irrelevant data redacted) accomplishes that goal in, I think, a very stupid way. It returns the appropriate data occasionally, but not reliably. (It times out 60% of the time.) When I console log very thoroughly, it seems as though I have bogged down my server with multiple while loops running constantly, interfering with each other.
I'll also receive this error occasionally:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /book
import sys, requests, time, json
from flask import Flask, request
# create the Flask app
app = Flask(__name__)
# main booking route
#app.route('/book', methods=['POST']) #GET requests will be blocked
def book():
# defining the api-endpoints
PRICING_ENDPOINT = ...
# data to be sent to api
data = {...}
# sending post request and saving response as response object
try:
r_pricing = requests.post(url = PRICING_ENDPOINT, data = data)
except requests.exceptions.RequestException as e:
return e
sys.exit(1)
# extracting response text
POLL_ENDPOINT = r_pricing.headers['location']
# setting data for poll
data_for_poll = {...}
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
# poll loop, looking for 'UpdatesComplete'
j = 1
poll_json = r_poll.json()
update_status = poll_json['Status']
while update_status == 'UpdatesPending':
time.sleep(2)
j = float(j) + float(1)
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
poll_json = r_poll.json()
update_status = poll_json['Status']
return r_poll.text
This is more of an architectural issue more than a Flask issue. Long-running tasks in Flask views are always a poor design choice. In this case, the route's response is dependent on two endpoints of another server. In effect, apart from carrying the responsibility of your app, you are also carrying the responsibility of another server.
Since the application's design seems to be a proxy for another service, I would recommend creating the proxy in the right way. Just like book() offers the proxy for PRICING_ENDPOINT POST request, create another route for POLL_ENDPOINT GET request and move the polling logic to the client code (JS).
Update:
If you cannot for some reason trust the client (browser -> JS) with the POLL_ENDPOINT information in a hidden proxy like situation, then maybe move the polling to a task runner like Celery or Python RQ. Although, it will introduce additional components to your application, it would be the right way to go.
Probably you get that error because of the HTTP connection time out with your API server that is looping. There are some standards for HTTP time connection and loop took more time that is allowed for the connection. The first (straight) solution is to "play" with Apache configs and increase the HTTP connection time for your wsgi. You can also make a socket connection and in it check the update status and close it while the goal was achieved. Or you can move your logic to the client side.
I am working with slack command (python code is running behind this), it works fine, but this gives error
This slash command experienced a problem: 'Timeout was reached' (error detail provided only to team owning command).
How to avoid this ?
According to the Slack slash command documentation, you need to respond within 3000ms (three seconds). If your command takes longer then you get the Timeout was reached error. Your code obviously won't stop running, but the user won't get any response to their command.
Three seconds is fine for a quick thing where your command has instant access to data, but might not be long enough if you're calling out to external APIs or doing something complicated. If you do need to take longer, then see the Delayed responses and multiple responses section of the documentation:
Validate the request is okay.
Return a 200 response immediately, maybe something along the lines of {'text': 'ok, got that'}
Go and perform the actual action you want to do.
In the original request, you get passed a unique response_url parameter. Make a POST request to that URL with your follow-up message:
Content-type needs to be application/json
With the body as a JSON-encoded message: {'text': 'all done :)'}
you can return ephemeral or in-channel responses, and add attachments the same as the immediate approach
According to the docs, "you can respond to a user commands up to 5 times within 30 minutes of the user's invocation".
After dealing with this issue myself and having my Flask app hosted on Heroku I found that the simplest solution was to use threading. I followed the example from here:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xi-email-support
from threading import Thread
def backgroundworker(somedata,response_url):
# your task
payload = {"text":"your task is complete",
"username": "bot"}
requests.post(response_url,data=json.dumps(payload))
#app.route('/appmethodaddress',methods=['POST','GET'])
def receptionist():
response_url = request.form.get("response_url")
somedata = {}
thr = Thread(target=backgroundworker, args=[somedata,response_url])
thr.start()
return jsonify(message= "working on your request")
All the slow heavy work is performed by the backgroundworker() function. My slack command points to https://myappaddress.com/appmethodaddress where the receptionist() function takes the response_url of the received Slack message and passes it alongside any other optional data to the backgroundworker(). As the process is now split it simply returns the "working on your request" message to your Slack channel pretty much instantly and upon completion backgroundworker() sends the second message "your task is complete".
I too was facing this error frequently:
"Darn – that slash command didn't work (error message: Timeout was reached). Manage the command at slash-command"
I was writing a Slack slash-command "bot" on AWS Lambda that sometimes needed to perform slow operations (invoking other external APIs etc). The Lambda function would take greater than 3 seconds in some cases causing the Timeout was reached error from Slack.
I found #rcoup's excellent answer here and applied it in the context of AWS Lambda. The error doesn't appear any more.
I did this with two separate Lambda functions. One is a "dispatcher" or "receptionist" that greets the incoming Slack slash command with a "200 OK" and returns the simple "Ok, got that" type of message to the user. The other is the actual "worker" Lambda function that starts the long-ish operation asynchronously and posts the result of that operation to the Slack response_url later.
This is the dispatcher/receptionist Lambda function:
def lambda_handler(event, context):
req_body = event['body']
try:
retval = {}
# the param_map contains the 'response_url' that the worker will need to post back to later
param_map = _formparams_to_dict(req_body)
# command_list is a sequence of strings in the slash command such as "slashcommand weather pune"
command_list = param_map['text'].split('+')
# publish SNS message to delegate the actual work to worker lambda function
message = {
"param_map": param_map,
"command_list": command_list
}
sns_response = sns_client.publish(
TopicArn=MY_SNS_TOPIC_ARN,
Message=json.dumps({'default': json.dumps(message)}),
MessageStructure='json'
)
retval['text'] = "Ok, working on your slash command ..."
except Exception as e:
retval['text'] = '[ERROR] {}'.format(str(e))
return retval
def _formparams_to_dict(req_body):
""" Converts the incoming form_params from Slack into a dictionary. """
retval = {}
for val in req_body.split('&'):
k, v = val.split('=')
retval[k] = v
return retval
As you can see from the above, I didn't invoke the worker Lambda Function directly from the dispatcher (though this is possible). I chose to use AWS SNS to publish a message that the worker receives and processes.
Based on this StackOverflow answer, this is the better approach as it's non-blocking (asynchronous) and scalable. Also it was easier to use SNS to decouple the two functions in the context of AWS Lambda, direct invocation is trickier for this use-case.
Finally, here's how I consume the SNS event in my worker Lambda Function:
def lambda_handler(event, context):
message = json.loads(event['Records'][0]['Sns']['Message'])
param_map = message['param_map']
response_url = param_map['response_url']
command_list = message['command_list']
main_command = command_list[0].lower()
# process the command as you need to and finally post results to `response_url`
I'm facing problem in emiting messages from RabbitMQ to User via SocketIO.
I have Flask application with SocketIO integration.
Current user flow seems like
The problem is i'm not able to set up RabbitMQ listener which forward messages to browser via SocketIO. Every time i'm getting different error. Mostly is that connection is closed, or i'm working outside of application context.
I tried many approaches, here is my last one.
# callback
def mq_listen(uid):
rabbit = RabbitMQ()
def cb(ch, method, properties, body, mq=rabbit):
to_return = [0] # mutable
message = Message.load(body)
to_return[0] = message.get_message()
emit('report_part', {"data": to_return[0]})
rabbit.listen('results', callback=cb, id=uid)
# this is the page, which user reach
#blueprint.route('/report_result/<uid>', methods=['GET'])
def report_result(uid):
thread = threading.Thread(target=mq_listen, args=(uid,))
thread.start()
return render_template("property/report_result.html", socket_id=uid)
where rabbit.listen method is abstraction like:
def listen(self, queue_name, callback=None, id=None):
if callback is not None:
callback_function = callback
else:
callback_function = self.__callback
if id is None:
self.channel.queue_declare(queue=queue_name, durable=True)
self.channel.basic_qos(prefetch_count=1)
self.consumer_tag = self.channel.basic_consume(callback_function, queue=queue_name)
self.channel.start_consuming()
else:
self.channel.exchange_declare(exchange=queue_name, type='direct')
result = self.channel.queue_declare(exclusive=True)
exchange_name = result.method.queue
self.channel.queue_bind(exchange=queue_name, queue=exchange_name, routing_key=id)
self.channel.basic_consume(callback_function, queue=exchange_name, no_ack=True)
self.channel.start_consuming()
which resulted into
RuntimeError: working outside of request context
I will be happy for any tip or example of usage.
Thanks a lot
I had a similar issue, in the end of the day it's because when you make a request flask passes the request context to client. But the solution is NOT to add with app.app_context(). That is hackey and will definitely have errors as you're not natively sending the request context.
My solution was to create a redirect so that the request context is maintained like:
def sendToRedisFeed(eventPerson, type):
eventPerson['type'] = type
requests.get('http://localhost:5012/zmq-redirect', json=eventPerson)
This is my redirect function, so whenever there is an event I'd like to push to my PubSub it goes through this function, which then pushes to that localhost endpoint.
from flask_sse import sse
app.register_blueprint(sse, url_prefix='/stream')
#app.route('/zmq-redirect', methods=['GET'])
def send_message():
try:
sse.publish(request.get_json(), type='greeting')
return Response('Sent!', mimetype="text/event-stream")
except Exception as e:
print (e)
pass
Now, whenever an event is pushed to my /zmq-redirect endpoint, it is redirected and published via SSE.
And now finally, just to wrap everything up, the client:
var source = new EventSource("/stream");
source.addEventListener(
"greeting",
function(event) {
console.log(event)
}
)
The error message suggests that it's a Flask issue. While handling requests, Flask sets a context, but because you're using threads this context is lost. By the time it's needed, it is no longer available, so Flask gives the "working outside of request context" error.
A common way to resolve this is to provide the context manually. There is a section about this in the documentation: http://flask.pocoo.org/docs/1.0/appcontext/#manually-push-a-context
Your code doesn't show the socketio part. But I wonder if using something like flask-socketio could simplify some stuff... (https://flask-socketio.readthedocs.io/en/latest/). I would open up the RabbitMQ connection in the background (preferably once) and use the emit function to send any updates to connected SocketIO clients.
Given, when a user requests /foo on my server, I send the following HTTP response (not closing the connection):
Content-Type: multipart/x-mixed-replace; boundary=-----------------------
-----------------------
Content-Type: text/html
foo
When the user goes to /bar (which will send 204 No Content so the view doesn't change), I want to send the following data in the initial response.
-----------------------
Content-Type: text/html
bar
How would I get the second request to trigger this from the initial response? I'm planning on possibly creating a fancy [engines that support multipart/x-mixed-replace (currently only Gecko)]-only email webapp that does server-push and Ajax effects without any JavaScript, just for fun.
No complete answer, but:
In your question, you're describing a Comet-style architecture. Regarding support of Comet-style techniques in Python/WSGI, there is a StackOverflow question, which talks about various Python servers with support for long-running requests a la Comet.
Also interesting is this mail thread in the Python Web-SIG: "Could WSGI handle Asynchronous response?". In May 2008, there was a broad discussion in the Web-SIG about the topic of asynchronous requests in WSGI.
A recent development is evserver, a lightweight WSGI server, which implements the Asynchronous WSGI extension proposed by Christopher Stawarz in the Web-SIG in May 2008.
Finally, the Tornado web server supports non-blocking asynchronous requests. It has a chat example application using long polling, which has similarities with your requirements.
If the problem is to pass some command from /bar application to /foo application and you are using some servlet-like approach (the Python code is loaded once and not for each request as in CGI), you can just change some class property of the /foo application and be ready to react to the change in the /foo instance (by checking the property state).
Obviously the /foo application should not return right after the first request and yield content line by line.
Thought this is just theory, I have not tried that myself.
I have created some small example (just for fun, you know :))
import threading
num = 0
cond = threading.Condition()
def app(environ, start_response):
global num
cond.acquire()
num += 1
cond.notifyAll()
cond.release()
start_response("200 OK", [("Content-Type", "multipart/x-mixed-replace; boundary=xxx")])
while True:
n = num
s = "--xxx\r\nContent-Type: text/html\r\n\r\n%s\n" % n
yield s
# wait for num change:
cond.acquire()
while num == n:
cond.wait()
cond.release()
from cherrypy.wsgiserver import CherryPyWSGIServer
server = CherryPyWSGIServer(("0.0.0.0", 3000), app)
try:
server.start()
except KeyboardInterrupt:
server.stop()
# Now whenever you visit http://127.0.0.1:3000/, the number increases.
# It also automatically increases in all previously opened windows/tabs.
The idea of a shared variable and thread synchronization (using condition variable object) is based on the fact that WSGI server provided by CherryPyWSGIServer is threaded.
Not sure if this is quite what you're looking for, but there is a fairly old way of doing server push using a mime content of multipart/x-mixed-replace
Basically you compose the response as a mime object with content type multipart/x-mixed-replace, and send the first "version" of a document down. The browser will keep the socket open.
Then as the server decides to push more data, a new "version" of the document gets sent from the server, and the browser will intelligently replace (within whatever frame/iframe contains the content) the content.
This was an early way of doing webcams, where the server would send down (push) image after image, and the browser would just keep replacing the image in the document over and over. This is also a way of doing a "Loading..." message over a single HTTP request.