I have a pipeline where a request hits one server, and that server calls another server and that one executes a job for two seconds, then should return to the main server for it to do some minor computation, then return to the client. The problem is that my current setup blocks if number of concurrent requests > number of workers, and I don't know how to use Python threading to make it async. Any ideas on how to implement this?
Main server -> Outside server -> 2 seconds -> Main server
:Edit
The line that takes 2 seconds is the one with the "find_most_similar_overall(image_name, classifier, labels)" call. That function takes 2 seconds, which means that the worker stops right there.
#app.route("/shoes/<_id>")
def classify_shoe(_id):
if request.method == 'GET':
unique_user = request.cookies.get('uniqueuser')
shoe = Shoe.query.filter_by(id = _id)
if shoe.count() is not 0:
shoe = shoe.first()
image_name = shoe.image_name
shoes,category = find_most_similar_overall(image_name, classifier, labels)
return render_template('category.html', same_shoes = similar_shoes,shoes=shoes,main_shoe=shoe,category=category, categories=shoe_categories)
Related
I have a flask app which I need to start and then shut down within the same process and repeat this multiple times. I am using gevents in my application so I'm using gevents.pywsgi as my WSGI server. Now I'm trying to gracefully shutdown the server so I can restart it in the same process after I do a few other things.
So the following code creates a basic flask app, which when receives a POST request with any valid data on /hit endpoint populates the data field. A greenlet is running in parallel with this app and when it sees that the data field is populated, it shuts down the server.
def func():
global data
data = None
app = Flask(__name__)
#app.route('/hit', methods=['POST'])
def hit():
global data
data = request.json
if data is not None:
return "Input Recieved, Server closed "
else:
return "Invalid Input, Try again"
def shutdown_server(_server):
global data
while data is None:
sleep(0.5)
_server.stop()
_server.close()
server = WSGIServer(('0.0.0.0',5100), app)
start = spawn(server.start)
stop = spawn(shutdown_server, request, server)
joinall([start, stop])
return True
Now this code is running fine if I run the server once, but if I try to run the server again within the same process it throws the following error:
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
File "/home/batman/Documents/genisys/lib/python3.6/site-packages/gevent/baseserver.py", line 308, in start
self.start_accepting()
File "/home/batman/Documents/genisys/lib/python3.6/site-packages/gevent/baseserver.py", line 160, in start_accepting
self._watcher = self.loop.io(self.socket.fileno(), 1)
AttributeError: 'WSGIServer' object has no attribute 'socket'
I'm not sure if this is even possible, or if I have to run the server in a separate process if I want to run it multiple times. Can anybody tell me why I'm facing this error and is there a better and cleaner way to shutdown the server so I don't face this error?
EDIT : I give IP and port as arguments to func, so I have tried call func with a different port multiple times and with the same port multiple times, I still get the same error.
I think problem is you already running the server on port 5100 so again you can't run process on same port , so you have to dynamically change the port numbers when you want start the process
I know this question has been asked prior, but I found no answer that addressed by particular problem.
I have a Python script to create kubernetes clusters and nodes in Azure, which takes anywhere between 5-10 mins. There's a function(get_cluster_end) to get the cluster endpoint, but this fails as the endpoint is not yet ready when this function is call. The func I wrote(wait_for_end)does not seem to be correct.
def wait_for_endpoint(timeout=None):
endpoint = None
start = time.time()
while not endpoint:
if timeout is not None and (time.time() - start > timeout):
break
endpoint = **get_cluster_end()**
time.sleep(5)
return endpoint
My main func:
def main():
create_cluster()
start = time.time()
job.set_progress("Waiting for cluster IP address...")
endpoint = wait_for_endpoint(timeout=TIMEOUT)
if not endpoint:
return ("FAILURE","No IP address returned after {} seconds".format(TIMEOUT),
"")
The script fails, because no endpoint has yet been created. How do I set the sleep after the cluster has been created and before the "wait_for_endpoint()" is called?
I am encountering an undesired delay in the celery process that I cannot explain. My intent is to manage live processing of incoming data (at a rate of 10 to 60 data per seconds). Processing of one piece of data is divided into two fully sequential tasks but parallelization is used to start processing the next piece of data (with task 1) while processing the current one (with task 2) is not finished yet. Getting the shortest delay in the process is of at-most importance since it is a live application.
Once in a while, I encounter a freeze in the process. To see where this problem came from I started monitoring the occupation of my workers. It appeared that it happened during the communication between workers. I designed the lightest and simplest example to illustrate it here.
Here is my code, as you can see I have two tasks doing nothing but waiting 10ms each. I call them by using celery chains once every 20ms. I track each workers occupation by using prerun and postrun along with logging. In most of the case all is happening sequentially as time spent by both the workers doesn't exceed the send rate.
from __future__ import absolute_import
import time
from celery import chain
from celery.signals import task_prerun, task_postrun
from celery import Celery
from kombu import Queue, Exchange
N_ITS = 100000 # Total number of chains sent
LOG_FILE = 'log_file.txt' # Path to the log file
def write_to_log_file(text):
with open(LOG_FILE, 'a') as f:
f.write(text)
# Create celery app
app = Celery('live')
app.config_from_object('celeryconfig')
default_exchange = Exchange('default', type='direct')
app.conf.task_queues = tuple(Queue(route['queue'], default_exchange, routing_key=route['queue'])
for route in app.conf.task_routes.values() + [{'queue': 'default'}])
app.conf.update(result_expires=3600)
# Define functions that record timings
#task_prerun.connect()
def task_prerun(signal=None, sender=None, task_id=None, task=None, **kwargs):
text = 'task_prerun; {0}; {1:.16g}\n'.format(task.name, time.time())
write_to_log_file(text)
#task_postrun.connect()
def task_postrun(signal=None, sender=None, task_id=None, task=None, **kwargs):
text = 'task_postrun; {0}; {1:.16g}\n'.format(task.name, time.time())
write_to_log_file(text)
# Define tasks
#app.task
def task_1(i):
print 'Executing task_1: {}'.format(i)
time.sleep(0.01)
#app.task
def task_2(i):
print 'Executing task_2: {}'.format(i)
time.sleep(0.01)
# Send chained tasks
def main():
celery_chains = []
for i in range(N_ITS):
print '[{}] - Dispatching tasks'.format(i)
celery_chains.append(chain(task_1.si(i) | task_2.si(i))())
time.sleep(0.02)
# wait for all tasks to complete
[c.get() for c in celery_chains]
if __name__ == '__main__':
main()
I also give the configuration of celery if needed:
from __future__ import absolute_import
import os
name = 'live'
broker_url = 'pyamqp://{}'.format(os.environ.get('RMQ_HOST', 'localhost'))
print 'broker_url:', broker_url
include = ['live']
DEFAULT_QUEUE = 'celery'
# A named queue that's not already defined in task_queues will be created automatically.
task_create_missing_queues = True
broker_pool_limit = 10000
task_routes = {
'live.task_1': {'queue': 'worker_1'},
'live.task_2': {'queue': 'worker_2'}
}
# We always set the routing key to be the queue name so we do it here automatically.
for v in task_routes.values():
v.update({'routing_key': v['queue']})
task_serializer = 'pickle'
result_serializer = 'pickle'
accept_content = ['json', 'pickle']
timezone = 'Europe/Paris'
enable_utc = True
For the broker, I use the docker image rabbitmq:3.6-alpine with basic configurations appart that I enabled rabbitmq_management.
This resuts in the following worker occupation chronogram: (the color indicates the index of the data being processed, so you can link tasks belonging to the same chain)
As you can see, usually everything goes well and task 2 is called right after task 1 is finished. However, sometimes (indicated by the arrows on the figure) task 2 doesn't start immediately even though worker 2 isn't occupied. It imputes a delay of 27ms, which is more than twice the duration of a single task. This happened approximately every 2 seconds during this execution.
I made some additionnal investigation using firehose to study the message exchange in rabbitmq and it resulted that the messages are effectively sent on time. To my understanding, the worker waits to go fetch the message and process the task, but I cannot understand why.
I tried setting the broker pool limit to a high number but the issue remains.
Context
I have a server called "server.py" that functions as a post-commit webhook from GitLab.
Within "server.py", there is a long-running process (~40 seconds)
SSCCE
#!/usr/bin/env python
import time
from flask import Flask, abort, jsonify
debug = True
app = Flask(__name__)
#app.route("/", methods=['POST'])
def compile_metadata():
# the long running process...
time.sleep(40)
# end the long running process
return jsonify({"success": True})
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8082, debug=debug, threaded=True)
Problem Statement
GitLab's webhooks expect return codes to be returned quickly. Since my webhook returns after or around 40 seconds; GitLab sends a retry sending my long running process in a loop until GitLab tries too many times.
Question
Am I able to return a status code from Flask back to GitLab, but still run my long running process?
I've tried adding something like:
...
def compile_metadata():
abort(200)
# the long running process
time.sleep(40)
but abort() only supports failure codes.
I've also tried using #after_this_request:
#app.route("/", methods=['POST'])
def webhook():
#after_this_request
def compile_metadata(response):
# the long running process...
print("Starting long running process...")
time.sleep(40)
print("Process ended!")
# end the long running process
return jsonify({"success": True})
Normally, flask returns a status code only from python's return statement, but I obviously cannot use that before the long running process as it will escape from the function.
Note: I am not actually using time.sleep(40) in my code. That is there only for posterity, and for the SSCCE. It will return the same result
Have compile_metadata spawn a thread to handle the long running task, and then return the result code immediately (i.e., without waiting for the thread to complete). Make sure to include some limitation on the number of simultaneous threads that can be spawned.
For a slightly more robust and scalable solution, consider some sort message queue based solution like celery.
For the record, a simple solution might look like:
import time
import threading
from flask import Flask, abort, jsonify
debug = True
app = Flask(__name__)
def long_running_task():
print 'start'
time.sleep(40)
print 'finished'
#app.route("/", methods=['POST'])
def compile_metadata():
# the long running process...
t = threading.Thread(target=long_running_task)
t.start()
# end the long running process
return jsonify({"success": True})
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8082, debug=debug, threaded=True)
I was able to achieve this by using multiprocessing.dummy.Pool. After using threading.Thread, it proved unhelpful as Flask would still wait for the thread to finish (even with t.daemon = True)
I achieved the result of returning a status code before the long-running task like such:
#!/usr/bin/env python
import time
from flask import Flask, jsonify, request
from multiprocessing.dummy import Pool
debug = True
app = Flask(__name__)
pool = Pool(10)
def compile_metadata(data):
print("Starting long running process...")
print(data['user']['email'])
time.sleep(5)
print("Process ended!")
#app.route('/', methods=['POST'])
def webhook():
data = request.json
pool.apply_async(compile_metadata, [data])
return jsonify({"success": True}), 202
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8082, debug=debug, threaded=True)
When you want to return a response from the server quickly, and still do some time consuming work, generally you should use some sort of shared storage like Redis to quickly store all the stuff you need, then return your status code. So the request gets served very quickly.
And have a separate server routinely work that semantic job queue to do the time consuming work. And then remove the job from the queue once the work is done. Perhaps storing the final result in shared storage as well. This is the normal approach, and it scales very well. For example, if your job queue grows too fast for a single server to keep up with, you can add more servers to work that shared queue.
But even if you don't need scalability, it's a very simple design to understand, implement, and debug. If you ever get an unexpected spike in request load, it just means that your separate server will probably be chugging away all night long. And you have peace of mind that if your servers shut down, you won't lose any unfinished work because they're safe in the shared storage.
But if you have one server do everything, performing the long running tasks asynchronously in the background, I guess maybe just make sure that the background work is happening like this:
------------ Serving Responses
---- Background Work
And not like this:
---- ---- Serving Responses
---- Background Work
Otherwise it would be possible that if the server is performing some block of work in the background, it might be unresponsive to a new request, depending on how long that time consuming work takes (even under very little request load). But if the client times out and retries, I think you're still safe from performing double work. But you're not safe from losing unfinished jobs.
I'm working on a small flask app.
The idea is when a user clicks on "update all", a utility class submits all defined servers to the updater which calls each server's "get()" method.
route.py:
#servers.route('/update/<id>')
def update(id):
servers = UcxServer.query.filter(id != None)
def update_server(server):
server.create_ucx()
server.update_ucx()
return threading.current_thread().name, server
with Pool(max_workers=5) as executor:
start = timer()
for name, server in executor.map(update_server, servers):
print("%s %s" % (name, server)) # print results
db.session.add(server)
db.session.commit()
print('time:', timer() - start)
flash('All servers have been updated')
return redirect(url_for('servers.index'))
The problem appears when this button is used multiple times in that it keeps spawning new threads. If I have 5 servers, first time I use the button I will get 5 threads. Next time 10, and so on.
What is the proper way to do this thread management so that I don't end up with a million threads after the apps has been up for a spell?
Thanks