I'm working on a small flask app.
The idea is when a user clicks on "update all", a utility class submits all defined servers to the updater which calls each server's "get()" method.
route.py:
#servers.route('/update/<id>')
def update(id):
servers = UcxServer.query.filter(id != None)
def update_server(server):
server.create_ucx()
server.update_ucx()
return threading.current_thread().name, server
with Pool(max_workers=5) as executor:
start = timer()
for name, server in executor.map(update_server, servers):
print("%s %s" % (name, server)) # print results
db.session.add(server)
db.session.commit()
print('time:', timer() - start)
flash('All servers have been updated')
return redirect(url_for('servers.index'))
The problem appears when this button is used multiple times in that it keeps spawning new threads. If I have 5 servers, first time I use the button I will get 5 threads. Next time 10, and so on.
What is the proper way to do this thread management so that I don't end up with a million threads after the apps has been up for a spell?
Thanks
Related
I have below code in standalone script which is using django orm (outside django) with multithreading.
import threading
MAX_THREADS = 30
semaphore = threading.Semaphore(value=MAX_THREADS)
books = Books.objects.all()
for book in books:
book_id = book.id
t = threading.Thread(target=process_book, args=[book_id])
t.start()
threads.append(t)
for t in threads:
t.join()
def process_book(book_id):
semaphore.acquire()
book = Books.objects.get(id=book_id)
# Do some time taking stuff here
book.save()
semaphore.release()
Once the number of threads reaches MAX_CLIENT_CONN setting of postgres (which is 100 by default), I start getting following error with further calls:
operationalError at FATAL: remaining connection slots are reserved for
non-replication superuser connections
Researching this I got to solutions of using a database pooler like pgbouncer, However that only stalls the new connections until connections are available but again after django's query wat timeout I hit
OperationalError at / query_wait_timeout server closed the
connection unexpectedly This probably means the server terminated
abnormally before or while processing the request.
I understand that this is happening because the threads are not closing the db connections they are making but I am not sure how to even close the orm call connections? Is there something I could be doing differently in above code flow to reduce the number of connections?
I need to do a get on individual instances in order to update them because .update() or .save() wont work on queryset items.
this update the field in all the books on the database
for book in books:
Books.objects.filter(id=book.id).bulk_update(book, ['field to update'])
update each single book
def process_book(book_id):
semaphore.acquire()
book = get_object_or_404(Books, id=book_id).update(field) # Do some time taking stuff here
semaphore.release()
Just close the database connections at the end of your threads
from django import db
def process_book(book_id):
semaphore.acquire()
book = Books.objects.get(id=book_id)
# Do some time taking stuff here
book.save()
semaphore.release()
db.connections.close_all()
I have a worker thread which only task is to query a list of active users every 10 minutes from the database, and to send them an SMS message if a certain condition is fulfilled (which is checked every minute); also the worker thread does not hinder the main application at all.
So far I managed to get the thread up and running and sending SMS works also just fine. However, for some reasons the thread stops/gets killed after some random time (hours). I run a try: except Exception as e: within a while True, to catch occurring errors. Additionally, I print out a messages saying what error occurred.
Well, I never see any message and the thread is definitely down. Therefore, I suspect Gunicorn or Django to kill my thread sort of gracefully.
I have put log and print statements all over the code but haven't found anything indicating why my thread is getting killed.
My wsgi.py function where I call the function to start my thread
"""
WSGI config for django_web project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/2.1/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_web.settings')
application = get_wsgi_application()
'''
Start background services
Import has to happen after "get_wsgi_application()"; otherwise docker container crashes
'''
try:
from threading import Thread
from timesheet.admin import runWorkmateServices
runWorkmateServices()
except Exception as exp:
print(exp)
The function which is called from within the wsgi.py. I double check if the thread was started to avoid having two up and running.
def runWorkmateServices(request=None):
service_name = 'TimeKeeperWorkMateReminderService'
thread_found = False
for thread in threading.enumerate():
if service_name in thread.name:
thread_found = True
break # Leave loop now
if thread_found:
print(f'Service has already been started: {service_name}')
if request:
messages.add_message(request, messages.ERROR, f'Service has already been started:: {service_name}')
else:
Thread(target=WorkMateReminders, args=(), name=service_name, daemon=True).start()
print(f'Started Service: {service_name}')
if request:
messages.add_message(request, messages.SUCCESS, f'Started Service: {service_name}')
The worker thread itself
def WorkMateReminders():
print('Thread Started: WorkMateReminders')
timer = 0
employees = User.objects.none()
while True:
try:
# Update user list every n * sleep time (10 minutes)
if timer % 10 == 0:
timer = 0
# Get active employees
employees = User.objects.filter(is_active=True, profile__workmate_sms_reminders_activated=True)
print(f'Employees updated at {datetime.now().date()} - {datetime.now().time()}: {employees}')
WorkMateCheckClockOffTimes(employees=employees)
WorkMateClockOnReminder(employees=employees)
WorkMateEndOfBreakReminder(employees=employees)
timer += 1 # increment timer
except Exception as exp:
print(f'Error: {exp}')
time.sleep(60 * 1)
My goal is to have this worker thread running for as long as Django is up.
Most WSGI servers spawn workers that are killed/recycled fairly regularly, spawning threads from these workers is not the best solution to your problem. There are several ways to go about this
Cron
Create a management command that does what you want and configure cron to run it every 10 minutes
Celery/Celerybeat
Set up a celery worker, this is a process that runs asynchronously to your Django application and using celerybeat you can have tasks run at intervals
I know this question has been asked prior, but I found no answer that addressed by particular problem.
I have a Python script to create kubernetes clusters and nodes in Azure, which takes anywhere between 5-10 mins. There's a function(get_cluster_end) to get the cluster endpoint, but this fails as the endpoint is not yet ready when this function is call. The func I wrote(wait_for_end)does not seem to be correct.
def wait_for_endpoint(timeout=None):
endpoint = None
start = time.time()
while not endpoint:
if timeout is not None and (time.time() - start > timeout):
break
endpoint = **get_cluster_end()**
time.sleep(5)
return endpoint
My main func:
def main():
create_cluster()
start = time.time()
job.set_progress("Waiting for cluster IP address...")
endpoint = wait_for_endpoint(timeout=TIMEOUT)
if not endpoint:
return ("FAILURE","No IP address returned after {} seconds".format(TIMEOUT),
"")
The script fails, because no endpoint has yet been created. How do I set the sleep after the cluster has been created and before the "wait_for_endpoint()" is called?
I am building an interface using python that takes from a collector table -> hands over to a common interface table -> transform and fire to an external system. I also have poller to update status back to both the interface table and collector table.
#synch.route("/ssr_synch", methods=['GET', 'POST'])
def callSSRSynch():
batch_txn_id = request.args.get('batch_txn_id')
if request.method == 'POST':
batch_txn_id = request.form['batch_txn_id']
if request.form['synch'] == 'synch':
if app.config['stub'] == 'False':
ssr_collector.main()
ssr_provisioning.main() #for interface table
ssr_poller.main()
form = "Synch completed for batch transaction id %s" % (batch_txn_id,)
Currently, after the call is trigger by the request, the collector and provisioning run before the poller. I want the poller to be started independent from the collector and provisioning.
Not sure if threading is the best approach or is there an easier solution?
Update 9-Dec-2015:
Threading seemed to be the simplest way to do this. managed to make this work:
threads = list()
thread1 = threading.Thread(target=ssr_collector.main, args=(batch_txn_id,))
thread2 = threading.Thread(target=ssr_provisioning.main)
thread3 = threading.Thread(target=ssr_poller.main)
threads.append(thread1)
threads.append(thread2)
threads.append(thread3)
for t in threads: t.start()
for t in threads: t.join()
I think that you went in wrong direction. You should separate different functionality to different applications. Application based on flask framework is your entry point where client can run new task or get information about tasks which currently running or already done. This application should not run task itself or block client connection until task done! There are plenty nice tools which was created special for this use case, Celery for example.
Api method:
#synch.route("/ssr_synch", methods=['POST'])
def callSSRSynch():
batch_txn_id = request.args.get('batch_txn_id')
if request.method == 'POST':
batch_txn_id = request.form['batch_txn_id']
if request.form['synch'] == 'synch':
if app.config['stub'] == 'False':
collector.delay()
provisioning.delay()
poller.delay()
form = "Synching for batch transaction id %s" % (batch_txn_id,)
Tasks:
#celery.task
def collector():
ssr_collector.main()
#celery.task
def provisioning():
ssr_provisioning.main()
#celery.task
def poller():
ssr_poller.main()
I have a pipeline where a request hits one server, and that server calls another server and that one executes a job for two seconds, then should return to the main server for it to do some minor computation, then return to the client. The problem is that my current setup blocks if number of concurrent requests > number of workers, and I don't know how to use Python threading to make it async. Any ideas on how to implement this?
Main server -> Outside server -> 2 seconds -> Main server
:Edit
The line that takes 2 seconds is the one with the "find_most_similar_overall(image_name, classifier, labels)" call. That function takes 2 seconds, which means that the worker stops right there.
#app.route("/shoes/<_id>")
def classify_shoe(_id):
if request.method == 'GET':
unique_user = request.cookies.get('uniqueuser')
shoe = Shoe.query.filter_by(id = _id)
if shoe.count() is not 0:
shoe = shoe.first()
image_name = shoe.image_name
shoes,category = find_most_similar_overall(image_name, classifier, labels)
return render_template('category.html', same_shoes = similar_shoes,shoes=shoes,main_shoe=shoe,category=category, categories=shoe_categories)