I'm developing a testing suite for a flask app using celery for processing background tasks.
I am working on integration tests and have been trying to configure a embedded live worker as per the documentation (https://docs.celeryproject.org/en/latest/userguide/testing.html)
conftest.py
#pytest.fixture(scope='session')
def celery_config():
return {
'broker_url': 'memory://localhost/',
'result_backend': 'memory://localhost/',
}
#pytest.fixture(scope='module')
def create_flask_app():
#drop all records in testDatabase before strting new test module
db = connect(host=os.environ["MONGODB_SETTINGS_TEST"], alias="testConnect")
for collection in db["testDatabase"].list_collection_names():
db["testDatabase"].drop_collection(collection)
db.close()
# Create a test client using the Flask application configured for testing
flask_app = create_app()
return flask_app
#pytest.fixture(scope='function')
def test_client(create_flask_app):
"""
Establish a test client for use within each test module
"""
with create_flask_app.test_client() as testing_client:
with create_flask_app.app_context():
yield testing_client
#pytest.fixture(scope='function')
def celery_app(create_flask_app):
from celery.contrib.testing import tasks
from app import celery
return celery
I'm trying to run the tests using local memory as the backend. Yet, the tasks hang and the test suite never finishes executing.
When I run the tests with a redis backend (and initialize redis in my development machine) everything works fine. But I'd like to not be dependent on redis when running the tests.
Am I doing something wrong with the setup? Does anyone have any idea on why the tasks are hanging?
Related
I have a Flask application that needs to run code on startup. In my case, it detects available cloud resources and writes those to a table in a database.
The problem is that when I run flask db upgrade, then flask-migrate executes the application startup code, including the code that tries to write to the db table. Since the migration itself is what creates the table, the table doesn't exist yet, and the migration fails.
Here is the code, with irrelevant parts removed:
def create_app():
app = Flask(__name__, static_url_path=None)
with app.app_context():
db = set_up_database_connection()
Migrate(app, db)
# This is what fails because the table doesn't exist yet
run_startup_tasks()
#app.get('/')
def health_check():
return 'OK'
app.logger.info("Application created.")
return app
I did a temporary workaround by using the #app.before_first_request annotation. With the workaround, the startup code runs just before the first request comes through. This is not ideal, though, since that causes the first request to the application to take a long time.
#app.before_first_request
def perform_startup_tasks():
run_startup_tasks()
How do I run startup tasks that require the database without breaking flask-migrate?
I ended up solving this problem by just creating a separate file, which runs the startup tasks:
app = Flask(__name__, static_url_path=None)
with app.app_context():
db = set_up_database_connection()
run_startup_tasks()
This file is run before the main application is run, and after the migrations are run:
flask db upgrade && python run_startup_tasks.py && flask run
This fixes the problem, since the main application (which flask-migrate uses) doesn't run startup tasks. It's run in a separate script.
Hello fellow developers,
I'm actually trying to create a small webapp that would allow me to monitor multiple binance accounts from a dashboard and maybe in the futur perform some small automatic trading actions.
My frontend is implemented with Vue+quasar and my backend server is based on python Flask for the REST api.
What I would like to do is being able to start a background process dynamically when a specific endpoint of my server is called. Once this process is started on the server, I would like it to communicate via websocket with my Vue client.
Right now I can spawn the worker and create the websocket communication, but somehow, I can't figure out how to make all the threads in my worker to work all together. Let me get a bit more specific:
Once my worker is started, I'm trying to create at least two threads. One is the infinite loop allowing me to automate some small actions and the other one is the flask-socketio server that will handle the sockets connections. Here is the code of that worker :
customWorker.py
import time
from flask import Flask
from flask_socketio import SocketIO, send, emit
import threading
import json
import eventlet
# custom class allowing me to communicate with my mongoDD
from db_wrap import DbWrap
from binance.client import Client
from binance.exceptions import BinanceAPIException, BinanceWithdrawException, BinanceRequestException
from binance.websockets import BinanceSocketManager
def process_message(msg):
print('got a websocket message')
print(msg)
class customWorker:
def __init__(self, workerId, sleepTime, dbWrap):
self.workerId = workerId
self.sleepTime = sleepTime
self.socketio = None
self.dbWrap = DbWrap()
# this retrieves worker configuration from database
self.config = json.loads(self.dbWrap.get_worker(workerId))
keys = self.dbWrap.get_worker_keys(workerId)
self.binanceClient = Client(keys['apiKey'], keys['apiSecret'])
def handle_message(self, data):
print ('My PID is {} and I received {}'.format(os.getpid(), data))
send(os.getpid())
def init_websocket_server(self):
app = Flask(__name__)
socketio = SocketIO(app, async_mode='eventlet', logger=True, engineio_logger=True, cors_allowed_origins="*")
eventlet.monkey_patch()
socketio.on_event('message', self.handle_message)
self.socketio = socketio
self.app = app
def launch_main_thread(self):
while True:
print('My PID is {} and workerId {}'
.format(os.getpid(), self.workerId))
if self.socketio is not None:
info = self.binanceClient.get_account()
self.socketio.emit('my_account', info, namespace='/')
def launch_worker(self):
self.init_websocket_server()
self.socketio.start_background_task(self.launch_main_thread)
self.socketio.run(self.app, host="127.0.0.1", port=8001, debug=True, use_reloader=False)
Once the REST endpoint is called, the worker is spawned by calling birth_worker() method of "Broker" object available within my server :
from custom_worker import customWorker
#...
def create_worker(self, workerid, sleepTime, dbWrap):
worker = customWorker(workerid, sleepTime, dbWrap)
worker.launch_worker()
def birth_worker(workerid, 5, dbwrap):
p = Process(target=self.create_worker, args=(workerid,10, botPipe, dbWrap))
p.start()
So when this is done, the worker is launched in a separate process that successfully creates threads and listens for socket connection. But my problem is that I can't use my binanceClient in my main thread. I think that it is using threads and the fact that I use eventlet and in particular the monkey_patch() function breaks it. When I try to call the binanceClient.get_account() method I get an error AttributeError: module 'select' has no attribute 'poll'
I'm pretty sure about that it comes from monkey_patch because if I use it in the init() method of my worker (before patching) it works and I can get the account info. So I guess there is a conflict here that I've been trying to resolve unsuccessfully.
I've tried using only the thread mode for my socket.io app by using async_mode=threading but then, my flask-socketio app won't start and listen for sockets as the line self.socketio.run(self.app, host="127.0.0.1", port=8001, debug=True, use_reloader=False) blocks everything
I'm pretty sure I have an architecture problem here and that I shouldn't start my app by launching socketio.run. I've been unable to start it with gunicorn for example because I need it to be dynamic and call it from my python scripts. I've been struggling to find the proper way to do this and that's why I'm here today.
Could someone please give me a hint on how is this supposed to be achieved ? How can I dynamically spawn a subprocess that will manage a socket server thread, an infinite loop thread and connections with binanceClient ? I've been roaming stack overflow without success, every advice is welcome, even an architecture reforge.
Here is my environnement:
Manjaro Linux 21.0.1
pip-chill:
eventlet==0.30.2
flask-cors==3.0.10
flask-socketio==5.0.1
pillow==8.2.0
pymongo==3.11.3
python-binance==0.7.11
websockets==8.1
import sys
import os
import logging
import requests
import json
import pytest
from multiprocessing import Process
from flask_file import main
#pytest.fixture
def endpoint():
return "http://127.0.0.1:8888/"
def test_send_request(endpoint: str):
server = Process(target=main)
server.start()
time.sleep(30)
# check that the service is up and running
service_up = requests.get(f"{endpoint}")
server.terminate()
server.join()
I wanted to spin up and spin down a server locally from within a test to test some requests. I know the server itself works because I can run main() from the flask_file itself using python flask_file and it will spin up the server...and I can ping it just fine. When I use the above method, the test does seem to do the full 30s sleep without failing, but in those 30s I cannot open the endpoint on my browser and see the expected "hello world".
When you run the Flask builtin development server (e.g. flask run or app.run(), only one connection is possible. So when your test accesses the app, you cannot access it via browser.
Anyway, you should rewrite your test and fixture to use the test_client, see the official documentation
https://flask.palletsprojects.com/en/1.1.x/testing/
I am trying to add websocket functionality to an existing application. The existing structure of the app is
In /server/__init__.py:
from connexion import App
...
connexion_app = App(__name__, specification_dir='swagger/') # Create Connexion App
app = connexion_app.app # Configure Flask Application
...
connexion_app.add_api('swagger.yaml', swagger_ui=True) # Initialize Connexion api
In startserver.py:
from server import connexion_app
connexion_app.run(
processes=8,
debug=True
)
In this way, I was able to specify the number of processes. There are some long-running tasks that make it necessary to have as many processes as possible.
I have modified the application to include websocket functionality as below. It seems to be that I only have one process available. Once the application attempts to run one of the long-running processes, all API calls hang. Also, if the long-runnign process fails, the application is stuck in a hanging state
In /server/__init__.py:
from connexion import App
import socketio
...
connexion_app = App(__name__, specification_dir='swagger/') # Create Connexion App
sio = socketio.Server() # Create SocketIO for websockets
app = connexion_app.app # Configure Flask Application
...
connexion_app.add_api('swagger.yaml', swagger_ui=True) # Initialize Connexion api
In startserver.py:
import socketio
import eventlet
from server import sio
from server import app
myapp = socketio.Middleware(sio, app)
eventlet.wsgi.server(eventlet.listen(('', 5000)), myapp)
What am I missing here?
(side note: If you have any resources available to better understand the behemoth of the Flask object, please point me to them!!)
Exact answer to question: Eventlet built-in WSGI does not support multiple processes.
Approach to get the best solution for described problem: share one file that contains absolute minimum code required to reproduce problem. Maybe here https://github.com/eventlet/eventlet/issues or any other way you prefer.
Way of hope. Random stuff to poke at: eventlet.monkey_patch(), isolate Eventlet and long blocking calls in separate threads or processes.
I have a very simple Celery task that runs a (long running) shell script:
import os
from celery import Celery
os.environ['CELERY_TIMEZONE'] = 'Europe/Rome'
os.environ['TIMEZONE'] = 'Europe/Rome'
app = Celery('tasks', backend='redis', broker='redis://OTHER_SERVER:6379/0')
#app.task(name='ct.execute_script')
def execute_script(command):
return os.system(command)
I have this task running on server MY_SERVER and I launch it from OTHER_SERVER where is also running the Redis database.
The task seems to run successfully (I see the result of executing the script on the filesystem) but the I always start getting the following error:
INTERNAL ERROR: ConnectionError('Error 111 connecting to localhost:6379. Connection refused.',)
What could it be? Why is it trying to contact localhost while I've set the Redis server to be redis://OTHER_SERVER:6379/0 and it works (since the task is launched)? Thanks
When you set the backend argument, Celery will use it as the result backend.
On your code, you tell Celery to use local redis server as the result backend.
You seen ConnectionError， because celery can't save the reult to local redis server.
You can disable result backend or start an local redis server or set it to OTHER_SERVER.
ref:
http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html#keeping-results
http://celery.readthedocs.org/en/latest/configuration.html#celery-result-backend