Develop Reference

python : dynamically spawn multithread workers with flask-socket io and python-binance

Hello fellow developers,
I'm actually trying to create a small webapp that would allow me to monitor multiple binance accounts from a dashboard and maybe in the futur perform some small automatic trading actions.
My frontend is implemented with Vue+quasar and my backend server is based on python Flask for the REST api.
What I would like to do is being able to start a background process dynamically when a specific endpoint of my server is called. Once this process is started on the server, I would like it to communicate via websocket with my Vue client.
Right now I can spawn the worker and create the websocket communication, but somehow, I can't figure out how to make all the threads in my worker to work all together. Let me get a bit more specific:
Once my worker is started, I'm trying to create at least two threads. One is the infinite loop allowing me to automate some small actions and the other one is the flask-socketio server that will handle the sockets connections. Here is the code of that worker :
customWorker.py
import time
from flask import Flask
from flask_socketio import SocketIO, send, emit
import threading
import json
import eventlet
# custom class allowing me to communicate with my mongoDD
from db_wrap import DbWrap
from binance.client import Client
from binance.exceptions import BinanceAPIException, BinanceWithdrawException, BinanceRequestException
from binance.websockets import BinanceSocketManager
def process_message(msg):
print('got a websocket message')
print(msg)
class customWorker:
def __init__(self, workerId, sleepTime, dbWrap):
self.workerId = workerId
self.sleepTime = sleepTime
self.socketio = None
self.dbWrap = DbWrap()
# this retrieves worker configuration from database
self.config = json.loads(self.dbWrap.get_worker(workerId))
keys = self.dbWrap.get_worker_keys(workerId)
self.binanceClient = Client(keys['apiKey'], keys['apiSecret'])
def handle_message(self, data):
print ('My PID is {} and I received {}'.format(os.getpid(), data))
send(os.getpid())
def init_websocket_server(self):
app = Flask(__name__)
socketio = SocketIO(app, async_mode='eventlet', logger=True, engineio_logger=True, cors_allowed_origins="*")
eventlet.monkey_patch()
socketio.on_event('message', self.handle_message)
self.socketio = socketio
self.app = app
def launch_main_thread(self):
while True:
print('My PID is {} and workerId {}'
.format(os.getpid(), self.workerId))
if self.socketio is not None:
info = self.binanceClient.get_account()
self.socketio.emit('my_account', info, namespace='/')
def launch_worker(self):
self.init_websocket_server()
self.socketio.start_background_task(self.launch_main_thread)
self.socketio.run(self.app, host="127.0.0.1", port=8001, debug=True, use_reloader=False)
Once the REST endpoint is called, the worker is spawned by calling birth_worker() method of "Broker" object available within my server :
from custom_worker import customWorker
#...
def create_worker(self, workerid, sleepTime, dbWrap):
worker = customWorker(workerid, sleepTime, dbWrap)
worker.launch_worker()
def birth_worker(workerid, 5, dbwrap):
p = Process(target=self.create_worker, args=(workerid,10, botPipe, dbWrap))
p.start()
So when this is done, the worker is launched in a separate process that successfully creates threads and listens for socket connection. But my problem is that I can't use my binanceClient in my main thread. I think that it is using threads and the fact that I use eventlet and in particular the monkey_patch() function breaks it. When I try to call the binanceClient.get_account() method I get an error AttributeError: module 'select' has no attribute 'poll'
I'm pretty sure about that it comes from monkey_patch because if I use it in the init() method of my worker (before patching) it works and I can get the account info. So I guess there is a conflict here that I've been trying to resolve unsuccessfully.
I've tried using only the thread mode for my socket.io app by using async_mode=threading but then, my flask-socketio app won't start and listen for sockets as the line self.socketio.run(self.app, host="127.0.0.1", port=8001, debug=True, use_reloader=False) blocks everything
I'm pretty sure I have an architecture problem here and that I shouldn't start my app by launching socketio.run. I've been unable to start it with gunicorn for example because I need it to be dynamic and call it from my python scripts. I've been struggling to find the proper way to do this and that's why I'm here today.
Could someone please give me a hint on how is this supposed to be achieved ? How can I dynamically spawn a subprocess that will manage a socket server thread, an infinite loop thread and connections with binanceClient ? I've been roaming stack overflow without success, every advice is welcome, even an architecture reforge.
Here is my environnement:
Manjaro Linux 21.0.1
pip-chill:
eventlet==0.30.2
flask-cors==3.0.10
flask-socketio==5.0.1
pillow==8.2.0
pymongo==3.11.3
python-binance==0.7.11
websockets==8.1

Invoking flask server from within pytest test

import sys
import os
import logging
import requests
import json
import pytest
from multiprocessing import Process
from flask_file import main
#pytest.fixture
def endpoint():
return "http://127.0.0.1:8888/"
def test_send_request(endpoint: str):
server = Process(target=main)
server.start()
time.sleep(30)
# check that the service is up and running
service_up = requests.get(f"{endpoint}")
server.terminate()
server.join()
I wanted to spin up and spin down a server locally from within a test to test some requests. I know the server itself works because I can run main() from the flask_file itself using python flask_file and it will spin up the server...and I can ping it just fine. When I use the above method, the test does seem to do the full 30s sleep without failing, but in those 30s I cannot open the endpoint on my browser and see the expected "hello world".

When you run the Flask builtin development server (e.g. flask run or app.run(), only one connection is possible. So when your test accesses the app, you cannot access it via browser.
Anyway, you should rewrite your test and fixture to use the test_client, see the official documentation
https://flask.palletsprojects.com/en/1.1.x/testing/

Non-blocking tornado instance on Openshift?

I am trying to create tornado webserver and quick start made me standard project of tornado, but according to documentation this configuration is blocking.
I am new to non-blocking python.
I have this wsgi file that lies in root folder in my PAAS server
#!/usr/bin/env python
import os
import imp
import sys
#
# Below for testing only
#
if __name__ == '__main__':
ip = 'localhost'
port = 8051
zapp = imp.load_source('application', 'wsgi/application')
from wsgiref.simple_server import make_server
httpd = make_server(ip, port, zapp.application)
httpd.serve_forever()
This is main handler file
#!/usr/bin/env python
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.render('index.html')
And Application folder contains this
# Put here yours handlers.
import tornado.wsgi
from . import handlers
handlers = [(r'/',MainHandler),]
application = tornado.wsgi.WSGIApplication(handlers, **settings)
In WSGI mode asynchronous methods are not supported
uses WSGI to deploy the python applications
Is it possible to configure python application on openshift to be fully non-blocking
Though i have seen project that seemed to work

If you are talking about OpenShift V2 (not V3 which uses Kubernetes/Docker), then you need to use the app.py file as described in:
http://blog.dscpl.com.au/2015/08/running-async-web-applications-under.html

How to wrap a zeromq bind socket in a twisted application service?

I am using txzmq and twisted to build a listener service that will process some data through a push-pull pattern. Here's a working code:
from txzmq import ZmqFactory, ZmqEndpoint, ZmqPullConnection
from twisted.internet import reactor
zf = ZmqFactory()
endpoint = ZmqEndpoint('bind', 'tcp://*:5050')
def onPull(data):
# do something with data
puller = ZmqPullConnection(zf, endpoint)
puller.onPull = onPull
reactor.run()
My question is - how can I wrap this code in a twisted application service? That is, how to wrap this into a specific service (e.g. MyService) that I can later run with:
from twisted.application.service import Application
application = Application('My listener')
service = MyService(bind_address='*', port=5050)
service.setServiceParent(application)
with the twistd runner?

IService defines what it means to be a service. Service is a base class that is often helpful when implementing a new service.
Just move your ZMQ initialization code into a startService method of an object that implements IService, perhaps a subclass of Service. If you want to do proper cleanup too, then add some cleanup code to the stopService method of that class.

Using Pyro4 to get the system information

I try to using the Pyro4 on autotesting, but now I confused for some ability for Pyro4.
Does there existed some method to get the system information by the Pyro4 object.
In my ideas, I expose a pyro object that can get the system information, and the remote machine can using this object to show the system information. But on my code, the remote machine always show the server information. I guess that I misunderstand or misuse the Pyro4
Sorry for this stupid question, i'm the newbie for Pyro4. The follow is my sample code.
The server exposes the pyro object
# Server.py
#! /usr/bin/env python
import Pyro4
import os
class Ex(object):
def run(self):
print os.uname()
if __name__ == "__main__":
daemon = Pyro4.Daemon()
ns = Pyro4.locateNS()
uri = daemon.register(Ex())
ns.register("ex", uri)
print uri
daemon.requestLoop()
and the client using the Pyro object
# Remote
#! /usr/bin/env python
import Pyro4
if __name__ == "__main__":
uri = raw_input("input the uri> ")
fn = Pyro4.Proxy(uri)
fn.run()
p.s
I know i can get the os information on the client side, but I want to use the Pyro object to get the information instead client does this itself.

I got the answer!
Just using the Pyro4.utils.flame. This module can provides the ability to using the module or function on remote machine.
Following is the sample code:
The remote machine
python -Wignore -m Pyro4.utils.flameserver --host <yourIP> --port 9999
The host
#! /usr/bin/env python
import Pyro4.utils.flame
if __name__ == "__main__":
flame = Pyro4.utils.flame.connect("192.168.16.29:64024")
os = flame.module("os")
print os.listdir('.')