I'm looking to run a web server that reads a stream on sys.stdin. That reading needs to happen continuously, e.g. in a while loop.
However, I'm also looking to run a Flask server that listens for requests to /data and sends the last bit of data read from sys.stdin to the requesting agent.
So far I've found that my while loop is halting the execution of my app, which makes absolute sense. Here's my setup:
from flask import Flask, jsonify
import sys
# state
frames = []
frame = []
while True:
l = sys.stdin.readline()
if 'end_frame' in l:
frames = [frame] + frames
frame = []
elif l.rstrip('\n'):
frame.append(l.rstrip('\n'))
# app
app = Flask(__name__, static_url_path='')
#app.route('/frames')
def get_frames():
return jsonify(frames)
app.run(host='0.0.0.0', port=5050)
Is there a way to run that while loop as a background process so as to free up the flask route listeners? Any suggestions would be helpful!
You can try running it in a thread
import threading
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for the thread function, define your while loop in it
Try looking into something like BackgroundScheduler. It runs tasks in the background as separate threads without stalling the flask listener.
from apscheduler.schedulers.background import BackgroundScheduler
...
...
def readlines():
l = sys.stdin.readline()
if 'end_frame' in l:
frames = [frame] + frames
frame = []
elif l.rstrip('\n'):
frame.append(l.rstrip('\n'))
with app.app_context():
scheduler = BackgroundScheduler()
scheduler.add_job(readlines, 'interval', seconds=10)
scheduler.start()
I ended up doing the following:
I have a little file publisher.py that reads from a port on a given host (though it could just as well read from sys.stdin using the code above) in a while loop. When it composes a frame, it publishes that frame to a redis datastore.
Then, inside my Flask route listener /frame, I just ping the redis data store and then jsonify the result. In that way my publisher and server work together to provide data to the client as it comes in...
publisher.py:
import sys, socket, redis, json
# config
stream = {'host': '127.0.0.1', 'port': 6000} # streaming data host / port
r = redis.Redis(host='127.0.0.1', port=6379) # redis instance host / port
# consume data from a host+port and publish to redis on localhost
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((socket.gethostbyname(stream['host']), stream['port'])) # host, port
# consume data
frame = [] # initialize the container obj that will hold all frame data
while True:
data = client.recv(1024).decode('utf8')
for l in data.split('\n'):
if 'end_frame' in l:
d = {i.split(':')[0]: i.split(':')[1] for i in frame if ':' in i}
r.set('frame', json.dumps(d))
frame = []
print(' * published frame', d.get('frame_number', ''))
elif l.rstrip('\n'):
frame.append(l.rstrip('\n'))
server.py:
from flask import Flask, jsonify
import redis, sys, os, json
# app
app = Flask(__name__, static_url_path='')
# redis
r = redis.Redis(host='127.0.0.1', port=6379) # redis instance host / port
# route listeners
#app.route('/api/frame')
def get_frame():
frame = json.loads(r.get('frame').decode('utf8'))
return jsonify(frame)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5050)
Related
I am working on a project building an API that is able to send the live location of vehicles to a frontend.
I get this location data by subscribing to a ZMQ stream by running a while loop. This is all working and if I just run my stream as a script I can print all kinds of information to the terminal (I'll store those in a database later on).
I also have the FastAPI server up and running
Now what I'd like to do is:
At startup start the server so I can make API calls
Start the while loop and start receiving data from the ZMQ stream
What happens instead:
It seems either / or.
I can import a function with the while loop but this blocks the server from starting up
Or I can run the server with no means to start the stream
Here is my code:
# General FastAPI Imports
from fastapi import Depends, FastAPI, Request
from data_collection.livestream import enable_data_stream
from client_service import client_api
app = FastAPI()
app.include_router(client_api.router, prefix="/API/V1")
#app.get('/')
def read_root(request: Request):
return {"Hello": "World"}
The Stream:
from gzip import GzipFile
from io import BytesIO
import zmq
import xml.etree.ElementTree as ET
context = zmq.Context()
subscriber = context.socket(zmq.SUB)
subscriber.connect("tcp://SERVER")
subscriber.setsockopt(zmq.SUBSCRIBE)
while True:
multipart = subscriber.recv_multipart()
address = multipart[0]
try:
contents = GzipFile('', 'r', 0, BytesIO(multipart[1])).read()
root = ET.fromstring(contents)
print("Updates Received:")
# Gets the timestamp
print('time', root[3].text)
print('X Coord: ', root[4][0][12].text)
print('Y Coord: ', root[4][0][13].text)
I tried looking into the multiprocess and threading implementations for python but I'm unsure how those tie in with starting the FastAPI process (as that's enabled from Uvicorn)
In the example below, the server and worker are started in separate processes because the While loop won't resolve. It seems that you were on the right track. In my example, I have these functions in one file, but there are no restrictions on someone breaking them out into their own files:
import uvicorn
import multiprocessing
import time
import zmq
import xml.etree.ElementTree as ET
from gzip import GzipFile
from io import BytesIO
from fastapi import FastAPI
app = FastAPI()
#app.get("/")
async def root():
return {"message": "Hello World"}
def server():
uvicorn.run(app, host="localhost", port=8000)
def worker():
context = zmq.Context()
subscriber = context.socket(zmq.SUB)
subscriber.connect("tcp://SERVER")
subscriber.setsockopt(zmq.SUBSCRIBE)
while True:
multipart = subscriber.recv_multipart()
address = multipart[0]
try:
contents = GzipFile('', 'r', 0, BytesIO(multipart[1])).read()
root = ET.fromstring(contents)
print("Updates Received:")
# Gets the timestamp
print('time', root[3].text)
print('X Coord: ', root[4][0][12].text)
print('Y Coord: ', root[4][0][13].text)
except Exception as e:
print(e)
print("Error: %s" % multipart[1])
break
if __name__ == '__main__':
# Runs server and worker in separate processes
p1 = multiprocessing.Process(target=server)
p1.start()
time.sleep(1) # Wait for server to start
p2 = multiprocessing.Process(target=worker)
p2.start()
p1.join()
p2.join()
Every time when I refresh the page from client side a new connection is made with the flask server and it runs the function 'backgroundFunction()' without exiting the recent opened function and the number increases as I refresh the page again and again.
from flask import Flask
from flask_socketio import SocketIO, send, emit
import socket
from time import sleep
import datetime
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret'
app.config['DEBUG'] = True
socketio = SocketIO(app , cors_allowed_origins="*" , async_mode = None , logger = False , engineio_logger = False)
def backgroundFunction():
while True:
data = "I am Data"
socketio.emit('data', data, broadcast=True)
socketio.sleep(2)
#socketio.on('connect')
def socketcon():
print('Client connected')
socketio.start_background_task(backgroundFunction)
if __name__ == ("__main__"):
socketio.run(app, port=5009)
Look at the example code in the Flask-SocketIO repository to learn one possible way to implement a background job that starts the first time an event is triggered.
Code is here. Here is the relevant excerpt:
thread = None
thread_lock = Lock()
def background_thread():
"""Example of how to send server generated events to clients."""
count = 0
while True:
socketio.sleep(10)
count += 1
socketio.emit('my_response',
{'data': 'Server generated event', 'count': count})
#socketio.event
def connect():
global thread
with thread_lock:
if thread is None:
thread = socketio.start_background_task(background_thread)
So I have an flask application which is extended by the flask_socketio package.
I currently have a vue front end, that connects to an socket.
Now the output of the socket connection in the following:
I also see an connection in the terminal of the flask application:
127.0.0.1 - - [23/Aug/2019 21:07:11] "GET /socket.io/?EIO=3&transport=polling&t=Mo_u1wR HTTP/1.1" 200
However, I have on my code that when the socket connects, there should start an thread with while true, to continious receive the data.
This is the code I have:
from flask import Flask, jsonify
from flask_socketio import SocketIO, send, emit
import threading
from threading import Lock
import time
import controllers.gpsController
# configuration
DEBUG = True
# instantiate the app
app = Flask(__name__)
app.config.from_object(__name__)
thread = None
thread_lock = Lock()
async_mode = None
# app.config['SECRET_KEY'] = 'secret!'
# enable CORS
socketio = SocketIO(app, cors_allowed_origins='*', async_mode=async_mode)
def activate_gps():
ser = controllers.gpsController.open_serial_connection()
while True:
data = controllers.gpsController.readGPS(ser)
if data != None:
socketio.emit('fetch_gps_data', data, broadcast=True)
socketio.sleep(0.1)
# Use sockets here
#socketio.on('connect')
def start_get_data_thread():
global thread
with thread_lock:
if thread is None:
thread = socketio.start_background_task(target=activate_gps)
if __name__ == '__main__':
socketio.run(app, host='127.0.0.1', port=12345)
Anyone an idea what I'm doing wrong here? If I have te code of activate_gps() in another file and call that, I get the wanted output.
I am building an app using flask_sockets library. How do I test it?
Here is a code sample I want to write unit tests for:
import flask
from flask import Flask
from flask_sockets import Sockets
app = Flask(__name__)
sockets = Sockets(app)
ws_conns = []
#sockets.route('/echo')
def echo_socket(ws):
#on connect
ws_conns.append(ws)
#while connected
while True:
# message = ws.receive()
# if message is None:
# #socket has closed/errored
# break
for c in ws_conns:
c.send('hello and goodbye!')
#disconnected
ws_conns.remove(ws)
ws.close()
I have using this code from this git repo.
My question is related to having an Arduino Uno communicate data through a socket to another client app. [A buzzer system communicating to a Jeopardy!-esque game frontend] To goal is to have the 'lockout' event emit.
Currently, the Arduino is running on its own thread and the Flask-SocketIO server is running as the main process. All code works, including the print statement saying "Emitting Socket", except for the line emitting the socket data after.
I feel like this is just a multithreading issue, but my experience with multithreading is minimal.
Suggestions?
# https://pymotw.com/2/threading/
# https://flask-socketio.readthedocs.org/en/latest/
import serial, time, threading
from flask import Flask, render_template
from flask_socketio import SocketIO
ser = serial.Serial('/dev/tty.usbmodem3d11', 9600, dsrdtr=1)
PORT = 3000
# Needed b/c Macs & DTR
time.sleep(5)
def getSerialData():
while True:
stuff = str(ser.readline().decode("utf-8"))
doEmit(1)
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app)
def doEmit(pNo):
print("Emitting Socket")
socketio.emit('lockout', {'playerNo': 1})
serialThread = threading.Thread(name='serialThread', target=getSerialData)
serialThread.start()
#socketio.on("questionRead")
def on_questionRead(data):
print("-------Start-------")
ser.write(b'y\r')
#socketio.on("resetLockout")
def on_resetLockout(data):
resetLockout()
def resetLockout():
print("--------Reset--------")
ser.write(b'n\r')
if __name__ == '__main__':
socketio.run(app, '127.0.0.1', PORT)
You emit a 'lockout' message here:
def doEmit(pNo):
print("Emitting Socket")
socketio.emit('lockout', {'playerNo': 1})
I don't see where you're providing the function to receive this emitted code. Perhaps adding something like this with the appropriate adjustments:
#socketio.on("lockout")
def on_lockout(data):
print("-------Lockout Player", data['playerNo'] , "-------")