Develop Reference

How do I run a time consuming calculation in the background using Python and FastAPI?

I've been trying to wrap my head around Python multiprocessing but I'm getting stuck. I thought I could solve my problems with the use of a Pool but not really sure how to keep it running between calls to the backend.
Basically I have this setup.
A web frontend
Python backend (fastapi)
What I need (want) to do is:
At one endpoints start a calculation (POST).
At another open a WebSocket and get updates regarding the calculation at regular intervals.
The calculation just needs to keep running. It stores information about its progress in a db which can be used to update the client.
One of the issues (I think) is that I use a context manager when starting the calculation and when the initial request finishes the pool is stopped and deleted.
Some help with this would be really appreciated. I'm probably missing something obvious...

Interactive Brokers TWS: How to handle daily restart in python?

I've built an IB TWS application in python. All seems to work fine, but I'm struggling with one last element.
TWS requires a daily logout or restart. I've opted for the daily restart at a set time so I could easily anticipate a restart of my application at certain times (at least, so I thought.)
My program has one class, called InteractiveBrokersAPI which subclasses the ECClient and EWrapper. Upon the start of my program, I create this instance and it successfully connects to and works with TWS. Now, say that TWS restarts daily at 23:00. I have implemented logic in my program that creates a new instance of my InteractiveBrokersAPI, and calls run() on it af 23:15. This too seems to work. I know this because upon creation, InteractiveBrokersAPI calls reqAccountUpdates() and I can see these updates coming in after my restart. When I try to actually commit a trade the next day, I get an error that it's not connected.
Does anyone else have experience in how to handle this? I am wondering how others have fixed this issue. Any guidance would be highly appreciated.

Well, this doesnt exactly answer your question, but have you looked at ib_insync

Can I have Python code to continue executing after I call Flask app.run?

I have just started with Python, although I have been programming in other languages over the past 30 years. I wanted to keep my first application simple, so I started out with a little home automation project hosted on a Raspberry Pi.
I got my code to work fine (controlling a valve, reading a flow sensor and showing some data on a display), but when I wanted to add some web interactivity it came to a sudden halt.
Most articles I have found on the subject suggest to use the Flask framework to compose dynamic web pages. I have tried, and understood, the basics of Flask, but I just can't get around the issue that Flask is blocking once I call the "app.run" function. The rest of my python code waits for Flask to return, which never happens. I.e. no more water flow measurement, valve motor steering or display updating.
So, my basic question would be: What tool should I use in order to serve a simple dynamic web page (with very low load, like 1 request / week), in parallel to my applications main tasks (GPIO/Pulse counting)? All this in the resource constrained environment of a Raspberry Pi (3).
If you still suggest Flask (because it seems very close to target), how should I arrange my code to keep handling the real-world events, such as mentioned above?
(This last part might be tough answering without seeing the actual code, but maybe it's possible answering it in a "generic" way? Or pointing to existing examples that I might have missed while searching.)

You're on the right track with multithreading. If your monitoring code runs in a loop, you could define a function like
def monitoring_loop():
while True:
# do the monitoring
Then, before you call app.run(), start a thread that runs that function:
import threading
from wherever import monitoring_loop
monitoring_thread = threading.Thread(target = monitoring_loop)
monitoring_thread.start()
# app.run() and whatever else you want to do
Don't join the thread - you want it to keep running in parallel to your Flask app. If you joined it, it would block the main execution thread until it finished, which would be never, since it's running a while True loop.
To communicate between the monitoring thread and the rest of the program, you could use a queue to pass messages in a thread-safe way between them.

The way I would probably handle this is to split your program into two distinct separately running programs.
One program handles the GPIO monitoring and communication, and the other program is your small Flask server. Since they run as separate processes, they won't block each other.
You can have the two processes communicate through a small database. The GIPO interface can periodically record flow measurements or other relevant data to a table in the database. It can also monitor another table in the database that might serve as a queue for requests.
Your Flask instance can query that same database to get the current statistics to return to the user, and can submit entries to the requests queue based on user input. (If the GIPO process updates that requests queue with the current status, the Flask process can report that back out.)
And as far as what kind of database to use on a little Raspberry Pi, consider sqlite3 which is a very small, lightweight file-based database well supported as a standard library in Python. (It doesn't require running a full "database server" process.)
Good luck with your project, it sounds like fun!

Hi i was trying the connection with dronekit_sitl and i got the same issue , after 30 seconds the connection was closed.To get rid of that , there are 2 solutions:
You use the decorator before_request:in this one you define a method that will handle the connection before each request
You use the decorator before_first_request : in this case the connection will be made once the first request will be called and the you can handle the object in the other route using a global variable
For more information https://pythonise.com/series/learning-flask/python-before-after-request

should i be using threads multiprocessing or asycio for my project?

I am trying to build a temperature control module that can be controlled over a network or with manual controls. the individual parts of my program all work but I'm having trouble figuring out how to make them all work together.also my temperature control module is python and the client is C#.
so far as physical components go i have a keypad that sets a temperature and turns the heater on and off and an lcd screen that displays temperature data and of course a temperature sensor.
for my network stuff i need to:
constantly send temperature data to the client.
send a list of log files to the client.
await prompts from the client to either set the desired temperature or send a log file to the client.
so far all the hardware works fine and each individual part of the network functions work but not together. I have not tried to use both physical and network components.
I have been attempting to use threads for this but was wondering if i should be using something else?
EDIT:
here is the basic logic behind what i want to do:
Hardware:
keypad takes a number inputs until '*' it then sets a temp variable.
temp variable is compared to sensor data and the heater is turned on or off accordingly.
'#' turns of the heater and sets temp variable to 0.
sensor data is written to log files while temp variable is not 0
Network:
upon client connect the client is sent a list of log files
temperature sensor data is continuously sent to client.
prompt handler listens for prompts.
if client requests log file the temperature data is halted and the file sent after which the temperature data is resumed.
client can send a command to the prompt handler to set the temp variable to trigger the heater
client can send a command to the prompt handler to stop the heater and set temp variable to 0
commands from either the keypad or client should work at all times.

Multiprocessing is generally for when you want to take advantage of the computational power of multiple processing cores. Multiprocessing limits your options on how to handle shared state between components of your program, as memory is copied initially on process creation, but not shared or updated automatically. Threads execute from the same region of memory, and do not have this restriction, but cannot take advantage of multiple cores for computational performance. Your application does not sound like it would require large amounts of computation, and simply would benefit from concurrency to be able to handle user input, networking, and a small amount of processing at the same time. I would say you need threads not processes. I am not experienced enough with asyncio to give a good comparison of that to threads.
Edit: This looks like a fairly involved project, so don't expect it to go perfectly the first time you hit "run", but definitely very doable and interesting.
Here's how I would structure this project...
I see effectively four separate threads here (maybe small ancillary dameon threads for stupid little tasks)
I would have one thread acting as your temperature controller (PID control / whatever) that has sole control of the heater output. (other threads get to make requests to change setpoint / control mode (duty cycle / PID))
I would have one main thread (with a few dameon threads) to handle the data logging: Main thead listens for logging commands (pause, resume, get, etc.) dameon threads to poll thermometer, rotate log files, etc..
I am not as familiar with networking, and this will be specific to your client application, but I would probably get started with http.server just for prototyping, or maybe something like websockets and a little bit of asyncio. The main thing is that it would interact with the data logger and temperature controller threads with getters and setters rather than directly modifying values
Finally, for the keypad input, I would likely just make up a quick tkinter application to grab keypresses, because that's what I know. Again, form a request with the tkinter app, but don't modify values directly; use getters and setters when "talking" between threads. It just keeps things better organized and compartmentalized.

How to collect continuous data with Python Telnet

I have a python script that connects to a Power Supply via a Telnet session. The flow of the script is as follows:
# Connect to Device
tn = telnetlib.Telnet(HOST,PORT)
# Turn On
tn.write("OUT 1\r")
# Get Current Voltage
current_voltage = tn.write("MV?\r")
# Turn Off
tn.write("OUT 0\r")
What I'd like to do is be able to get the Current Voltage every t milliseconds(ms) and be able to display it on my Tkinter GUI until the device is commanded to be turned off. Ideally I'd like to display it on a chart such that I have Voltage vs. time, but i can live with just a dynamic text display for now. The current_voltage variable will store a string representing the current voltage value. What is the best way I can accomplish this? Thanks.

Every millisecond is probably more than tkinter can handle. It depends a bit on how expensive it is to fetch the voltage. If it takes longer than a millisecond, you're going to need threads or multiprocessing.
The simplest solution is to use after to schedule the retrieval of the data every millisecond, though again, I'm not sure it can keep up. The problem is that the event loop needs time to process events, and giving it such a tiny window of time when it's not fetching voltages may result in a laggy GUI.
The general technique is to write a function that does some work, and then calls after to have itself called again in the future.
For example:
root = tk.Tk()
...
def get_voltage():
<your code to get the voltage goes here>
# get the voltage again in one millisecond
root.after(1, get_voltage)
...
get_voltage()
root.mainloop()
the other choice is to use threads, where you have a thread that does nothing but get the voltage information and put it on a queue. Then, using the same technique as above, you can pull the latest voltage(s) off of the queue for display.