I am using Python 3.6.1 with Flask, with the goal being to load a webpage, display live loading information via websockets, then once the loading is complete redirect or otherwise allow the user to do something.
I have the first two steps working- I can load the page fine, and the websocket interface (running on a separate thread using SocketIO) updates properly with the needed data. But how can I make something happen once the functions that need to load are finished? To my understanding once I return a webpage in Flask it is simply static, and there's no easy way to change it.
Specific code examples aren't necessary, I'm just looking for ideas or resources. Thanks.
Related
I am building an application in Flask API and React.
The first page of the app presents the user with an upload file form. The user selects a file (700 MB) and click uploads.
Once this is done, the backend:
Takes the file, unzip it
Run some ML model
Returns a JSON containing the right data
When this is over, react gets the JSON and renders a new page.
These three steps takes more than 10 minutes therefore I get an error 500 which I believe is due to the long time request timeout.
I would like to know if there is a way to make timeout=None.
I looked for some answers and they suggest to use Celery. However, I am not sure if this is the right approach for my task.
I second with #TheIncorrigible suggestion to solve with some kind of event driven architecture what you are doing is Web Worker Architecture. Ref
Your problem reminds me one of the AWS service called control tower where launching landing zone of that service takes more than >10min and AWS gracefully handles that. When you try to launch it gives me a banner saying it is progress and would take 1 hour. In console log I noticed they were using Promise(Not exactly sure how they are achieving and how long it can handle).
May be you could try using Promises in react for asynchronous computations. I am not expert but it looks like you can achieve this using that. You may watch this short video for basic understanding.
There is also signalr that allows server code to send asynchronous notifications to client-side web applications. You can check if that can be applied in your case signalr in python dicussion
I have written a python module for extracting archives and processing some data from the extracted files, then optionally deleting the extracted files. Currently I am using it by writing user-code in a separate python script. e.g:
import my_module
with my_module.archive("data.rar") as a:
a.extract()
a.convert("data.csv", "data.xlxs")
a.delete(keep="data.xlsx")
But for non-programmers to use it, I am trying to create a webapp user-interface for it with flask.
What are the best practices for implementing some communication between the user and the module through a webapp? For example
how could my module to communicate to the webapp-user that the extraction or the data processing has finished?
how should the it display the list of extracted files?
how could the user select a file to be processed or to be deleted?
Basically I need to interleave my library code with some user code and I would like the webapp to generate and execute the user-code and to display the output of the processing module.
(I sense a close flag coming since this is a broad topic and it's been a while since "best practices" topics were allowed on SO. )
Based on my on faults I'd recommend to:
implement the backend first, don't waste time on web design if the backend functionality is not crystal clear yet. Read about flask blueprints and create a blueprint for all public calls like list directory, select file, unzip file, etc... test it out, use requests, post stuff, check responses, iterate.
if you are satisfied with basic functionality, start implementing the frontend. For interactivity you can use the very same api calls via javascript you already tested, use XMLHttpRequest (or use jQuery - I am not a big fan of that) for posting stuff. The catch here is that when you post (from the browser), you can define a callback, so you may use the flask response to update the interface.
add some css, eventlisteners to your templates to make it pretty and comfy.
Get ideas from here and here and here and here (just googled for some random pages, some seemed on topic for you).
You probably want to read this too.
I'll briefly explain what I'm trying to achieve: We have a lot of servers behind ipvsadm VIPs (LVS load balancing) and we regularly move servers in/out of VIPs manually. To reduce risk (junior ops make mistakes...) I'd like to abstract it to a web interface.
I have a Python daemon which repeatedly runs "ipvsadm -l" to get a list of servers and statistics, then creates JSON from this output. What I'd now like to do is server this JSON, and have a web interface that can pass commands. For example, selecting a server in a web UI and pressing remove triggers an ipvsadm -d <server>... command. I'd also like the web UI to update every 10 seconds or so with the statistics from the list command.
My current Python daemon just outputs to a file. Should I somehow have this daemon also be a web server and serve its file and accept POST requests with command identifiers/arguments? Or a second daemon for the web UI? My only front end experience is with basic Bootstrap and jQuery usually backed by Laravel, so I'm not sure if there's a better way of doing this with sockets and some fancy JS modern-ism.
If there is a more appropriate place for this post, please move it if possible or let me know where to re-post.
You don't need fancy js application. To take the path of least resistance, I would create some extra application - if you like python, I recommend flask for this job. If you prefer php, then how about slim?
In your web application, if you want to make it fast and easy, you can even implement ajax mechanism fetching results based on interval to refresh servers' data every 10 seconds. You will fetch it from json served by independent, already existing deamon.
Running commands clicked on Web UI can be done by your web application.
Your web application is something extra and I find it nice to be separated from deamon which fetch data about servers and save it as json. Anytime you can turn off the page, but all statistics will be still fetching and available for console users in json format.
I have written a scraper tool in Python which when executed produces a CSV file of information. I wish to embed it in a HTML, so within the page the user is able to run it and then the results are displayed on the page from the CSV file. How can I do this?
If you only need to display the information on a website, no interaction back with your python script, all you need to do is just write the results into a HTML file and .format() it appropriately.
However, since this requires the user to be able to run and view the results with interaction with your python script, you would need a web server.
You could try the two most popular python based web servers that can accomplish this: Flask and Django or you could use the less common but more lightweight Simple HTTP Server.
But, do keep in mind that an easier alternative (if possible) will be to just use a GUI such as Tkinter or PyQt or EasyGUI.
So I'm currently working on adding a recommendation engine to a Django project and need to do some heavy processing (in an external module) for one of my view functions. This significantly slows down page load time because I have to load in some data, transform it, perform my calculations based on parameters sent by the request, and then return the suggestions to the view. This has to be done every time the view is loaded.
I was wondering if there was some way I could have the recommender module load and transform the data in memory, and then wait for parameters to be sent from the view, have calculations run on those parameters and then send it back to the view.
Any help would be greatly appreciated.
Celery is a task queue that reeally excels at this sort of thing.
It would allow you to do something like:
user makes request to view
view starts an async task that does the heavy lifting, then returns to the user immediately
you can poll from javascript to see if your task is done and load the results when it is
Might not quite be the flow you're looking for but celery is definitetly worth checking out
Celery has a great django package too, extremely easy to use
Rereading your question, i think it would also be possible to create a local webservice around your recommendation engine. On startup it can load all the data into memory, then you can just make requests to it from your django app?