I am building an application in Flask API and React.
The first page of the app presents the user with an upload file form. The user selects a file (700 MB) and click uploads.
Once this is done, the backend:
Takes the file, unzip it
Run some ML model
Returns a JSON containing the right data
When this is over, react gets the JSON and renders a new page.
These three steps takes more than 10 minutes therefore I get an error 500 which I believe is due to the long time request timeout.
I would like to know if there is a way to make timeout=None.
I looked for some answers and they suggest to use Celery. However, I am not sure if this is the right approach for my task.
I second with #TheIncorrigible suggestion to solve with some kind of event driven architecture what you are doing is Web Worker Architecture. Ref
Your problem reminds me one of the AWS service called control tower where launching landing zone of that service takes more than >10min and AWS gracefully handles that. When you try to launch it gives me a banner saying it is progress and would take 1 hour. In console log I noticed they were using Promise(Not exactly sure how they are achieving and how long it can handle).
May be you could try using Promises in react for asynchronous computations. I am not expert but it looks like you can achieve this using that. You may watch this short video for basic understanding.
There is also signalr that allows server code to send asynchronous notifications to client-side web applications. You can check if that can be applied in your case signalr in python dicussion
Related
I recently got a project that involves creating a front end application that would connect to a backend program. The backend program is a machine learning code that inputs some parameters and outputs a graph. This machine learning code was made in Python. The goal of the frontend program is for users over the web to input their data and parameters needed for the ML code to work and then receive back the graph to the front end for the users to save as a GIF or something similar.
I've never done anything of this level before, so I've been scouring the internet for answers. I've come to the answer that the front end will be html and some CSS and will connect to some API program which acts as the middle man between the front end and the backend programs. Is this the correct direction? Any references or YouTube videos about how to do something like this is greatly welcomed.
I've also looked into straight connecting from the frontend program to the ML code.
Thanks All!
So this is a very general question, hence it is hard to answer. I will give you a short pointer on how to achieve this, please note that there are many ways and I am trying to give you the easiest.
App requirements I assume:
The look is not important, functionality is the base
Any framework can be used
A beginner should be able to do this. - hence js frameworks will be avoided.
I've come to the answer that the front end will be html and some CSS and will connect to some API program that acts as the middle man between the front end and the backend programs.
Yes, this is somewhat correct, you most likely will sprinkle some JS into the frontend, but it is not required and everything is possible with just HMTL, CSS, and an API. In the frontend, you will need a <form> which will submit the input data as a post request to your API (a server). The API will then need to invoke the ML script/program and catch the resulting Image. Then you need to save the image on your server. Till now the frontend is still waiting for the request to finish so, your server has not returned anything!
Side note in a real project this would be bad because ML code can take a long time for execution and awaiting a request is caped and you could get a timeout error.
So your API/server receives the image and saves the image into a public folder which is exposed to the web, often called statics or public. Then you can redirect the user onto a second html page, which you can then dynamically render with a template language, e.g. Jinja2.
As backend technology, I would suggest that you use something which is not too opinionated because those usually take more weight from but are harder to learn. Therefore, look at Flask to build an API. For the frontend use https://getbootstrap.com/ so you do not need to write your own css. And as a template language use Jinja2.
to get yourself started I would recommend one of these sources:
https://www.youtube.com/watch?v=Z1RJmh_OqeA // Flask - explains Jinja2 as well
https://www.youtube.com/watch?v=qz0aGYrrlhU // html
https://www.youtube.com/watch?v=K-ccGZYRWzs // bootstrap froms
I am using Python 3.6.1 with Flask, with the goal being to load a webpage, display live loading information via websockets, then once the loading is complete redirect or otherwise allow the user to do something.
I have the first two steps working- I can load the page fine, and the websocket interface (running on a separate thread using SocketIO) updates properly with the needed data. But how can I make something happen once the functions that need to load are finished? To my understanding once I return a webpage in Flask it is simply static, and there's no easy way to change it.
Specific code examples aren't necessary, I'm just looking for ideas or resources. Thanks.
Looking for a bit of advice.
I have a current architecture of Django and PostgreSQL, where a whole lot of activity is happening to the data via the ORM, through scheduled jobs. The data on the backend is being processed and updated on roughly 30 second intervals.
The data is available to the front-end through a bunch of DRF serialisers (basic REST API). This is just being piped to standard HTML templates at the moment.
I'd like the React front-end to mirror this behaviour, and am looking for best-practice advice on how this is typically done. I know in practice how this works in other frameworks but am not certain of doing this well (namely, connecting React's DOM automation to server-side updates).
(I don't want to get involved with websockets, at all.)
Theoretically, I understand there is two ways to do this:
Front-end AJAX polling the API for new data
HTTP/2 Server Push
Something built into React that will load stuff in incrementally?
Appreciate the advice - (short examples would be really helpful if possible).
First use Django channels, documentation is great btw.
Django Channels
Next what is for you is connect React on some event from models, when you save something in model or create new instance after method save, call channels to expose that object in some group. Of course you need to write URL-s where you will be able to get response from channels.
I'll briefly explain what I'm trying to achieve: We have a lot of servers behind ipvsadm VIPs (LVS load balancing) and we regularly move servers in/out of VIPs manually. To reduce risk (junior ops make mistakes...) I'd like to abstract it to a web interface.
I have a Python daemon which repeatedly runs "ipvsadm -l" to get a list of servers and statistics, then creates JSON from this output. What I'd now like to do is server this JSON, and have a web interface that can pass commands. For example, selecting a server in a web UI and pressing remove triggers an ipvsadm -d <server>... command. I'd also like the web UI to update every 10 seconds or so with the statistics from the list command.
My current Python daemon just outputs to a file. Should I somehow have this daemon also be a web server and serve its file and accept POST requests with command identifiers/arguments? Or a second daemon for the web UI? My only front end experience is with basic Bootstrap and jQuery usually backed by Laravel, so I'm not sure if there's a better way of doing this with sockets and some fancy JS modern-ism.
If there is a more appropriate place for this post, please move it if possible or let me know where to re-post.
You don't need fancy js application. To take the path of least resistance, I would create some extra application - if you like python, I recommend flask for this job. If you prefer php, then how about slim?
In your web application, if you want to make it fast and easy, you can even implement ajax mechanism fetching results based on interval to refresh servers' data every 10 seconds. You will fetch it from json served by independent, already existing deamon.
Running commands clicked on Web UI can be done by your web application.
Your web application is something extra and I find it nice to be separated from deamon which fetch data about servers and save it as json. Anytime you can turn off the page, but all statistics will be still fetching and available for console users in json format.
So I'm currently working on adding a recommendation engine to a Django project and need to do some heavy processing (in an external module) for one of my view functions. This significantly slows down page load time because I have to load in some data, transform it, perform my calculations based on parameters sent by the request, and then return the suggestions to the view. This has to be done every time the view is loaded.
I was wondering if there was some way I could have the recommender module load and transform the data in memory, and then wait for parameters to be sent from the view, have calculations run on those parameters and then send it back to the view.
Any help would be greatly appreciated.
Celery is a task queue that reeally excels at this sort of thing.
It would allow you to do something like:
user makes request to view
view starts an async task that does the heavy lifting, then returns to the user immediately
you can poll from javascript to see if your task is done and load the results when it is
Might not quite be the flow you're looking for but celery is definitetly worth checking out
Celery has a great django package too, extremely easy to use
Rereading your question, i think it would also be possible to create a local webservice around your recommendation engine. On startup it can load all the data into memory, then you can just make requests to it from your django app?