Looking for a bit of advice.
I have a current architecture of Django and PostgreSQL, where a whole lot of activity is happening to the data via the ORM, through scheduled jobs. The data on the backend is being processed and updated on roughly 30 second intervals.
The data is available to the front-end through a bunch of DRF serialisers (basic REST API). This is just being piped to standard HTML templates at the moment.
I'd like the React front-end to mirror this behaviour, and am looking for best-practice advice on how this is typically done. I know in practice how this works in other frameworks but am not certain of doing this well (namely, connecting React's DOM automation to server-side updates).
(I don't want to get involved with websockets, at all.)
Theoretically, I understand there is two ways to do this:
Front-end AJAX polling the API for new data
HTTP/2 Server Push
Something built into React that will load stuff in incrementally?
Appreciate the advice - (short examples would be really helpful if possible).
First use Django channels, documentation is great btw.
Django Channels
Next what is for you is connect React on some event from models, when you save something in model or create new instance after method save, call channels to expose that object in some group. Of course you need to write URL-s where you will be able to get response from channels.
Related
I recently got a project that involves creating a front end application that would connect to a backend program. The backend program is a machine learning code that inputs some parameters and outputs a graph. This machine learning code was made in Python. The goal of the frontend program is for users over the web to input their data and parameters needed for the ML code to work and then receive back the graph to the front end for the users to save as a GIF or something similar.
I've never done anything of this level before, so I've been scouring the internet for answers. I've come to the answer that the front end will be html and some CSS and will connect to some API program which acts as the middle man between the front end and the backend programs. Is this the correct direction? Any references or YouTube videos about how to do something like this is greatly welcomed.
I've also looked into straight connecting from the frontend program to the ML code.
Thanks All!
So this is a very general question, hence it is hard to answer. I will give you a short pointer on how to achieve this, please note that there are many ways and I am trying to give you the easiest.
App requirements I assume:
The look is not important, functionality is the base
Any framework can be used
A beginner should be able to do this. - hence js frameworks will be avoided.
I've come to the answer that the front end will be html and some CSS and will connect to some API program that acts as the middle man between the front end and the backend programs.
Yes, this is somewhat correct, you most likely will sprinkle some JS into the frontend, but it is not required and everything is possible with just HMTL, CSS, and an API. In the frontend, you will need a <form> which will submit the input data as a post request to your API (a server). The API will then need to invoke the ML script/program and catch the resulting Image. Then you need to save the image on your server. Till now the frontend is still waiting for the request to finish so, your server has not returned anything!
Side note in a real project this would be bad because ML code can take a long time for execution and awaiting a request is caped and you could get a timeout error.
So your API/server receives the image and saves the image into a public folder which is exposed to the web, often called statics or public. Then you can redirect the user onto a second html page, which you can then dynamically render with a template language, e.g. Jinja2.
As backend technology, I would suggest that you use something which is not too opinionated because those usually take more weight from but are harder to learn. Therefore, look at Flask to build an API. For the frontend use https://getbootstrap.com/ so you do not need to write your own css. And as a template language use Jinja2.
to get yourself started I would recommend one of these sources:
https://www.youtube.com/watch?v=Z1RJmh_OqeA // Flask - explains Jinja2 as well
https://www.youtube.com/watch?v=qz0aGYrrlhU // html
https://www.youtube.com/watch?v=K-ccGZYRWzs // bootstrap froms
What is the best way to update a UI element depending on a change in the database? For example, whenever someone comments on a post, Facebook automatically updates the element for each user - how is this done?
I know that data pulling is one way to do it but are there any better procedures?
I would like to know how something like this can be done with Python (Django) but any other generic solution is welcome as well.
What you're looking for is the websockets protocol:
WebSocket is a computer communications protocol, providing full-duplex
communication channels over a single TCP connection
and usually the default way to do with Django is to use the django-channels project:
Channels augments Django to bring WebSocket, long-poll HTTP, task
offloading and other async support to your code, using familiar Django
design patterns and a flexible underlying framework that lets you not
only customize behaviours but also write support for your own
protocols and needs.
You will probably need to spend some time to configure the channels setup and modify your application to use it, but if you're looking to "push" data from backend to the frontend after a certain action has finished, this is probably the way to do it.
I would recommend looking into the channels-examples repository for an example chat implementation which uses django-channels.
I have a rethinkdb. Data will get in database for every five minutes.
I want to create a website to real-time inspect this data flow from rethinkdb.
That is, when surfing the webpage, the data from db on webpages can update automatically without refreshing the webpage.
I know there are several ways to make it real-time such as django channels or websockets. However, model in django does not support rethinkdb.
Sorry I am a layman of making website and may express things inaccurately.
Can someone give me a keyword or hint?
If you make your question more specific, the community here will be able to offer you better support.
However, here is a general solution to your problem.
You will need to do two things:
Create a backend API that allows you to:
Check if new data has been added to the database
Fetch new data via a REST api request
Make frontend AJAX requests to this api
Fetch data
Periodically (every 30sec) check if there is new data
Fetch data again if new data is detected
To do this using Django as the backend, I would recommend using the Django Rest Framework to create your API.
This API should have two endpoints:
ListView of your data
Endpoint returning the id and timestamp of the last datapoint
Next you will have to create a frontend that uses javascript to make requests to these endpoints. When you fetch data, store the id and timestamp of the most recent data point. Use this to check if there is new data.
I would recommend using a Javascript framework such as Angular or react but depending on your needs these may be overkill.
EDIT:
Now that you have updated your answer to be more specific, here is my advice. It sounds like your number one priority is rethinkDB and real time data. Django is not well suited this because it is not compatible with rethinkDB. Real time support has come a long way in Django with Django channels however.
It sounds like you are early on in your project and have little to no codebase in Django. I would recommend using horizon along with rethink db. Horizon is a javascript backend built for real time data from rethinkdb.
I have developed a RESTful API using the Django-rest-framework in python. I developed the required models, serialised them, set up token authentication and all the other due diligence that goes along with it.
I also built a front-end using Angular, hosted on a different domain. I setup CORS modifications so I can access the API as required. Everything seems to be working fine.
Here is the problem. The web app I am building is a financial application that should allow the user to run some complex calculations on the server and send the results to the front-end app so they can be rendered into charts and other formats. I do not know how or where to put these calculations.
I chose Django for the back-end as I expected that python would help me run such calculations wherever required. Basically, when I call a particular api link on the server, I want to be able to retrieve data from my database, from multiple tables if required, and use the data to run some calculations using python or a library of python (pandas or numpy) and serve the results of the calculations as response to the API call.
If this is a daunting task, I at least want to be able to use the API to retrieve data from the tables to the front-end, process the data a little using JS, and send it to a python function located on the server with this processed data, and this function would run the necessary complex calculations and respond with results which would be rendered into charts / other formats.
Can anyone point me to a direction to move from here? I looked for resources online but I think I am unable to find the correct keywords to search for them. I just want a shell code kind of a thing to integrate into my current backed using which I can call some python scripts that I write to run these calculations.
Thanks in advance.
I assume your question is about "how do I do these calculations in the restful framework for django?", but I think in this case you need to move away from that idea.
You did everything correctly but RESTful APIs serve resources -- basically your model.
A computation however is nothing like that. As I see it, you have two ways of achieving what you want:
1) Write a model that represents the results of a computation and is served using the RESTful framework, thus your computation being a resource (can work nicely if you store the results in your database as a way of caching)
2) Add a route/endpoint to your api, that is meant to serve results of that computation.
Path 1: Computation as Resource
Create a model, that handles the computation upon instantiation.
You could even set up an inheritance structure for computations and implement an interface for your computation models.
This way, when the resource is requested and the restful framework wants to serve this resource, the computational result will be served.
Path 2: Custom Endpoint
Add a route for your computation endpoints like /myapi/v1/taxes/compute.
In the underlying controller of this endpoint, you will load up the models you need for your computation, perform the computation, and serve the result however you like it (probably a json response).
You can still implement computations with the above mentioned inheritance structure. That way, you can instantiate the Computation object based on a parameter (in the above case taxes).
Does this give you an idea?
I was wondering what is the 'best' way of passing data between views. Is it better to create invisible fields and pass it using POST or should I encode it in my URLS? Or is there a better/easier way of doing this? Sorry if this question is stupid, I'm pretty new to web programming :)
Thanks
There are different ways to pass data between views. Actually this is not much different that the problem of passing data between 2 different scripts & of course some concepts of inter-process communication come in as well. Some things that come to mind are -
GET request - First request hits view1->send data to browser -> browser redirects to view2
POST request - (as you suggested) Same flow as above but is suitable when more data is involved
Django session variables - This is the simplest to implement
Client-side cookies - Can be used but there is limitations of how much data can be stored.
Shared memory at web server level- Tricky but can be done.
REST API's - If you can have a stand-alone server, then that server can REST API's to invoke views.
Message queues - Again if a stand-alone server is possible maybe even message queues would work. i.e. first view (API) takes requests and pushes it to a queue and some other process can pop messages off and hit your second view (another API). This would decouple first and second view API's and possibly manage load better.
Cache - Maybe a cache like memcached can act as mediator. But then if one is going this route, its better to use Django sessions as it hides a whole lot of implementation details but if scale is a concern, memcached or redis are good options.
Persistent storage - store data in some persistent storage mechanism like mysql. This decouples your request taking part (probably a client facing API) from processing part by having a DB in the middle.
NoSql storages - if speed of writes are in other order of hundreds of thousands per sec, then MySql performance would become bottleneck (there are ways to get around by tweaking mysql config but its not easy). Then considering NoSql DB's could be an alternative. e.g: dynamoDB, Redis, HBase etc.
Stream Processing - like Storm or AWS Kinesis could be an option if your use-case is real-time computation. In fact you could use AWS Lambda in the middle as a server-less compute module which would read off and call your second view API.
Write data into a file - then the next view can read from that file (real ugly). This probably should never ever be done but putting this point here as something that should not be done.
Cant think of any more. Will update if i get any. Hope this helps in someway.