I'm putting the finishing touches on my Django-based Webfaction-hosted server. It has a REST API that will deliver content to mobile devices, and will accept POST requests from a different source every few minutes. So far, the POST part is going fine -- it's been accepting data for a week now with no major issues.
However, I'm worried about what happens when I release the mobile app - I expect a fair number of users and I don't want to run into a dead app if my API can't handle the load from all the GET requests.
How do I load test my Django API? Are there tools available online to simulate several hundred GET requests at once, or should I build a test from scratch?
You can check out Locust
This is an open source load testing tool and will help you test your api as well.
Related
I would like to automate this process of viewing logs in dashboard and typing the information (Total messages sent in a time period, total errors, CPU usage, memory usage), this task is very time consuming at the moment.
The info is gathered from mulesoft anypoint platform. I'm currently thinking of a way to extract all of the data using python webscraping but I don't know how to use it perfectly.
You'll find here a screenshot of the website i'm trying to get the data off of, you can choose to see the logs specific to a certain time and date. My question is, do I start learning python webscrapping or is there another way of doing things that I am just unaware of ?
Logs website example
It doesn't make any sense to use web scrapping. All services in Anypoint Platform have a REST API. Most of them are documented at https://anypoint.mulesoft.com/exchange/portals/anypoint-platform/. Scrapping may broke with any minor change to the UI. The REST API is stable.
The screenshot seems to be from Anypoint Monitoring. I see in the catalog Anypoint Monitoring Archive API. I'm not sure if the API for getting Monitoring Dashboards data is documented. You could alternatively use the older CloudHub Dashboards API. It is probably not exactly the same but it will approximate.
I am building an application in Flask API and React.
The first page of the app presents the user with an upload file form. The user selects a file (700 MB) and click uploads.
Once this is done, the backend:
Takes the file, unzip it
Run some ML model
Returns a JSON containing the right data
When this is over, react gets the JSON and renders a new page.
These three steps takes more than 10 minutes therefore I get an error 500 which I believe is due to the long time request timeout.
I would like to know if there is a way to make timeout=None.
I looked for some answers and they suggest to use Celery. However, I am not sure if this is the right approach for my task.
I second with #TheIncorrigible suggestion to solve with some kind of event driven architecture what you are doing is Web Worker Architecture. Ref
Your problem reminds me one of the AWS service called control tower where launching landing zone of that service takes more than >10min and AWS gracefully handles that. When you try to launch it gives me a banner saying it is progress and would take 1 hour. In console log I noticed they were using Promise(Not exactly sure how they are achieving and how long it can handle).
May be you could try using Promises in react for asynchronous computations. I am not expert but it looks like you can achieve this using that. You may watch this short video for basic understanding.
There is also signalr that allows server code to send asynchronous notifications to client-side web applications. You can check if that can be applied in your case signalr in python dicussion
I'm tasked with creating a web app (I think?) for my job that will tracker something in our system. It'll be an internal tool that staff uses to keep track of the status of one of the things we do. It should look like trello, with cards that drag from step to step. That frontend exists, but my job is to make the system update when the cards are dragged. This requires using an API in Python and isn't that complicated to grab from/update. I have no idea how to put all of this together. My job is almost completely nontechnical and there's no one internally who knows what I'm doing except for me. I'm in so over my head here and have no idea where to begin. Is this something I should deploy on Elastic Beanstalk? EC2? How do I tie this together and put it somewhere?
Are you trying to pull in live data from Trello or from your companies own internal project management tool?
An EC2 might be useful, but honestly, it may be completely unnecessary if your company has its own servers. An EC2 is basically just a collection of rental computers to help with scaling. I have never used beanstalk so my input would be useless there.
From what I can assume from the question, you could have a python script running to pull from the API and make the changes without an EC2.
First thing you should do is gather as much information about what the end product should look like. From your question, I have the feeling that you have only a vague idea of what the stakeholders want. Don't be afraid to ask more clarification about an unclear task. It's better to spend 30 minutes discussing and taking note than to show the end-product after a month and realizing that's not what your boss/team wanted.
Question I would Ask
Who is going to be using this app? (technical or non-technical person)
For what purpose is this being developed?
Does it need to be on the web or can it be used locally?
How many users need to have access to this application?
Are we handling sensitive information with this application?
Will this need to be augmented with other functionality at some point?
This is just a sample of what I would ask, during the conversation with the stakeholder a lot more will pop up for sure.
What I think you have to do
You need to make a monitoring system for the tasks that need to be done by your development team (like a Kanban)
What I think you already have
A frontend with the card that are draggable to each bin. I also assume that you can create a new card and delete one in the frontend. The frontend is most likely written in React, Angular or Vue.js. You might also have no frontend framework (a mix of jQuery and vanilla js), but usually frontend developper end up picking a framework of sort to help the development.
A backend API in Python (in Flask or with Django-rest-framework most likely) that is communicating with a SQL database like postgresql or a Document database like MongoDB.
I'm making a lot of assumption here, but your aim should be to understand the technology you will be working with in order to check which hosting would work best. For instance, if the database that is setup is a MySQL database you might have some trouble with some hosting provider.
What I think you are missing
Currently the frontend and the backend don't communicate to each other. When you drag a card it won't persist if you refresh the page. Also, all of this is sitting in your computer and cannot be used by any one from your staff. You need to first connect the frontend with the backend so that the application has persistance. Then you need to deploy this application somewhere so that it is reachable by your staff.
What I would do is first work locally to make sure that the layer of persistance is working. This imply having the API server, the frontend server and the database server running simultaneously on your computer to develop. You should then fetch data from the API to know which cards are there in the database and then create them visually in your frontend at the right spot.
When you drop a card to a new spot after having dragging it should trigger a POST request to your API server in order to update the status of this particular card (look at the documentation of your API to check what you need to send).
The server should be sending back an updated version of the cards status if the POST request was sucessful, so your application should then just redraw the card at the right spot (it won't make a difference for you since they are already at the right spot and your frontend framework will most likely won't act on this change since the state hasn't changed). That's all I would do for that part.
I would then move to the deployment phase to make sure that whatever you did locally can still work online. I would use Heroku to start instead of jumping directly to AWS. Heroku is a service built on top of AWS which manage a lot of the complexity of AWS for you. This is great for prototyping and it means that when your stuff is ready you can migrate to AWS easily and be confident that a setup exist to make your app work. You might also be tied up to your company servers, which is another thing I would ask to the stakeholder (i.e. where can I put this application and where I can't put it).
The flow for a frontend + api + database application on Heroku is usually as follow. You create a github repo for your frontend (make it private) and you create an app on Heroku that will watch this repository for changes. It will re-deploy the application for you when it sees a change at a specific subdomain of Heroku hosting. You will need to configure some procfiles that will tell Heroku what to do with a given application type. This is where you need to double check what frontend you are using since that might change the procfiles used. It's most likely a node.js based frontend (React, Angular or Vue) so head over here for the documentation of how to put that online.
You will need to make a repo for the backend also that is separate from the frontend, these two entities are distinct and they only communicate through HTTP request (frontend->backend) and JSON (backend->frontend). You will need to follow the same idea as with the frontend to deploy, head over here.
Once you have these two online, you need to create a database on Heroku. This is done by adding a datastore to your api, head over here. There are some framework specific configuration you need to do to make the API talk to an online database, but then you will need to find that configuration on the framework documentation. The database could also be already up and living on your server, if this is the case you just need to configure your online backend to talk to that particular database at a particular address.
Once all of the above is done, re-test your application to check if you get the same behavior as before. This is a usable MVP, however there are no layer of security. Anyone with the right URL could just fetch your frontend and start messing around with your data.
There is more engineering that need to be done to make this a viable end product. This leads us to my final remark: why you are not using a product like Trello, Jira, or even Github Project? If it is to save some money on not paying for a subscription I think you should factor in the cost of development, security and maintenance of this application.
Hope it helps!
One simple option is Heroku for deploy your API and your frontend application.
I am trying to design a web based app at the moment, that involves requests being made by users to trigger analysis of their previously entered data. The background analysis could be done on the same machine as the web server or be run on remote machines, and should not significantly impede the performance of the website, so that other users can also make analysis requests while the background analysis is being done. The requests should go into some form of queueing system, and once an analysis is finished, the results should be returned and viewable by the user in their account.
Please could someone advise me of the most efficient framework to handle this project? I am currently working on Linux, the analysis software is written in Python, and I have previously designed dynamic sites using Django. Is there something compatible with this that could work?
Given your background and the analysys code already being written in Python, Django + Celery seems like an obvious candidate here. We're currently using this solution for a very processing-heavy app with one front-end django server, one dedicated database server, and two distinct celery servers for the background processing. Having the celery processes on distinct servers keeps the djangon front responsive whatever the load on the celery servers (and we can add new celery servers if required).
So well, I don't know if it's "the most efficient" solution but it does work.
I would like to generate reports in pdf format with following scenario: people would enter information on a web site and after submitting, data would be transfered to jasper reports server and pdf would be created.
Python would be language of choice for my task.
Is this scenario plausible with current jasper reports software (open source or similar), could it be done, and what would be steps in the right direction ?
Is this scenario plausible with current jasper reports software (open source or similar),
Yes.
could it be done
Yes.
and what would be steps in the right direction ?
Write a web server in Python. Your web server will allow a user to enter information on a web site and after submitting, data would be transfered to jasper reports server and pdf would be created. Your web server would provide the PDF back to the user.
You need to pick a framework, install the components, write the unit tests, write the code, debug the code and transition the code to production.
It's hard (given the question) to determine what part of this you actually need help with.
Write the interface for the user with the language of your choice. Then, having the data from the user, make an API request to the jasperserver's API requesting the report.
Make sure to account for the time the report may need to be generated if you want to make it synchronous.
Otherwise, the API allows you to generate a report and poll for it's completion. When it's done, just send the file to the user.
If you use the second approach, don't point the client ajax polling mechanism to the jasperserver as you might not want it to be accessible from the internet directly. You should do that in the backend of your app.
More information about the REST web services for Jasper Server here: https://community.jaspersoft.com/documentation/jasperreports-server-web-services-guide/v550/rest-web-services-overview
Good luck! :)
Use jasper reports server to publish the report and use its rest interface to produce the output. See Render HTML to PDF in Django site that shows a practical implementation of a python rest client