I have a backend server running on heroku. Right now for going through logs all I have been using is the 'heroku logs' command. I have been using that command also to track how long different requests to each endpoint are taking.
Is there a better way to see a list of how long requests to different endpoints are taking and a good way to track bottlenecks for what is slowing down these endpoints? Also is there any good add ons for heroku that can point out bad responses that are not status =200?
I am using python with django if that is relevant.
The best tool I found is newrelic.com It hooks nicely into django apps and heroku. It can even show you the bottlenecks due to queries and functions inside your views.
Related
I have a flask server which consists of multiple methods. I am aiming to automate the execution of these methods by using Airflow.
I am thinking of using the following steps:-
Setting up Airflow by defining multiple DAGS to call the relevant flask methods in a pipeline.
Deploying Flask Server.
Deploying Airflow (using docker-compose).
Mainly, I am thinking to seperate the Airflow and flask servers independently. Do you think this is a good plan? Any other suggestions would be highly appreciated.
It depends on a couple of things.
Can you run the methods from inside Airflow? For security reasons it is often required to keep some functionality in a different environment/cluster. Reasons for that could be the required database access that you want to give to the Airflow environment.
Is this functionality/methods also invoked from other locations or is it solely for Airflow?
What other functionality does the flask server have that you can't live without?
Are there python dependency conflicts? Even in that case you could use the VirtualEnvOperator of Airflow.
If there is no answer here that is completely blocking you from invoking these methods from inside Airflow, I would vote to do them completely inside Airflow. This will reduce coupling and also reduce the maintenance burden for you in the long term. Besides, Airflow will prevent you from needing to worry about a lot of things, like connectivity, exception codes and callbacks for when something went wrong.
I have a Django app hosted on AWS Lambda (deployed using Zappa tool) an using RDS for hosting the Postgres database. The app is having performance issues with some pages loading very slowly but not others, making me think that the issue is with poorly written data interactions.
I remember at work, the devs were using Azure Application Insights which was monitoring apps at a code level. I think the devs had to insert code throughout their apps to get the monitoring, and I think it monitored on a page by page and code line by line basis.
Is there any similar tool on AWS which I could use with my App?
I appreciate there are much simpler ways of solving my problem but I thought an AWS tool would be something neat to learn.
Yes, there are a few monitoring and troubleshooting tools that work on the AWS environment.
These tools will help you both dive-in into a specific invocation, thus let you understand what wasted the time, but also understand higher-level insights, such as the average time to a DB call or any 3rd party.
Disclosure: I'm working for Lumigo, which is a company that does exactly that. Here's a screenshot that shows this behavior:
I want to create web form that stays on forever on a single computer. Users can come to the computer fill out the form and submit it. After submitting, it will record the responses in an excel file and send emails. The next user can then come and fill out a new form automatically. I was planning on using Flask for this task since it is simple to create, but since I am not doing this on some production server, I will just have it running locally in development on the single computer.
I have never seen anyone do something like this with Flask so I was wondering if my idea is possible or if I should avoid it. I am also new to web development so I was wondering what problems there could be with keeping a flask application stay on 24/7 on a local development computer.
Thanks
There is nothing wrong with doing this in principle however, it is likely not the best solution for the time-to-reward payoff.
First, to answer your question, this could easily be done, even for a beginner, completing this in a few hours with minimal Python and HTML experience could definitely be done. Your app could crash in the background for many reasons (running out of space, bad memory addresses, etc) but most likely you will be fine.
As for specifically building it, it is all possible, there are libraries you can use to add the results to an excel file, or you can easily just append to a CSV (which is what I would recommend). Creating and sending an email, similarly is relatively simple, but again, doing it without python would be much easier.
If you are not set on flask/python, you could check out Google Forms but if you are set on python, or want to use it as a learning experience, it can definitely be done.
Your idea is possible and while there are many ways to do this kind of thing, what you are suggesting is not necessarily to be avoided.
All apps that run on a computer over a long period of time start a process and keep it going until closed. That is essentially what you are doing.
Having done this myself (and still currently doing it) at my business, I can say that it works great.
The only caveat is that to ensure that it will always be available, you need to have the process monitored by some tool to make sure that it gets restarted if it ever closes due to a variety of reasons.
In linux, supervisor is a great tool for doing that. In windows you could register it as a service. But you could also just create an easy way to restart and make it easy for the user to do so if it is down when they need it.
Yes, this could be done. It's very similar to the applications that run on the servers in data centers.
To keep the application running forever or restarting it after your system starts you'll need to use a system manager similar to systemd in Unix. You could use NSSM - the Non-Sucking Service Manager
or Service Control to monitor your application and restart it if it crashes. This will also have to be enabled on startup.
Other than this, you could use Waitres to serve your Flask application. Waitress is a WSGI web server with which you can easily configure the number of threads and workers to enable serving multiple users at the same time.
In a production environment, it's always suggested to use a web server interface like Gunicorn or Waitress.
I'm currently learning Django and getting a decent grasp of it. I know you can start the server with python manage.py runserver. But then there is also heroku ps:scale web=1 followed by heroku open to open the web page at the subdomain you chose on the host herokuapp.com. What is the point of using heroku vs just runserver? When it comes time to deploy to production is that what heroku is used for as I read it was for deployment? What is exactly is heroku used for and what are the reasons? When creating a professional django web site/application is heroku required or just very useful? If so why? I don't really understand the reason for using heroku. And I'm curious what the best practices are for django and whether they include using or not using heroku.
Aside from the heroku question, what are the best practices for django as far as setting a domain name and such (other than the herokuapp.com subdomain name) and also as far as what other components/software solutions are worth using with django?
One of the things I'm working on is an FTP portal on the website that people can login to and upload files. Does django have plugins/python modules for such a thing or must it be done custom? Which brings me to my final question, What kind of addons/modules does django offer to avoid reinventing the wheel when implementing a django website/app's certain features?
The tutorial I'm following: http://tutorial.djangogirls.org/en/django_orm/README.html
has helped me get a grasp on django, but I'm curious about the more advanced parts of it. Such as custom modules, custom styling, custom scripts (whether they are js or strictly python).
If someone could point me in the right direction or offer any suggestions it would be much appreciated. The tutorial has me using the modules: dj-database-url, whitenoise, gunicorn, python-dateutil, psycopg2 , requests. I know what some of them do, but what is the purpose of psycopg2, whitenoise, dateutil, and dj-database,url? Are there any other modules worth using?
Sorry for the wall of questions, but I'm trying to understand django as best I can so that I can do a correct assessment when putting together a django project.
Thanks in advance.
EDIT, final questions:
1) Also when i used heroku they gave me a subdomain at herokuapps.com Can I set my own URL and still use heroku?
2) And last but not least, heroku makes you register and gives you a subdomain on herokuapp.com Is there a way to use heroku with your own domain rather than the one they give you? Is there a specific way to do it?
EDIT 2:
When I do runserver it works fine, but when i run it with heroku ps:scale web=1 and then heroku open it opens but only shows the title at the top of the page, and no content. Not sure exactly why that is. Can someone explain the difference between runserver and heroku ps:scale web=1 followed by heroku open??
python manage.py runserver is only for development. Never use it in production.
Heroku makes it super easy to deploy your Django code. You only need some configs and git push heroku master. But it's not the only way to deploy your code. You can also set up a server with a webserver like Nginx, Gunicorn as WSGI server, and a database of your choice. To set things up, you need to know something about Linux and the command line.
So if you don't know how to set up a server, Heroku can be very useful.
Heroku supports custom domains, and it's easy to set up.
As Mikeec3 mentioned, "Two Scoops of Django" is a very good read.
If you are looking for django packages, www.djangopackages.com is a good place to start.
psycopg2 is for postgresql usage with your app. White noise allows you to serve your staticfiles without nginx if using gunicorn. Requests is an easy to use urllib2 library with pythonic calls to work with apis from other websites.
If you want to do some testing with say facebook logins Heroku is a great place to go to, especially if you need a url outside of localhost.
My suggestion is you read Two scoops of Django http://twoscoopspress.org/products/two-scoops-of-django-1-6
It goes over alot of best practices and serves as a great tutorial after polls. I suggest also incorporating django-bower into your project. Ill list a few great things to add.
Django Bower: https://github.com/nvbn/django-bower - Helps organize static files
Crispy Forms : https://github.com/maraujop/django-crispy-forms - Great form rendering, great with bootstrap
Django Rest Framework: https://github.com/tomchristie/django-rest-framework - REST APIs
These are just to start, there's alot to learn.
So I've been looking into ways to using Heroku for a small-scale personal project (Python Flask + MongoDB), however I can't seem to find much information on how to do simple continuos integration testing or simple unit testing on a Heroku staging instance. I feel that this would be necessary to make sure that everything will work in production, before actually making it public.
There doesn't seem to be much information on as to how I could achieve this. There are a couple of CI addons that would help, but they currently work only with Ruby/RoR (tddium, Rails on Fire) and proper testing on Heroku seems like a problem that should already be solved by a number of people. Buildpacks seem like a potential way to achieve what I need, but I'd rather use existing tools than re-invent the wheel myself.
So the question is, what are my options?
I wouldn't advise on running your tests on Heroku, as the platform isn't designed to do this. It will probably take you much longer to get the Platform to work than simply using another hosted service. There are lots of other alternatives (e.g. Codeship where I am one of the founders).
At Codeship we are currently working on Python support which will be released soon. MongoDB (as well as lots of other tools) is integrated nicely and works out of the box. We are also focusing very strongly on helping you deploy often and integrate that nicely, so you can work on your app and not your infrastructure.
CircleCi has Python support! It also directly supports MongoDB. You'll be able to set it up very easily.
None of the hosted CI solutions, Circle included, run directly on Heroku. We (Circle - I'm a founder) have looked into it, but the way people write tests make this awkward (they're really designed to be run on the same machine). Heroku is also very slow and memory constrained, while the main goal of a CI system is to get results to you quickly.