I run Django 1.11.20. I have just (finally) made the jump from Python 2.7 to 3.7.
I've noticed that since upgrading to Python 3, the very first page load after clearing the cache is very slow. So far I only have my Python 3 environment on my development machine, so I'm experiencing this when using ./manage.py runserver and the FileBasedCache backend. I don't know if I will experience the same thing on a production stack (nginx, uwsgi, redis, etc).
I'm using Django Debug Toolbar, which gives me some hints as to what the problem isn't, but I'm not sure where I should look to see what the problem is.
The times given by the debug toolbar run loading a simple page with an empty cache in the Python 2.7 environment on my development machine are:
CPU: 4877.89ms
SQL 462.41ms
Cache: 1154.54ms
The times given by the debug toolbar run loading the same page with an empty cache in the Python 3.7 environment on my development machine are:
CPU: 91661.71ms
SQL 350.44ms
Cache: 609.65ms
(I'm using the file based caching on this development machine, so when I say "with an empty cache" I just mean that I rm -r cache before loading the URL in my browser.)
So the SQL and Cache times got slightly faster with the upgrade, but the CPU time doubled. When I open the "Time" panel in the debug toolbar I see that the "timing attribute" that increased is "request".
This same thing happens on every page (including pages that are just HTML generated by TemplateView, so they're not doing anything tricky), but only when it is first loaded from an empty cache. After that very first response, all pages are back to about the same response time as in the Python 2 environment. So it is something to do with the initial request.
I'm not sure where to look to try to figure out what Django is doing only on the very first request. Any pointers? Or is there something obvious that I can expect to be slower in the jump from Python 2.7 to 3.7?
I was finally able to trace this problem back to the Django Debug Toolbar itself, which was bumped from 1.11 to 2.0 during my Python 3 upgrade. Specifically, the toolbar's SQL panel was causing the slowdown. I have a context processor which executes a query if a cached object does not exist. The query itself is fast, but for some reason Django Debug Toolbar 2.0 is extremely slow generating the stacktrace for it. I have not figured out what changed between 1.11 and 2.0 to cause this terrible performance loss, but now that I know where the problem lies I can work around it.
Related
I have a doc project where I have to test a massive number of urls and links.
To do that I was using python 2 linkchecker.
I upgraded to django 2.2 and python 3.6 and I am using a go binary called muffet (https://github.com/raviqqe/muffet).
linkchecker was gentle with the server, muffet on the other hand is more brutal (even with timeout options and other settings).
The problem I have is after some time, the requests timeout and the django local server crashes.
I heard about some kind of queue or cache for the local django server.
Is anyone knows how to increase the django limit in order not to DDOS myself while I am running my tests before deployment (this tool is not running in production).
Or any out of the box thinking to solve this.
Just for you to know, I run the server in background, and call the tool on the localhost url.
(from another terminal)
Thanks
Edit: https://github.com/django/django/blob/fba5d3b6e63fe4a07b1aa133186f997eeebf9aeb/django/core/servers/basehttp.py#L58 this seems something I can play with?
It seems, like using a proper server within the container and runing muffet command like muffet -t 30 -c 30 http://127.0.0.1 solve the issue.
Thanks to #Antwane to point me in the right direction ;)
The Issue
I am using my laptop with Apache to act as a server for a local project involving tensorflow and python which uses an API written in Flask to service GET and POST requests coming from an app and maybe another user on the local network.The problem is that the initial page keeps loading when I specifically import tensorflow or the object detection folder within the research folder in the tensorflow github folder, and it never seems to finish doing so, effectively getting it stuck. I suspect the issue has to do with the packages being large in size, but I didn't have any issue with that when running the application on the development server provided with Flask.
Are there any pointers that I should look for when trying to solve this issue? I checked the memory usage, and it doesn't seem to be rising substantially, as well as the CPU usage.
Debugging process
I am able to print basic hello world to the root page quite quickly, but I isolated the issue to the point when the importing takes place where it gets stuck.
The only thing I can think of is to limit the number of threads that are launched, but when I limited the number of threads per child to 5 and number of connections to 5 in the httpd-mpm.conf file, it didn't help.
The error/access logs don't provide much insight to the matter.
A few notes:
Thus far, I used Flask's development server with multi-threading enabled to serve those requests, but I found it to be prone to crashing after 5 minutes of continuous run, so I am now trying to use Apache using the wsgi interface in order to use Python scripts.
I should also note that I am not servicing html files, just basic GET and POST requests. I am just viewing them using the browser.
If it helps, I also don't use virtual environments.
I am using Windows 10, Apache 2.4 and mod_wsgi 4.5.24
The tensorflow module being a C extension module, may not be implemented so it works properly in Python sub interpreters. To combat this, force your application to run in the main Python interpreter context. Details in:
http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api
As a Django / Python newbie, should I try to debug on a server running 4 year old software versions, try to recreate the old software installations on my local, or just try to run the software in current version of Django/Python/Postgres/PostGIS on my local Mac OS X 10.9.5?
Background:
On a project where I was supposed to just load data into Postgres/PostGIS, I need to debug why a 2010 year old Django / Postgres / Postgis project is getting an error. I'm a LAMP developer who's never used Django or done much in Python, but I've been able to get a staging site working on the server, and make one or two changes. I thought it would make sense to debug locally on my Mac OS X 10.9.5. So I've used homebrew to install Django 1.7 and Postgres 9.3. Looking at the version differences, I'm worried it will be a more of a hassle now to try to migrate and upgrade the project than to attempt to debug it on the staging site instance running on the server.
FWIW, I know the lines of code that I'd like to investigate (seems like maybe an object is not getting loaded properly from db, since it is in the db), but I'm not positive what sort of echo statements to use. I like using IDE's to inspect things. The project is a bit of an orphan, as the first professional project of a developer who is no longer available to help. And of course, the deadline is last week. :(
Differences between your production and development environments can cause a myriad of headaches.
I recommend you use a tool such as Vagrant to set up a development environment running inside of a virtual machine that mirrors your production server.
Use VirtualEnv to emulate the necessary Django version. PostgreSQL is trickier, in theory you can have a second instance with the required version running simultaneously, but that can also cause very subtle conflicts. It would be better to have it running on another machine (virtual or physical) and access it through your local network.
The simplest way I think is to look at using unittest and mock object to set up some unit tests on the functions that you suspect are the cause of the problem. By using unittest and mock objects, you can control how the existing code interacts with Django and Postgres objects and allow for version differences by setting the expected return values.
With mock object, you can mock all or just part of an existing Python object, which reduces the dependencies you require for your development environment. Depending on how the code is structured, you might not need to install either Django or Postgres at all or a webserver for that matter. This blog explains Mock object in detail.
Even though you're pressed for time, you could do worse than setting up unittests for the whole project, future developers will thank you.
In terms of debugging, I personally can't reccomend pudb enough, it's an interactive command line debugger which you can use with unittest to zero in on what part of the code is causing the problem.
If you do need to install Django and Postgres, I would suggest looking at virtualenv which allows you to set up a virtual environment for Python. That way you can just install the specific dependencies you need without interfering with your global system wide installation. You can also install earlier versions of packages which would do the trick to emulate the existing system's state.
I have put my Django web site up to my web server and have it set up using apache2 and mod_wsgi.. everything works fine most of the time but occasionally it will just give the error that it can't import a module (usually from my views file). However, it's not an issue with that module as it usually works, for example, I will get the error "Cannot import classname from module" once, then reload the page and it works fine, I would say it's about 1 in 10 page loads where this occurs and it's just random as it will happen for any page on my site.
I have tried restarting apache2, restarting the server but the issue persists. I have tried it on different client machines, clearing out the user cache, etc but the issue persists. I don't know what might be doing this, would perhaps some sort of caching help prevent this as it seems that the server is just having an issue with sometimes not being able to fully process the request. I am using a cloud set up with not much memory on the server so maybe this is the problem? Any advice is appreciated
It is working most of the time because you likely have a multi process configuration and only one of the processes is affected.
You can try alternate WSGI script file as documented in:
http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html
The jury is still out as to whether the issue is the differences between development server and proper deployment systems using WSGI, or whether it is users not handling imports properly and causing order dependencies or even import cycles. Problems possibly only come up when URL visited in certain order and thus why random as to when it can happen.
I am developping a django app on Windows, SQLite and the django dev server . I have deployed it to my host server which is running Linux, Apache, FastCgi, MySQL.
Unfortunately, I have an error returned by the server on the prod while everything ok on the dev machine. I've asked my provider for a pre-production solution in order to be able to debug and understand the problem.
Anyway, what are according to you the most likely errors that can happen when moving a django app from dev to prod?
Best
Update: I think that a pre-prod is the best way to address this kind of problem. But I would like to build a check list of what must be done before to put in production.
Thanks for the very valuable answers that I received until now :)
Update: FYI, I 've implemented the preprod server and the email notification as suggested by shanyu and I can see that the error comes from the smart_if templatetag that I am using on this new version. Any trick with template tags?
Update: I think I've fixed the pb which was caused I think by the Filezilla FTP sending. I was using the "replace if newer" option which I guess is causing some unexpected results. Using the "replace all" option fix the issue. However, it was an opportunity for me to learn more about deployment. Thansk for your answers.
Problems I typically have include:
Misconfigured productions settings, whether in my production localsettings.py, wsgi/cgi, or apache site files in /etc/sites-available
Database differences. I use South for migrations and have run into some subtle issues when performing my migration on PostgreSQL when it worked smoothly in sqlite.
Static file hosting since I cheat and use the Django server in development
Permissions, both on the file system and within the database
Rare, but possible, network issues preventing me from getting my dependencies, whether on PyPi or some 3rd party site
Ways that I have mitigated these issues:
Use the same database in production and development (in your case, MySQL everywhere)
I've found it is useful to have a "test" environment which mimics production in every way possible (it can be on lower end hardware, or even the same machine). This way, if there are any issues in this "production-like" enivornment, I can solve them without taking my production server offline.
Script everything for repeatable deployments. I use fabric, but zc.buildout or Paver would also work. These tools help reduce typos while deploying and reduce the time to deploy my app.
Use version control (mercurial, git, subversion) and a schema migration tool (like South), so if something does go wrong when you deploy to production, you have the possibility of backing out the changes and allowing production to run on the old code with the old database schema.
I haven't set up an "egg proxy" yet, but I am considering it, to avoid issues when downloading dependencies.
I've found pip's freezing dependencies to be useful, in case a new, incompatible change to a library occurred since I downloaded it initially
Use a web testing framework like Windmill or Selenium to test my application in my "test" environment, so that I can get a lot of test coverage of my system very quickly.
Regarding your case, I can think of 2 simple things that may help you:
You can enable Django to send messages when exceptions occur giving details about them. Look at here for details.
You'll be better off if you set up a test environment on the prod server (say, test.example.com) so that you can check if things will go smoothly or not before you deploy the app.
I believe these were the podcasts I listened to recently (from Pycon 2009):
Locate Django in the Real World (PyCon 2009):
http://advocacy.python.org/podcasts/pycon.rss
Parts 1 to 3
Very good introduction to designing your apps for deployment, in particular for reuse and redeployment.
Regs.