Develop Reference

Django Apps and Python Packages Dependencies

I am new to web development. I am not familiar with Django. I have written some Python scripts which do some intense calculations and graphs plotting using Python packages such as numpy, matlibplot and so on. I want to publish it as a web application on a server to be accessed by other computers.
So I am wondering, do I need to copy all the required packages into the project directory before deploying the application to a server? Or Django will automatically handle the Python packages dependencies upon deploying?

All the needed packages need to be installed on the server where is the web application running. If you have all the packages installed on your personal computer before deploy you will need to install the on the server as well.

You have to create virtual environment.
A virtual environment is a tool that helps to keep dependencies required by different projects separate by creating isolated python virtual environments for them.

Django app - What do I install in virtualenv vs system wide?

I'm working on creating my first "real" web app using Django.
Yesterday I learned I should be using a web server like Nginx to serve static files and pass off requests for dynamic content to my web app. I also learned that I need something like Gunicorn as the intermediary between the web server (Nginx) and my Django app.
My question is about virtualenv. It makes sense that we would contain app related software in it's own separate environment. What should I install in virtualenv, and what gets installed system wide? For example, in this guide we seem to install Python, Nginx and the database system wide (because they're installed before virtualenv is installed) while Django and Gunicorn are installed in virtualenv. It makes sense that Gunicorn would have to go in the virtualenv since its importing our python app, as explained here. Are the other things required to be installed system wide? Or can I pick either way? Is one way preferred over another?
Thanks!

Virtualenv is for managing Python libraries. It is not for managing Python itself, or for external services such as databases; it does however manage the Python libraries you use to access the database.
There's no room for confusion here, because there's simply no way to install Python itself or a database within a virtualenv.

Getting Django 1.7 to work on Google App Engine

Can anyone help to point us to instructions on how to get Django >1.5 working on Google App Engine? I have seen a number of people claim they have Django 1.6 working. We'd like to get 1.6 or 1.7 running. I have searched here for instructions on how to set this up. No luck so far.
Update:
In our development machine we have Django 1.7 installed (both /user/local and on virtualenv). However, if we modify GAE yaml to use Django 1.7 we get the following error messages:
google.appengine.api.yaml_errors.EventError: django version "1.7" is not supported, use one of: "1.2", "1.3", "1.4", "1.5" or "latest" ("latest" recommended for development only) in "./app.yaml",
The version 1.9.12 GoogleAppEngine sdk install in our /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib directory shows the following Django versions listed:
django-0.96 django-1.2 django-1.3 django-1.4 django-1.5
My question is related to how to get our development environment setup correctly for Django 1.7 on Google App Engine and how to make sure we successfully deploy our app with Django 1.7 when we deploy to Google App Engine in production. How do we get the Django 1.7 to deploy on GAE when we deploy our app?

You can use any pure Python third party libraries in your Google App Engine application. In order to use a third party library, simply include the files in your application's directory, and they will be uploaded with your application when you deploy it to our system. You can import the files as you would any other Python files with your application.
I have application using Django 1.7 this way and everything is working fine. However, sometimes you may need to sort of hack something due to the App Engine limitations and its specifics. But it depends on your use cases.
I would also suggest to use virtual environment for your project. Install each library that is not supported by App Engine directly via pip and then create a symlink in your application directory pointing to the given library.
This way you can keep all required packages in a file (e.g. requirements.txt) that can be stored in SCM system (e.g. Git) along with your source files and other team members can quite easily replicate your working environment.
Provided that you use virtual environment and install all needed libraries (Django, ...) via pip, here is the directory layout that should work for you.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
app.yaml
django-project-root
django-app-root
symlink-to-django -> lib/python2.7/site-packages/django
symlink-to-another-lib -> lib/python2.7/site-packages/...
Such a layout can be easily deployed with the below command.
$ appcfg.py update app-engine-project-root
Or tested with App Engine development server.
$ dev_appserver.py app-engine-project-root
UPDATE
Since App Engine Python SDK version 1.9.15 you can use the vendoring mechanism to set up third party libraries. You do not have to create symlinks in your application directory pointing to the Python lib folder anymore.
Create lib directory directly in your application root and tell your app how to find libraries in this directory by means of appengine_config.py file.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
New directory layout follows.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
lib
app.yaml
appengine_config.py
django-project-root
django-app-root
Use pip with the -t lib flag to install libraries in this directory.
$ pip install -t lib [lib-name]
Or
$ pip install -t lib -r requirements.txt

You cannot - GAE only supports 1.5, and even that is marked as experimental. If you need django 1.7, perhaps you should use Google Compute Engine, which is Google's brand name for virtual machines that you can spool up.
If you are not married to Google App Engine, Heroku supports django 1.7 without issues.
Do you have specific a guide on how to move a Django 1.7 project to
Google Compute Engine? There is a bunch of Google stuff without any
guides on how to make them work.
Here are the steps, but they are the same had you deployed on any other server because GCE just gives you a linux instance:
First, make sure your developer account has a billing method attached to it.
Go to the developer console
Create a new project by clicking on Projects, then Create Project.
Wait as the project is being created (you'll see a progress window on the bottom right of your screen).
Once the project is finished creating, the console will automatically shift to that project's settings:
You can create a new instance, or deploy a ready-made template from the second column. You can see there are popular stacks and software applications for which templates are created.
As there is no django template yet, you will start by creating an instance.
Billing is controlled on a per-project basis, so you'll have enable billing at this point if you haven't done so already.
The next page is where you configure the instance. The fields are self-explanatory. You set the type of machine you like (how many virtual CPUs and memory), where (physically) you prefer the machine to be located, if you want both HTTP and HTTPS ports open, and then a disk image from which the instance will boot:
Once you have configured the machine, it will be brought online booted up and then you'll have access to the terminal via SSH.
From this point forward, you should treat this instance like any linux server. Install whatever you need to make your project work using the normal packaging tools; upload your files, etc.
For Amazon, the process is a bit simpler as there is a large library of AMIs that you can use for a one-click deployment process. AMI is Amazon Machine Image - a template from which you can deploy an instance.
For Heroku, as its a PaaS, you don't have to worry about the hardware components; however as with most PaaS platforms, you don't have write access to the filesystem. So to manage your static assets you have to do some extra work. The easiest option is to create a S3 bucket on Amazon and use that with django-storages. The official django tutorial at heroku suggests the use of dj-static to serve files directly from Heroku. This works fine for testing, but if you want to start uploading files, then you need to handle those correctly.
However, once you sort that out the steps are even simpler:
Pre-requisites:
git
heroku toolbelt
dj-database-url Python package
gunicorn Python package
The basic steps:
Create a git repository (if you have not done already) in your source code directory with git init.
Create a requirements.txt at the root of your project. pip freeze > requirements.txt should do it if you are using a virtual environment. Otherwise, you can create a text file and list the packages you need.
Adjust your settings.py, by adding this line at the very bottom: import dj_database_url
DATABASES['default'] = dj_database_url.config()
Create a Procfile (case is important). This is how you tell Heroku what kind of dyno (process) you need for your application. For django, you need a web dyno so in this file the following line should do: web: gunicorn yourproject.wsgi --log-file -
Create an app on Heroku and deploy. You should run these commands from your source code directory:
heroku create --buildpack https://github.com/heroku/heroku-buildpack-python
heroku addons:add heroku-postgresql:dev
git push heroku master
heroku run python yourproject/manage.py migrate --noinput
heroku run python web/manage.py collectstatic
You only do the first two steps once, then whenever you need to update your application simply git push heroku master to create a new revision on Heroku.

App Engine's Python environment currently knows how to provide Django up to version 1.5 via the libraries: configuration mechanism. This doesn't mean that later versions of Django won't work, only that they aren't yet built in. (I'm not sure why the latest built-in version is 1.5. It may have something to do with AE's historical policy of bundling each supported version of Django with the SDK, which probably needs to be revised to keep the SDK from getting too large.)
You can try to include Django 1.7 with your application files. I haven't tried this with 1.7 specifically yet, but it's worked with previous versions. Some adjustments to sys.path will be needed in your main.py.
Note that there is a limit of 10,000 application files. If you're concerned about this limit, one option is to use Python's zipimport and include Django as a zip archive. https://docs.python.org/2/library/zipimport.html

python django project and folder structure (differing from WAMP)

I have my development environment setup on Win 7 like this:
Django development structure
Apache -server- C:\Program Files (x86)\Apache Software Foundation\Apache2.4
PostgreSQL -database- C:\Program Files\PostgreSQL\9.2
Django -framework- C:\Python27\Lib\site-packages\django
Python -code- C:\Python27
Project -root- C:\mysite
|----------apps
|----------HTML
|----------CSS
|----------JavaScript
|----------assets
I am attempting to keep this extremely simple to start out. There are 5 main directories each with a distinct purpose. All the code resides in the project folder.
compared to WAMP structure:
C:\WAMP
|----------C:\Apache
|----------C:\MySQL
|----------C:\PHP
|----------C:\www
I like how Apache, MySQL, and PHP all reside in a neat directory. I know to keep the root project OUTSIDE in another directory in Django for security reasons.
Is it fine that Apache, PostgreSQL, and Python are installed all over the place in the Django environment?
Did I miss a core Django component and/or directory?
Will deploying and scaling be a problem?
I want this to be a guideline for beginning Django web programmers.

I can answer the question one by one:
Is if fine that Apache, PostgreSQL, and Python are installed all over the place in the Django environment?
All over the place sounds weird but yes it is totally fine.
Did I miss a core Django component and/or directory?
No you don't miss anything, Django core is in site-packages folder already and your site code is mysite, which can be located anywhere you want.
Will deploying and scaling be a problem?
No it won't be a problem with current structure. You will deploy your mysite only, the other will be installed separately.
Something you should get familiar with when starting with Django development:
Most likely when you deploy your project, it will be on a Linux server, so install and learn Linux maybe?
virtualenv: Soon you will have to install Django, then a bunch of external packages to support your project. virtualenv helps you isolate your working environment. Well it's "unofficial" a must when you start with python development.
virtualenvwrapper to make your life easier when working with virtualenv
git and github or bitbucket: if you don't know git yet, you should now.

Apache is just web server, it is used to serve files, but to make a website you do not necessary need it. Django comes with its own development server. See :
python manage.py runserver
Apache is required when you are developing PHP websites because your computer do not know how to compile and interpret it. But for Django, you use the Python language, and you have already install it if you are using Django.
Read https://docs.djangoproject.com/en/1.5/intro/tutorial01/
And where it will be the time to set up your own server using Apache look at :
https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/modwsgi/.

Scaling will be a problem on windows. Python in Apache on windows gets 64 threads in one process. Couple this with the GIL and you will have scaling issues.
Python and Apache on Linux don't have this same problem. Under Linux wsgi can create multiple processes that have multiple threads each, minimizing GIL issues.
WSGI in Apache on windows is not a scalable solution in my opinion.
However you can develop there and move to linux for deployment, I do it all the time.
You will want to take advantage of the Apache Alias directive to serve all your static content like css, js, favicon.ico. This frees up python to only handle requests that require logic.

Are there differences in the AppEngine Python SDK development server code between platforms?

I have created a buildout for a Pyramid project on GAE and as part of the buildout, it pulls http://googleappengine.googlecode.com/files/google_appengine_1.6.3.zip and places it in the project structure. This works fine for Linux and OS X (although of course extras like the Launcher are not present).
Are the Python code differences with the GAE Windows SDK such that I can't use the .zip file or is the developer server code the same between platform versions? (the differences just being extras and the install method)

The Python code for the tools (dev_appserver and appcfg) are identical for all three platforms.