Develop Reference

Python Django project - What to include in code repository?

I'd like to know whether I should add below files to the code repository:
manage.py
requirements.txt
Also created the very core of the application that includes settings.py. Should that be added to the repo?
And the last one. After creating a project, a whole .idea/ folder was created. It was not included in the .gitignore file template, so how about that?

manage.py
This is part of the Django package, so you do not need to include it. Anyone who install's django will have that installed with it.
requirements.txt
This is how you actually how you tell people WHAT to install to run your project. So at the very least, for a Django project, you will want Django to be in that file. So yes, this file should be included in your git repo. Then anyone you pulls your code can simple run pip install -r requirements.txt and have the requirements installed.
settings.py
This is where things get slightly more into personal preference, but in general, yes it (or something like it) should be included. I like to follow the "Two Scoops of Django" pattern (https://startcodingnow.com/two-scoops-django-config/) of having different settings for different environments.
.idea/
This is actually IDE specific information. JetBrains has a sample file for what they recommend ignoring and what they think you should keep in that folder
(https://github.com/github/gitignore/blob/master/Global/JetBrains.gitignore), but I think it is far more common to just completely ignore that folder all together.

Django : Why should I add package name inside INSTALLED_APPS?

In Djano, why should I add third-party package names inside the INSTALLED_APPS for some packages such as django-filter, DRF, debug-toolbar etc, while I don't want to add for some packages such as Celery, Requests etc ?
I couldn't figure out why should add them to a specific list, even though they all are similar pip packages and I installed them in the same way. Thanks in advance!

From docs :
Package? App?
A Python package provides a way of grouping related Python code for
easy reuse. A package contains one or more files of Python code (also
known as “modules”).
A package can be imported with import foo.bar or from foo import bar.
For a directory (like polls) to form a package, it must contain a
special file init.py, even if this file is empty.
A Django application is just a Python package that is specifically
intended for use in a Django project. An application may use common
Django conventions, such as having models, tests, urls, and views
submodules.
From above statements we understand that every well-written python code could be a package, now if this package is a bunch of python code in a directory, installing it would mean just copy the directory in your project and import them wherever needed in code.
But now there is this package which was already a django app and maybe used some special advantages of being one. Like exposing a restful API or delivering some sort of static resources or even working with special django-specific classes. these kind's of operation's need your direct permission in settings.py so that your project will know how to deal with this package. or even they may require to include their url path's to your project so that people visiting your site could access them.
I assume all this manual job is an act of security.

You need to add apps with models(non-abstract), templates, templatetags or management commands so that django-admin finds and loads them

GAE - Including external python modules without adding them to the repository?

I'm current working on a python based Google App Engine project. Specifically, I'm using Flask for the application. I'm wondering what the accepted method of including external python modules is, specifically when it comes to the repository. From what I can tell, including other people's code in my repository is bad form for several reasons. However, other people will be working on the same repository, so we should be using the same external modules to insure the same results.
Specifically, I need to include Flask (and its dependencies) to my application. The easiest way to do this with Google App Engine is just to throw them into the root level:
MyProject
app.yaml
main.py
MyApp
Flask
...
What is the proper way to bring in these external modules in such a project? Both a generalized answer and one specific to my case would be useful. Also, any other related recommendations would be appreciated. Thank you much.

While it is indeed possible to include third party libraries as submodules or symlinks from external repositories, in practice it's not a good idea. Here are two scenarios on what could go wrong:
If the third party library releases a new version that breaks the functionality, you will have to either make all the necessary changes to meet the new requirements or simply find the previous version to keep working and break the external connection. Usually this happens when you are very close to deadlines.
If the third party library releases a new version and one of your colleagues is upgraded and made all the necessary changes to support the new version, on your side the code will be broken until you will upgrade as well.
The above examples are much more visible in big projects with lots of dependencies and as more people joining the project in the long run it becomes a huge problem! I could come up with more examples, but I think you can see the point.
Your best option is to include the external libraries into your repository, which also has the advantage that you are able to have the whole project up and running on a new machine without many dependencies. There are many ways on how to organize your third party libraries and all of them needs to be included on the same or deeper level with your app.yaml file. Just as #dragonx mentioned include only the core library code.
Also do not afraid putting stuff into your repository cause space is not an issue today and these libraries usually not updating that often so your repository size is not getting too much bigger over time.
Since you mentioned Flask on Google App Engine, you can check out my gae-init project, where you can see in practice how the external libraries are organised.

You're actually asking two questions here.
How do I include the external library in my GAE project?
You've got the right idea. Whatever way you go about it, you must somehow include Flask and its dependencies in the root of your GAE project. One way is to put a copy directly in there.
The second way is to use a symbolic link to the folder that contains the external library. I'm not sure about Flask, but often times external repos contain the actual library code in a subdirectory - so often you don't want the root of the repo in your GAE app, just the root of the actual source. In this case, it's easier to put a symlink that links to the source folder.
How do I manage external libraries in my source repo?
This is a harder question to answer since it depends what source control tool you're using. Yes, you do want to have everyone use the same versions of external libraries, and they should be included in your source control somehow.
If you're using git, git submodule is the way to go. It's a bit confusing to start with but it'll get the job done.
I'd recommend a repo structure that looks something like this
repo/
thirdparty/
flask/
other_dependency/
another_dependency/
README.TXT
setup.py
src/
app/
app.yaml
your_source.py
softlink_to_flask
softlink_to_other_dependency
softlink_to_another_dependency_src
In this example you keep the source to your external libraries in the thirdparty folder. These may be git submodules. In the app folder you have your source, and softlinks to the appropriate files that are actually needed for your app to run. In this case, the actual code for another_dependency may be in the another_dependency/src folder rather than the actual root of another dependency. This way you don't need to include the unnecessary files in your deployment folder, but you can still keep the entire library in your repo.

You can't just create requirements.txt and put it to GAE. Your code must include all pure python libraries that used your project and doesn't supported by GAE (https://developers.google.com/appengine/docs/python/tools/libraries27).
If you look at flask deploy example for GAE (http://flask.pocoo.org/docs/quickstart/#deploying-to-a-web-server and https://github.com/kamalgill/flask-appengine-template) you can find some dependencies like flask, werkzeug and etc. and all this dependencies you must push to GAE server.
So I see three solutions:
Use local requirements for local development and make custom build function that will download all dependencies, put with your application and upload to GAE server.
Add tools for local deployment when you just start project that put required libraries with your application (don't forget about .gitignore).
Use something like git submodules to requirements repositories.

There is two case for using python third party packages in google app engine project:
If your library is one of the supported runtime-provided third-party libraries of GAE section
just add it to your app.yml file under libraries
libraries:
- name: package_name
version: latest
Add your code
import pack_name
Sometimes you need to install the package with
pip install package_name
Make sure you're using the right interpreter, by using
pip freeze
you can make sure the package is installed successfully to the right path.
Otherwise, if GAE does not support you library, you need to download it manually and save it locally under root/Lib directory:
or through GIT
or through pip (pip install package_name -t path/to/your/Lib/dir)
After that, we should declare Lib directory as source dir in pycharm
pycharm->preferences->Project Structure
Choose Lib directory and mark it as source.
Then, import it.
import pack_name
Pay attention that when you're doing the import, you choosing the local path and not your python path.
In general, that's recommended to have requirements.txt file, that includes all the used packages names, and then the pycharm will recognize the uninstalled packages and suggest you to install them.
Good Luck

Installed apps in Django - what about versions?

After looking at the reusable apps chapter of Practical Django Projects and listening to the DjangoCon (Pycon?) lecture, there seems to be an emphasis on making your apps pluggable by installing them into the Python path, namely site-packages.
What I don't understand is what happens when the version of one of those installed apps changes. If I update one of the apps that's installed to site-packages, then won't that break all my current projects that use it? I never noticed anything in settings.py that let's you specify the version of the app you're importing.
I think in Ruby/Rails, they're able to freeze gems for this sort of situation. But what are we supposed to do in Python/Django?

Having multiple versions of the same package gets messy (setuptools can do it, though).
I've found it cleaner to put each project in its own virtualenv. We use virtualevwrapper to manage the virtualenvs easily, and the --no-site-packages option to make every project really self-contained and portable across machines.
This is the recommended setup for mod_wsgi servers.

You definitely don't want to put your Django apps into site-packages if you have more than one Django site.
The best way, as Ken Arnold answered, is to use Ian Bicking's virtualenv (Virtual Python Environment Builder). This is especially true if you have to run multiple versions of Django.
However, if you can run a single version of Python and Django then it might be a little easier to just install the apps into your project directory. This way if an external app gets updated you can upgrade each of your projects one at a time as you see fit. This is the structure Pinax used for external Django apps at one time, but I think it's using virtualenv + pip (instead of setuptools/distutils) now.

What we do.
We put only "3rd-party" stuff in site-packages. Django, XLRD, PIL, etc.
We keep our overall project structured as a collection of packages and Django projects. Each project is a portion of the overall site. We have two separate behaviors for port 80 and port 443 (SSL).
OverallProject/
aPackage/
anotherPackage/
djangoProject80/
settings.py
logging.ini
app_a_1/
models.py # app a, version 1 schema
app_a_2/
models.py # app a, version 2 schema
app_b_2/
models.py
app_c_1/
models.py
djangoProject443/
test/
tool/
We use a version number as part of the app name. This is the major version number, and is tied to the schema, since "uses-the-same-schema" is one definition of major release compatibility.
You have to migrated the data and prove that things work in the new version. Then you can delete the old version and remove the schema from the database. Migrating the data is challenging because you can't run both apps side-by-side.
Most applications have just one current version installed.

Initialize project layout in python?

Suppose a programmer has the following problem: he wants to start a new python project. He needs a basic layout of boilerplate stuff, like test directory, source directory, setuptools script etc.. How does he create all this stuff and layout with a single command ?
For example, paster (as suggested in one of the answers, provides you this service)
paster create
Selected and implied templates: PasteScript#basic_package
A basic setuptools-enabled package
but paster is part of a tool whose main scope is not the deployment of packages. What if I want to have a template for a library, and a template for an application? How can I modify the template to add my own personal stuff to it ?

You need something that supports templating to pull this off. The most used in the python community is pastescript.
easy_install pastescript # A one-time install
paster create
If you've already decided on the name of the package, than it's just:
paster create mypackage
If you want to customize the template, than the easiest way is to create your own python package that includes the custom template you want. Once you've installed it into your environment, you can then use this custom template as much as you want. (This is the sort of thing used by frameworks like pylons to create a template for a web application).
paster create -t libtemplate mypackage
paster create -t apptemplate mypackage
For more details on how to create templates (which consist of a mix of code and source files) take a look at: http://pythonpaste.org/script/developer.html#templates You'll notice that templates support inheritance, so that you can, e.g. just build upon the included template, or create your own, from-scratch templates.
For a good example of a customized template, you can take a look at the pylons template in source, here: Pylons Template Code
In addition, if you're not already using it, you should take a look at Ian Bicking's virtualenv. It allows you to create temporary 'virtual' environments that allow you to install python packages without using and/or conflicting with whatever system-wide packages you may have installed.
A standard setup with virtualenv and pastescript might look something like this:
mkdir mypackage && cd mypackage
virtualenv --distribute env
source env/bin/activate # 'Turns on / activates' the environment
easy_install pastescript
paster create mypackage

I'm using modern-package-template to layout my Python projects.
modern-package-template is a PasteScript template to create an initial layout for your Python projects using modern tools and practices followed in the Python community. Thus, your projects will have the following characteristics:
Use Distribute instead of setuptools as the BDFL himself supports it.
Buildout support, though you are not required to make use of it.
README.txt and NEWS.txt automatically included in your package metadata as long_description, thus making them appear in the PyPI page for your project.
Automatic script (or .exe) creation using Distribute
More info and download from pypi: http://pypi.python.org/pypi/modern-package-template

You can make your own templates. Really useful, for instance for in-house project structure standards.
Best way to start making your own is to start with an existing example and to copy/paste relevant bits from it. Suggestion: ZopeSkel as it is a quite big one with lots of examples. Browse the source code.

I've been using cookiecutter. It's written in python but can be used for any kind of project; not just python. It uses Jinja for templating and features pre and post hooks (written in python or bash) that can easily create/manage one's virtualenvs or anything else you can think of. You can store your own templates in a local directory or pull other peoples directly from the Internet and run them without storing them locally first. It seems much more versatile, simpler to use, and more useful IMHO then paster (disclosure: I've not tried paster). It also is in active development as well.