Develop Reference

GAE - Including external python modules without adding them to the repository?

I'm current working on a python based Google App Engine project. Specifically, I'm using Flask for the application. I'm wondering what the accepted method of including external python modules is, specifically when it comes to the repository. From what I can tell, including other people's code in my repository is bad form for several reasons. However, other people will be working on the same repository, so we should be using the same external modules to insure the same results.
Specifically, I need to include Flask (and its dependencies) to my application. The easiest way to do this with Google App Engine is just to throw them into the root level:
MyProject
app.yaml
main.py
MyApp
Flask
...
What is the proper way to bring in these external modules in such a project? Both a generalized answer and one specific to my case would be useful. Also, any other related recommendations would be appreciated. Thank you much.

While it is indeed possible to include third party libraries as submodules or symlinks from external repositories, in practice it's not a good idea. Here are two scenarios on what could go wrong:
If the third party library releases a new version that breaks the functionality, you will have to either make all the necessary changes to meet the new requirements or simply find the previous version to keep working and break the external connection. Usually this happens when you are very close to deadlines.
If the third party library releases a new version and one of your colleagues is upgraded and made all the necessary changes to support the new version, on your side the code will be broken until you will upgrade as well.
The above examples are much more visible in big projects with lots of dependencies and as more people joining the project in the long run it becomes a huge problem! I could come up with more examples, but I think you can see the point.
Your best option is to include the external libraries into your repository, which also has the advantage that you are able to have the whole project up and running on a new machine without many dependencies. There are many ways on how to organize your third party libraries and all of them needs to be included on the same or deeper level with your app.yaml file. Just as #dragonx mentioned include only the core library code.
Also do not afraid putting stuff into your repository cause space is not an issue today and these libraries usually not updating that often so your repository size is not getting too much bigger over time.
Since you mentioned Flask on Google App Engine, you can check out my gae-init project, where you can see in practice how the external libraries are organised.

You're actually asking two questions here.
How do I include the external library in my GAE project?
You've got the right idea. Whatever way you go about it, you must somehow include Flask and its dependencies in the root of your GAE project. One way is to put a copy directly in there.
The second way is to use a symbolic link to the folder that contains the external library. I'm not sure about Flask, but often times external repos contain the actual library code in a subdirectory - so often you don't want the root of the repo in your GAE app, just the root of the actual source. In this case, it's easier to put a symlink that links to the source folder.
How do I manage external libraries in my source repo?
This is a harder question to answer since it depends what source control tool you're using. Yes, you do want to have everyone use the same versions of external libraries, and they should be included in your source control somehow.
If you're using git, git submodule is the way to go. It's a bit confusing to start with but it'll get the job done.
I'd recommend a repo structure that looks something like this
repo/
thirdparty/
flask/
other_dependency/
another_dependency/
README.TXT
setup.py
src/
app/
app.yaml
your_source.py
softlink_to_flask
softlink_to_other_dependency
softlink_to_another_dependency_src
In this example you keep the source to your external libraries in the thirdparty folder. These may be git submodules. In the app folder you have your source, and softlinks to the appropriate files that are actually needed for your app to run. In this case, the actual code for another_dependency may be in the another_dependency/src folder rather than the actual root of another dependency. This way you don't need to include the unnecessary files in your deployment folder, but you can still keep the entire library in your repo.

You can't just create requirements.txt and put it to GAE. Your code must include all pure python libraries that used your project and doesn't supported by GAE (https://developers.google.com/appengine/docs/python/tools/libraries27).
If you look at flask deploy example for GAE (http://flask.pocoo.org/docs/quickstart/#deploying-to-a-web-server and https://github.com/kamalgill/flask-appengine-template) you can find some dependencies like flask, werkzeug and etc. and all this dependencies you must push to GAE server.
So I see three solutions:
Use local requirements for local development and make custom build function that will download all dependencies, put with your application and upload to GAE server.
Add tools for local deployment when you just start project that put required libraries with your application (don't forget about .gitignore).
Use something like git submodules to requirements repositories.

There is two case for using python third party packages in google app engine project:
If your library is one of the supported runtime-provided third-party libraries of GAE section
just add it to your app.yml file under libraries
libraries:
- name: package_name
version: latest
Add your code
import pack_name
Sometimes you need to install the package with
pip install package_name
Make sure you're using the right interpreter, by using
pip freeze
you can make sure the package is installed successfully to the right path.
Otherwise, if GAE does not support you library, you need to download it manually and save it locally under root/Lib directory:
or through GIT
or through pip (pip install package_name -t path/to/your/Lib/dir)
After that, we should declare Lib directory as source dir in pycharm
pycharm->preferences->Project Structure
Choose Lib directory and mark it as source.
Then, import it.
import pack_name
Pay attention that when you're doing the import, you choosing the local path and not your python path.
In general, that's recommended to have requirements.txt file, that includes all the used packages names, and then the pycharm will recognize the uninstalled packages and suggest you to install them.
Good Luck

Distributing python code with virtualenv?

I want to distribute some python code, with a few external dependencies, to machines with only core python installed (and users that unfamiliar with easy_install etc.).
I was wondering if perhaps virtualenv can be used for this purpose? I should be able to write some bash scripts that trigger the virtualenv (with the suitable packages) and then run my code.. but this seems somewhat messy, and I'm wondering if I'm re-inventing the wheel?
Are there any simple solutions to distributing python code with dependencies, that ideally doesn't require sudo on client machines?

Buildout - http://pypi.python.org/pypi/zc.buildout
As sample look at my clean project: http://hg.jackleo.info/hyde-0.5.3-buildout-enviroment/src its only 2 files that do the magic, more over Makefile is optional but then you'll need bootstrap.py (Make file downloads it, but it runs only on Linux). buildout.cfg is the main file where you write dependency's and configuration how project is laid down.
To get bootstrap.py just download from http://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py
Then run python bootstap.py and bin/buildout. I do not recommend to install buildout locally although it is possible, just use the one bootstrap downloads.
I must admit that buildout is not the easiest solution but its really powerful. So learning is worth time.
UPDATE 2014-05-30
Since It was recently up-voted and used as an answer (probably), I wan to notify of few changes.
First of - buildout is now downloaded from github https://raw.githubusercontent.com/buildout/buildout/master/bootstrap/bootstrap.py
That hyde project would probably fail due to buildout 2 breaking changes.
Here you can find better samples http://www.buildout.org/en/latest/docs/index.html also I want to suggest to look at "collection of links related to Buildout" part, it might contain info for your project.
Secondly I am personally more in favor of setup.py script that can be installed using python. More about the egg structure can be found here http://peak.telecommunity.com/DevCenter/PythonEggs and if that looks too scary - look up google (query for python egg). It's actually more simple in my opinion than buildout (definitely easier to debug) as well as it is probably more useful since it can be distributed more easily and installed anywhere with a help of virtualenv or globally where with buildout you have to provide all of the building scripts with the source all of the time.

You can use a tool like PyInstaller for this purpose. Your application will appear as a single executable on all platforms, and include dependencies. The user doesn't even need Python installed!
See as an example my logview package, which has dependencies on PyQt4 and ZeroMQ and includes distributions for Linux, Mac OSX and Windows all created using PyInstaller.

You don't want to distribute your virtualenv, if that's what you're asking. But you can use pip to create a requirements file - typically called requirements.txt - and tell your users to create a virtualenv then run pip install -r requirements.txt, which will install all the dependencies for them.
See the pip docs for a description of the requirements file format, and the Pinax project for an example of a project that does this very well.

How do I package a python application to make it pip-installable?

I'm writing a django application in my spare time for a footy-tipping competition we're running at work. I figured I'd use this time wisely, and get up to speed on virtualenv, pip, packaging, django 1.3, and how to write an easily redistributable application. So far, so good.
I'm up to the packaging part. A lot of the django apps on GitHub for instance are mostly bundled (roughly) the same way. I'll use django-uni-forms as an example.
An assumption I'm making is that the MANIFEST.in and setup.py are the only required pieces that pip needs to do its job. Is that correct? What other components are necessary if my assumption is wrong?
Are the required packaging files generally generated, or are they crafted by hand? Can dependencies be described and then installed also? My application depends on django-uni-forms, and I have it listed in a requirements.txt file within my app which I used to install the dependency; but is that something that the packaging system can take care of?
What are the steps I need to follow to package my application in such a way that pip will be able to install it and any dependencies?

Yes, MANIFEST.in and setup.py should be sufficient.
This blog post really has some good information on this topic:
Packaging a Django reusable app
And here's another good, detailed overview that helped me a lot:
Python Packaging User Guide
Especially the tips to get your static files (templates) included are important as this might not be obvious at first.
And yes, you can specify required packages in your setup.py which are automatically fetched when installing your app.
For example:
install_requires = [
'django-profiles',
'django-uni-forms',
],
Obviously now we have two places where dependencies are defined, but that doesn't necessarily mean that these information are duplicated: setup.py vs requirements.txt
With this setup your package should be installable via pip.
As Pierre noted in the comments, there's now also a relevant section in Django's official documentation: Packaging your app
And then there is this "completely incomplete" guide, which really gives a great overview over packaging and uploading a package to PyPI: Sharing Your Labor of Love: PyPI Quick And Dirty

Creating portable Django apps - help needed

I'm building a Django app, which I comfortably run (test :)) on a Ubuntu Linux host. I would like to package the app without source code and distribute it to another production machine. Ideally the app could be run by ./runapp command which starts a CherryPy server that runs the python/django code.
I've discovered several ways of doing this:
Distributing the .pyc files only and building and installing all the requirements on target machine.
Using one of the many tools to package Python apps into a distributable package.
I'm really gunning for nr.2 option, I'd like to have my Django app contained, so it's possible to distribute it without needing to install or configure additional things. Searching the interwebs provided me with more questions than answers and a very sour taste that Django packing is an arcane art that everybody knows but nobody speaks about. :)
I've tried Freeze (fails), Cx_freeze (easy install version fails, repository version works, but the app output fails) and red up on dbuilder.py (which is supposed to work but doesn't work really - I guess). If I understand correctly most problems originate form the way that Django imports modules (example) but I have no idea how to solve it.
I'll be more than happy if anyone can provide any pointers or good resources online regarding packing/distributing standalone Django applications.

I suggest you base your distro on setuptools (a tool that enhances the standard Python distro mechanizm distutils).
Using setuptools, you should be able to create a Python egg containing your application. The egg's metadata can contain a list of dependencies that will be automatically installed by easy_install (can include Django + any third-party modules/packages that you use).
setuptools/distutils distros can include scripts that will be installed to /usr/bin, so that's how you can include your runapp script.
If you're not familiar with virtualenv, I suggest you take a look at that as well. It is a way to create isolated Python environments, it will be very useful for testing your distro.
Here's a blog post with some info on virtualenv, as well as a discussion about a couple of other nice to know tools: Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip

The --noreload option will stop Django auto-detecting which modules have changed. I don't know if that will fix it, but it might.
Another option (and it's not ideal) is to obscure some of your core functionality by packaging it as a dll, which your plain text code will call.

Initialize project layout in python?

Suppose a programmer has the following problem: he wants to start a new python project. He needs a basic layout of boilerplate stuff, like test directory, source directory, setuptools script etc.. How does he create all this stuff and layout with a single command ?
For example, paster (as suggested in one of the answers, provides you this service)
paster create
Selected and implied templates: PasteScript#basic_package
A basic setuptools-enabled package
but paster is part of a tool whose main scope is not the deployment of packages. What if I want to have a template for a library, and a template for an application? How can I modify the template to add my own personal stuff to it ?

You need something that supports templating to pull this off. The most used in the python community is pastescript.
easy_install pastescript # A one-time install
paster create
If you've already decided on the name of the package, than it's just:
paster create mypackage
If you want to customize the template, than the easiest way is to create your own python package that includes the custom template you want. Once you've installed it into your environment, you can then use this custom template as much as you want. (This is the sort of thing used by frameworks like pylons to create a template for a web application).
paster create -t libtemplate mypackage
paster create -t apptemplate mypackage
For more details on how to create templates (which consist of a mix of code and source files) take a look at: http://pythonpaste.org/script/developer.html#templates You'll notice that templates support inheritance, so that you can, e.g. just build upon the included template, or create your own, from-scratch templates.
For a good example of a customized template, you can take a look at the pylons template in source, here: Pylons Template Code
In addition, if you're not already using it, you should take a look at Ian Bicking's virtualenv. It allows you to create temporary 'virtual' environments that allow you to install python packages without using and/or conflicting with whatever system-wide packages you may have installed.
A standard setup with virtualenv and pastescript might look something like this:
mkdir mypackage && cd mypackage
virtualenv --distribute env
source env/bin/activate # 'Turns on / activates' the environment
easy_install pastescript
paster create mypackage

I'm using modern-package-template to layout my Python projects.
modern-package-template is a PasteScript template to create an initial layout for your Python projects using modern tools and practices followed in the Python community. Thus, your projects will have the following characteristics:
Use Distribute instead of setuptools as the BDFL himself supports it.
Buildout support, though you are not required to make use of it.
README.txt and NEWS.txt automatically included in your package metadata as long_description, thus making them appear in the PyPI page for your project.
Automatic script (or .exe) creation using Distribute
More info and download from pypi: http://pypi.python.org/pypi/modern-package-template

You can make your own templates. Really useful, for instance for in-house project structure standards.
Best way to start making your own is to start with an existing example and to copy/paste relevant bits from it. Suggestion: ZopeSkel as it is a quite big one with lots of examples. Browse the source code.

I've been using cookiecutter. It's written in python but can be used for any kind of project; not just python. It uses Jinja for templating and features pre and post hooks (written in python or bash) that can easily create/manage one's virtualenvs or anything else you can think of. You can store your own templates in a local directory or pull other peoples directly from the Internet and run them without storing them locally first. It seems much more versatile, simpler to use, and more useful IMHO then paster (disclosure: I've not tried paster). It also is in active development as well.