Develop Reference

django: taking DRY & code reuse to the next level?

In the django culture, I have encountered the concept of app reuse but not snippet reuse. Here is an example of what I mean by snippet reuse: I have a function getDateTimeObjFromString( sDateTime ), obviously you pass a string date time and it returns a python date-time object.
Back in the late 1980's or early 1990's, I was exposed to the idea of snippet reuse at a FoxPro developers conference. If you write code for a specific problem and find it is useful elsewhere in your project, move it to a project library. If you find that code is useful for other projects, move it to a generic library that can be accessed by all projects.
(At the FoxPro DevCon, they did not call it snippet reuse. I coined that term to make clear that I am referring to reuse of chunks of code smaller than an entire app. The FoxPro DevCon was long ago, I do not remember exactly what they called it.)
I read the most recent "Two Scoops of Django", and it does mention reusing snippets within a single project but I did not find any mention of the concept of snippet reuse across multiple projects.
I wrote and used getDateTimeObjFromString() long before I tackled my django app. It is in packages I keep under /home/Common/pyPacks. On my computers, I set PYTHONPATH=/home/Common/pyPacks, so every project can access the code there. The code for getDateTimeObjFromString() is under a Time subdirectory in a file named Convert.py. So to use the code in any project:
from Time.Convert import getDateTimeObjFromString
My django app downloads data from an API, and that data includes timestamps. It would be nice if the API sent python date time objects, but what you get are strings. Hence the utility of getDateTimeObjFromString().
This is just one example, there are many little functions under /home/Common/pyPacks that I found convenient to access in my django project.
Yes /home/Common/pyPacks are under version control in github and yes I deploy on any particular machine via git pull.
When working on my django project from a development computer, PYTHONPATH works, and I can import the packages. But then I tried running my django app on a server via wsgi.py -- PYTHONPATH is disabled. I can set PYTHONPATH both at the OS and Apache2 level, but python ignores it, the functions cannot import.
I do not want to bother with making my personal generic library an official python package under PyPI.
Does the django community expect me to copy and paste?
I arrived at a work around: make /home/Common/pyPacks a psudeo site-package by putting a "pyPks" symlink in the virtual environment's site-packages directory to /home/Common/pyPacks, adding "pyPks" to the INSTALLED_APPS, then changing all the import statements as follows:
original:
from Time.Convert import getDateTimeObjFromString
work around update:
from pyPks.Time.Convert import getDateTimeObjFromString
I also had to update all my generic library files to handle both absolute imports via PYTHONPATH and relative imports.
How to fix “Attempted relative import in non-package” even with init.py
Is there a better way to reuse snippets in a django project?

How to override an app in Django properly?

I'm running Satchmo. There are many apps and I've changed some of the source in the Product app.
So my question is how can I override this properly because the change is site specific. Do I have to copy over the whole Satchmo framework and put it into my project or can I just copy one of the apps out and place it in say Satchmo>App>Products? (Kinda like with templates)
Thanks

What I have done which works is to copy the application that I have changed. In this case satchmo\apps\product.
I copied the app into my project folder
Amended my setting.py INSTALLED_APPS from 'product', to 'myproject.product',
This now carries the changes I've made to this app for this project only and leaves the original product app untouched and still able to be read normally from other projects.

When you add a 'Django App' to INSTALLED_APPS in your settings.py file, you're telling Django that there exists an importable python module with that name on your "python path". You can view your python path by viewing the contents of the list stored at sys.path.
Whenever Python (and in this case Django) attempts to import a module it checks each of the directories listed in sys.path in order, when it finds a module matching the given name it stops.
The solution to your problem then is too place your customized Django Apps, e.g., the Satchmo product module, into a location in your python path which will be checked before the "real" Satchmo product module.
Because I don't know how you have your directory structure laid out I'm basically making a guess here, but in your case, it sounds like you have the Satchmo apps living somewhere like /satchmo/apps/ and your project at /my_custom_path/my_project/. In which case you might want to add your customized product module to /my_custom_path/my_project/product/. Because the path at which your Django settings.py file lives is always checked first, that should mean that your customized product module will be found first and imported instead of the built in Satchmo one.
FYI: To check and see the order in which your Satchmo installation is checking directories for modules run python manage.py shell and then in the prompt do import sys; print sys.path.

Normally, I'd say the best thing is to fork Satchmo and keep a copy with your changes.
If you are willing to play with the python path, make sure that your app's directory appears before the other (default) directory. From my tests, if you have two apps/modules with identical names, the first one found is used.

Django, Python Modules, and Git Submodules

I am working on a django project that has uses multiple applications (python modules). Most of those python modules are maintained by other people in their own git repositories. I use the git-submodules command to import them into my project under the 'apps' directory like so:
mysite/
mysite/apps
mysite/apps/django-extensions
mysite/apps/django-celery
mysite/apps/django-comments
mysite/apps/myapp
...etc
Most of those submodules (take django-extensions for example) have a subfolder containing the actual python module: mysite/apps/django-extensions/django_extensions
This means I can't simply set my python path to include mysite/apps--I have to set it to include mysite/apps/django-extensions so it can import the django_extensions subfolder.
It gets annoying typing:
PYTHONPATH=mysite/apps/django-extensions:mysite/apps/django-celery... python manage.py runserver
Is there an easier way I should be laying out my repo? An easier process?
Just for fun, I tried a PYTHONPATH of mysite/apps/*, but that didn't work.

This is the wrong way to do it. Don't install other people's third-party code in your own project file. Instead, create a virtualenv, and install the code directly using pip.

After coming up blank on the internet, I hacked this solution together. It's straight forward and works well enough:
#At the top of settings.py
import sys, os
git_sub_modules = '/path/to/dir/containing/submodules' #Relative paths ok too
for dir in os.listdir(git_sub_modules):
path = os.path.join(git_sub_modules, dir)
if not path in sys.path:
sys.path.append(path)
time passes
UPDATE: It's much easier to use a virtualenv and/or something like dokku for deploying apps. I no longer use this. Although it is still a pain to checkout 3rd party apps that need 'tweaks' and use them in the project.

You could tuck those paths in a dependencies.pth file, and only have the .pth in your path. There are examples in your site-packages / dist-packages.

Could you try checking out just the wanted part of the repository? So if they have the actual code inside of what you're checking out, don't check out the extra part.
So instead of getting django-extensions get django-extensions/django-extensions.
Edit: I believe this is what you could do above.
Also, I believe you could get away with adding an __init__.py in the first django-extensions directory, but then you're going to have to add an extra django-extensions to your imports as well (__init__.py tells python it is a package). While I think this might work, I would recommend shooting for my first example.

Why are django projects python packages?

Why aren't they simply directories? Any good advice says to keep as much as possible in the apps and not to couple them to the project. The very ability to import an app as project.application discourages this. Why does django-admin.py create the __init__.py at all? The project is perfectly useful without it. What is the justification?

We have a single project that we "subclass" of sorts for other projects. So we have other projects that import stuff from the main project. I guess for us it provides the common namespace that contains all the other apps.
We could move to a package with all our apps in it separate from the projects i guess. Our system has grown rather than been planned.
So I guess my answer is, it provides a good root namespace. (for our needs) :)

The core of a project is a settings.py and a root urls.py. Both of those are Python modules, thus they need to be importable somehow. You can put the project directory directly on the Python path and thus make them importable as top-level modules, but that's arguably even worse practice. Better to have the project be a package and the settings and urls be modules within it.

There isn't a requirement that apps be inside the project's namespace, to my knowledge. Just that they be on the $PYTHONPATH. As such, they are usable by any other code on the system which shares the same PYTHONPATH.

I think the idea is that you can reuse the applications but you don't need to move them from the project where they were initially created. If the project weren't a package you would need to copy/move the application you want to reuse to a python package. Because of that being the project itself a proper python package you can reuse the applications in it without moving the applications to other place.

On the google app engine, why does my 'import' statement fail on Live, but work on Dev(localmachine)?

I have a python/django application that runs on the google app engine.
My views.py file has some imports...
from commands.userCommands import RegisterUserCommand
from commands.accountCommands import CreateNewAccountCommand, RenameAccountCommand
These imports work fine on my development environment (local machine). But when I upload to the google app engine, views.py fails with a "Could not import views. Error was: No module named userCommands" error.
Any idea why I can't import my commands.userCommands module?
My file structure looks as follows...
- app.yaml
- urls.py
- views.py
- etc...
- commands/__init__.py
- commands/userCommands.py
Note: I did try to append my application name to the module name/path. No luck.
Note: I did do an update with the --noisy argument, and it does appear to upload my commands folder successfully.

You could be running into a clash with Python's own commands module (which doesn't have submodules like yours) -- naming your own modules and packages in ways that are meant to hide ones in the standard library (just like naming your variables in ways that are meant to hide builtin names, like list or file) is always a perilous undertaking, even though it "should" work there's always potential for confusion.
Could you try renaming that commands package and its uses to something unambiguous and free from danger, such as mycommands, and see if that just makes the problem disappear? If that's the case you can then open a ticket on GAE's tracker (because it would show a minor but undeniable bug in GAE's runtime) but meanwhile your problem is solved!-) If the problem stays, ah well, at least we've eliminated one likely cause and can keep digging...

The __init__.py files are required to make Python treat the directories as containing packages, so you need a
commands/__init__.py
file in your directory structure. See http://docs.python.org/tutorial/modules.html.