Develop Reference

Django - How to port scripts to django

Before I ask my question I need to give some context:
I wrote a simple python script that read linux's syslog file and search for certain strings. I have other similar scripts like these (scripts that do file system stuff, scripts that interact with other servers and so on). Most of these scripts write simple write stuff to stdout.
I would like to port these scripts to a web-server so I could simple browser to https://server/syslog and get the same output that I would get by running the script on the command line interface.
According with my research Django seems to be a great choice. I followed some Django tutorials and I was capable of developing some basic django web apps.
My question is: Since django does not have a "controller" where should I place the scripts code? My best bet in the view, but according with djangos documentation it does not make sence.
Extracted from django doc: In our interpretation of MVC, the “view” describes the data that gets presented to the user. It’s not necessarily how the data looks, but which data is presented. The view describes which data you see, not how you see it. It’s a subtle distinction.

The description of MVC is not so important. The typical use of django is for database backed web applications. And this describes a design pattern or paradigm for that. It's completely possible to use django in other ways as well.
If you want to build a django app that is a web interface for your existing scripts, you might not even need the django ORM at all. In any case, you can put as much or as little logic in your view as you want. Your use case might just not fit neatly into the MVC or MVT paradigm. Django views are just python functions (or classes, but Django class based views are more tightly coupled with the ORM).

I would recommend:
leaving your scripts largely as they are, but wrap the parts you want to reuse as
functions. You can keep them functional as standalone scripts with an
if __name__=='__main__':
block to call the functions.
Import the functions to views.py - it doesn't matter where they are as long as your server will always be able to find them. I put mine right in the app directory.
Call the function(s) in your view(s), and return the text to a HttpResponse object which you return from the view. (I think this is more direct than creating a template and a context and calling render, but its not what I usually do so there may be some issues?)

Thats bit old code - but you will get enough idea to start - check https://github.com/alex2/django_logtail (Django_LogTail)

What is a good Django workflow?

I'm a beginner to Python and Django.
When starting a new project what do you do first before diving into the code?
For example, one could take the following steps:
Configure the settings.py file first
Configure models.py to lay out data structure
Create template files
Define the views/pages
Syncdb
etc
So my question is, what is a good workflow to get through the required steps for a Django application? This also serves as a checklist of things to do. In the definitive guide to Django, the author talks about approaching top down or bottom up. Can anyone expand further on this and perhaps share their process?
Thanks.

Follow the Agile approach. Finish one small case, from the start to the end. From the models to the tests to user experience. Then build on it. Iterate.
Thats the right way to software development.
To do it efficiently, you need: (don't bother right away, you will need it.)
Automated schema migration, automated build system, auto updating and deployment. - None of these, django has got anything to do with. Use pip, fabric, hudson, twill and south appropriately.
Take care not to over burden yourself with all these right away, particularly since you say, you are beginning.

the required steps for a Django application?
There are two required steps.
Write the settings. Write the urls.py
The rest of the steps are optional.
This also serves as a checklist of things to do.
Bad policy. You don't need a checklist of Django features. You need a collection of use cases or user stories which you must implement.
For some reason, you've omitted the two most important and valuable features of Django. Configure the default admin interface and write unit tests. The default admin interface is very high value. Unit testing is absolutely central.
You do it like this.
Gather use cases.
Prioritize the use cases.
Define the actors. The classes of actors becomes groups in the security model.
Define enough "applications" to satisfy the first release of use cases. Define the url structure. Cool URL's don't change.
Build the first use case: models (including security), admin, urls, tests, forms, views and templates. Note that these are the file names (models.py, admin.py, ...) except for templates. Also note that forms and admin should be defined in separate modules even though this isn't required. Also note that templates will be split between a generic templates directory for top-level stuff and application-specific templates.
Build the second use case: models (including security), admin, urls, tests, forms, views and templates.
...
n. Package for release. Tweak up the settings. Configure database and mod-wsgi

I personally can't make a template without writing the views (unless it's a photoshop draft) but in general that's the way I go after I have a plan.
What's extremely important for me is that I don't dive head-first into the code, and that I spend time mocking up the model structure based on the "screens" or "pages" that the user will see.
Once I have a user experience defined, I make sure the backend is robust enough to handle that experience. If I don't visualize the user experience, details get left out that are certainly accomplishable in the shell but not ideal for the website, default django admin, etc.
There are always tradeoffs between agile development and a huge spec: I think there's an important balance. Agile is good: there's no point planning every detail before writing your first line of code, as your needs will change by the time you get to the end. You don't know how your users will really use the site.
On the other hand, without a plan, you can end up with a messy foundation that affects all future code.
An educated guess is a good start. Don't think or assume too much, but definitely have a clear idea how your users will interact with your site for stage 1.

Always try to remember about a DRY rule. For example, why to write RequestContext every new view is defined, where you can simply write a function once, which will add it for you. Good description is here in another topic.
Try to keep a code written one way. Each time you upgrade a schema of your view, edit it in all already written views. That will help keep your code clear and save a lot time for you in future.
Generally good rule, and how do I write my applications is the rule of small steps. Start with writing a settings and urls, then add one model and one view. When it works, modify - add another models or another views. You won't even notice, when your project becomes bigger and bigger.
And the last useful rule for clarity of all the source. Keep files in folders. If you have two subsites based one one (for example "accounts" and "blogs") create two directories names the same. Remeber to put init.py file in each directory. It's really easy to forget. With this practice it's easy to write models and views dedicated to each category. By the way it's a good practice to keep urls like in a tree structure. Main urls.py should contain only links like this one:
(r'^accounts/', include('your_main_name.accounts.urls')),
and of course all media, static, css and so on. In accounts directory urls keep:
urlpatterns = patterns('your_main_name.accounts.views',
url(r'^$', 'index', name='index'),
)
with all views subdirectories.
Last one - keep code clear with actuall django version. Remeber, that the 3.0 release is comming soon.
Hope this will help.

I find that my process varies depending on a lot of variables, mainly whether I know something will work or if I'm experimenting and also whether I'm developing on my production server or in a development environment.
For example, I often do my development directly on the deployment server (most of my work is for intranet projects so there isn't any security risk, etc). But when I do this I really need to make sure the settings and urls are setup first and that gunicorn and nginx are configured.
If I know something should work, or am setting up a generic base set of code, sometimes I'll do all that coding for views and models before I even get enough setup to even run the development server. But when experimenting with new code I find it's good to be able to test every step of the way, so in that case you need your servers running.
In general I do settings, models, syncdb, views, urls, templates, collectstatic, graphics/aesthetics
In general I leave my base.html very plain until the everything else is working, then I add css/js etc.
I guess my point here is that there isn't really a wrong answer for how you do it, and there isn't even only one best practice (as far as I'm concerned). When you do more work, you'll find what you are comfortable with and it'll even vary from project to project.
Good luck, hopefully you learn to love django!

here is something I do in general,
configure basic settings
configure root url.py
configure settings, url.py for static (media) files
create model
sync db
write views (use simple template, if needed)
once you are done with back end implementation
think about UI
prepare styles, scripts
start working on template implementation

Django: "projects" vs "apps"

I have a fairly complex "product" I'm getting ready to build using Django. I'm going to avoid using the terms "project" and "application" in this context, because I'm not clear on their specific meaning in Django.
Projects can have many apps. Apps can be shared among many projects. Fine.
I'm not reinventing the blog or forum - I don't see any portion of my product being reusable in any context. Intuitively, I would call this one "application." Do I then do all my work in a single "app" folder?
If so... in terms of Django's project.app namespace, my inclination is to use myproduct.myproduct, but of course this isn't allowed (but the application I'm building is my project, and my project is an application!). I'm therefore lead to believe that perhaps I'm supposed to approach Django by building one app per "significant" model, but I don't know where to draw the boundaries in my schema to separate it into apps - I have a lot of models with relatively complex relationships.
I'm hoping there's a common solution to this...

Once you graduate from using startproject and startapp, there's nothing to stop you from combining a "project" and "app" in the same Python package. A project is really nothing more than a settings module, and an app is really nothing more than a models module—everything else is optional.
For small sites, it's entirely reasonable to have something like:
site/
models.py
settings.py
tests.py
urls.py
views.py

Try to answer question: "What does my
application do?". If you cannot answer
in a single sentence, then maybe you can
split it into several apps with cleaner
logic.
I read this thought somewhere soon after I've started to work with django and I find that I ask this question of myself quite often and it helps me.
Your apps don't have to be reusable, they can depend on each other, but they should do one thing.

What is to stop you using myproduct.myproduct? What you need to achieve that roughly consists of doing this:
django-admin.py startproject myproduct
cd myproduct
mkdir myproduct
touch myproduct/__init__.py
touch myproduct/models.py
touch myproduct/views.py
and so on. Would it help if I said views.py doesn't have to be called views.py? Provided you can name, on the python path, a function (usually package.package.views.function_name) it will get handled. Simple as that. All this "project"/"app" stuff is just python packages.
Now, how are you supposed to do it? Or rather, how might I do it? Well, if you create a significant piece of reusable functionality, like say a markup editor, that's when you create a "top level app" which might contain widgets.py, fields.py, context_processors.py etc - all things you might want to import.
Similarly, if you can create something like a blog in a format that is pretty generic across installs, you can wrap it up in an app, with its own template, static content folder etc, and configure an instance of a django project to use that app's content.
There are no hard and fast rules saying you must do this, but it is one of the goals of the framework. The fact that everything, templates included, allows you to include from some common base means your blog should fit snugly into any other setup, simply by looking after its own part.
However, to address your actual concern, yes, nothing says you can't work with the top level project folder. That's what apps do and you can do it if you really want to. I tend not to, however, for several reasons:
Django's default setup doesn't do it.
Often, I want to create a main app, so I create one, usually called website. However, at a later date I might want to develop original functionality just for this site. With a view to making it removable (whether or not I ever do) I tend to then create a separate directory. This also means I can drop said functionality just by unlinking that package from the config and removing the folder, rather than a complex delete the right urls from a global urls.py folder.
Very often, even when I want to make something independent, it needs somewhere to live whilst I look after it / make it independent. Basically the above case, but for stuff I do intend to make generic.
My top level folder often contains a few other things, including but not limited to wsgi scripts, sql scripts etc.
django's management extensions rely on subdirectories. So it makes sense to name packages appropriately.
In short, the reason there is a convention is the same as any other convention - it helps when it comes to others working with your project. If I see fields.py I immediately expect code in it to subclass django's field, whereas if I see inputtypes.py I might not be so clear on what that means without looking at it.

I've found the following blog posts very useful about django applications and projects:
http://www.b-list.org/weblog/2006/sep/10/django-tips-laying-out-application/
http://web.archive.org/web/20080302205555/www.pointy-stick.com/blog/2007/11/09/django-tip-developing-without-projects/
In principle, you have a lot of freedom with django for organizing the source code of your product.

If so... in terms of Django's project.app namespace, my inclination is to usemyproduct.myproduct, but of course this isn't allowed
There is nothing like not allowed. Its your project, no one is restricting you. It is advisable to keep a reasonable name.
I don't see any portion of my product being reusable in any context. Intuitively, I would call this one "application." Do I then do all my work in a single "app" folder?
In a general django project there are many apps (contrib apps) which are used really in every project.
Let us say that your project does only one task and has only a single app (I name it main as thethe project revolves around it and is hardly pluggable). This project too still uses some other apps generally.
Now if you say that your project is using just the one app (INSTALLED_APPS='myproduct') so what is use of project defining the project as project.app, I think you should consider some points:
There are many other things that the code other than the app in a project handles (base static files, base templates, settings....i.e. provides the base).
In the general project.app approach django automatically defines sql schema from models.
Your project would be much easier to be built with the conventional approach.
You may define some different names for urls, views and other files as you wish, but I don't see the need.
You might need to add some applications in future which would be real easy with the conventional django projects which otherwise it may become equally or more difficult and tedious to do.
As far as most of the work being done in the app is concerned, I think that is the case with most of django projects.

Here Django creators points out that difference themselves.
I think that thinking about Apps as they have to be reusable in other projects is good. Also a good way of thinking about Apps in Django provide modern web applications.
Imagine that you are creating big dynamic web app basing on JavaScript.
You can create then in django App named e.g "FrontEnd" <-- in thins app you will display content.
Then you create some backend Apps. E.g App named "Comments" that will store user comments. And "Comments" App will not display anything itself. It will be just API for AJAX requests of your dynamic JS website.
In this way you can always reuse your "Comments" app. You can make it open source without opening source of whole project. And you keep clean logic of your project.

models.py getting huge, what is the best way to break it up?

Directions from my supervisor:
"I want to avoid putting any logic in the models.py. From here on out, let's use that as only classes for accessing the database, and keep all logic in external classes that use the models classes, or wrap them."
I feel like this is the wrong way to go. I feel that keeping logic out of the models just to keep the file small is a bad idea. If the logic is best in the model, that's where it really should go regardless of file size.
So is there a simple way to just use includes? In PHP-speak, I'd like to propose to the supervisor that we just have models.py include() the model classes from other places. Conceptually, this would allow the models to have all the logic we want, yet keep file size down via increasing the number of files (which leads to less revision control problems like conflicts, etc.).
So, is there a simple way to remove model classes from the models.py file, but still have the models work with all of the Django tools? Or, is there a completely different yet elegant solution to the general problem of a "large" models.py file? Any input would be appreciated.

It's natural for model classes to contain methods to operate on the model. If I have a Book model, with a method book.get_noun_count(), that's where it belongs--I don't want to have to write "get_noun_count(book)", unless the method actually intrinsically belongs with some other package. (It might--for example, if I have a package for accessing Amazon's API with "get_amazon_product_id(book)".)
I cringed when Django's documentation suggested putting models in a single file, and I took a few minutes from the very beginning to figure out how to split it into a proper subpackage.
site/models/__init__.py
site/models/book.py
__init__.py looks like:
from .book import Book
so I can still write "from site.models import Book".
The following is only required for versions prior to Django 1.7, see
https://code.djangoproject.com/ticket/3591
The only trick is that you need to explicitly set each model's application, due to a bug in Django: it assumes that the application name is the third-to-last entry in the model path. "site.models.Book" results in "site", which is correct; "site.models.book.Book" makes it think the application name is "models". This is a pretty nasty hack on Django's part; it should probably search the list of installed applications for a prefix match.
class Book(models.Model):
class Meta: app_label = "site"
You could probably use a base class or metaclass to generalize this, but I haven't bothered with that yet.

Django is designed to let you build many small applications instead of one big application.
Inside every large application are many small applications struggling to be free.
If your models.py feels big, you're doing too much. Stop. Relax. Decompose.
Find smaller, potentially reusable small application components, or pieces. You don't have to actually reuse them. Just think about them as potentially reusable.
Consider your upgrade paths and decompose applications that you might want to replace some day. You don't have to actually replace them, but you can consider them as a stand-alone "module" of programming that might get replaced with something cooler in the future.
We have about a dozen applications, each model.py is no more than about 400 lines of code. They're all pretty focused on less than about half-dozen discrete class definitions. (These aren't hard limits, they're observations about our code.)
We decompose early and often.

I can't quite get which of many possible problems you might have. Here are some possibilities with answers:
multiple models in the same file
Put them into separate files. If there are dependencies, use import to pull in the
additional models.
extraneous logic / utility functions in models.py
Put the extra logic into separate files.
static methods for selecting some model instances from database
Create a new Manager in a separate file.
methods obviously related to the model
save, __unicode__ and get_absolute_url are examples.

MVC and django fundamentals

Pretty new to this scene and trying to find some documentation to adopt best practices. We're building a fairly large content site which will consist of various media catalogs and I'm trying to find some comparable data / architectural models so that we can get a better idea of the approach we should use using a framework we've never made use of before. Any insight / help would be greatly appreciated!

"data / architectural models so that we can get a better idea of the approach we should use using a framework we've never made use of before"
Django imposes best practices on you. You don't have a lot of choices and can't make a lot of mistakes.
MVC (while a noble aspiration) is implemented as follows:
Data is defined in "models.py" files using the Django ORM models.
urls.py file maps URL to view function. Pick your URL's wisely.
View function does all processing, making use of models and methods in models
Presentation (via HTML templates) invoked by View function. Essentially no processing can be done in presentation, just lightweight iteration and decision-making
The model is defined for you. Just stick to what Django does naturally and you'll be happy.
Architecturally, you usually have a stack like this.
Apache does two things.
serves static content directly and immediately
hands dynamic URL to Django (via mod_python, mod_wsgi or mod_fastcgi). Django apps map URL to view functions (which access to database (via ORM/model) and display via templates.
Database used by Django view functions.
The architecture is well-defined for you. Just stick to what Django does naturally and you'll be happy.
Feel free to read the Django documentation. It's excellent; perhaps the best there is.

first, forget all MVC mantra. it's important to have a good layered structure, but MVC (as defined originally) isn't one, it was a modular structure, where each GUI module is split in these tree submodules. nothing to use on the web here.
in web development, it really pays to have a layered structure, where the most important layer is the storage/modelling one, which came to be called model layer. on top of that, you need a few other layers but they're really not anything like views and controllers in the GUI world.
the Django layers are roughly:
storage/modelling: models.py, obviously. try to put most of the 'working' concepts there. all the relationships, all the operations should be implemented here.
dispatching: mostly in urls.py. here you turn your URL scheme into code paths. think of it like a big switch() statement. try hard to have readable URLs, which map into user intentions. it will help a lot to add new functionality, or new ways to do the same things (like an AJAX UI later).
gathering: mostly the view functions, both yours and the prebuilt generic views. here you simply gather all the from the models to satisfy a user request. in surprisingly many cases, it just have to pick a single model instance, and everything else can be retrieved from relationships. for these URLs, a generic view is enough.
presentation: the templates. if the view gives you the data you need, it's simple enough to turn it into a webpage. it's here where you'll thank that the model classes have good accessors to get any kind of relevant data from any given instance.

To understand django fundementals and the django take on MVC, consult the following:
http://www.djangobook.com/
As a starting point to getting your hands dirty with ...
"...trying to find some comparable data / architectural models"
Here is a quick and dirty way to reverse engineer a database to get a models.py file,
which you can then inspect to see how django would handle it.
1.) get an er diagram that closely matches your target. For example something like this
http://www.databaseanswers.org/data_models/product_catalogs/index.htm
2.) create an sql script from the er diagram and create the database,
I suggest Postgre, as some MySQL
table type will not have forgien key constraints, but in a pinch MySQL or SQLITE
will do
3.) create and configure a django app to use that database. Then run:
python manage.py inspectdb
This will at least give you a models.py file which you can read to see how django attempts
to model it.
Note that the inspect command is intended to be a shortcut for dealing with legacy
database when developing in django, and as such is not perfect. Be sure to read the
following before attempting this:
http://docs.djangoproject.com/en/dev/ref/django-admin/#ref-django-admin