I'm having problems structuring classes in the Model part of an MVC pattern in my Python app. No matter how I turn things, I keep running into circular imports. Here's what I have:
Model/__init__p.y
should hold all Model class names so
I can do a "from Model import User"
e.g. from a Controller or a unit
test case
Model/Database.py
holds Database class
needs to import all Model classes to do ORM
initialization should be performed on first module import, i.e. no extra init calls or instantiations (all methods on Database class are #classmethods)
Model/User.py
contains User model class
needs access to Database class to do queries
should inherit from base class common to all Model classes to share functionality (database persistency methods, parameter validation code etc.)
I have yet to see a real world Python app employing MVC, so my approach is probably un-Pythonic (and possibly a language-agnostic mess on top of that...) - any suggestions on how to solve this?
Thanks, Simon
There is an inconsistency in your specification. You say Database.py needs to import all Model classes to do ORM but then you say the User class need access to the Database to do queries.
Think of these as layers of an API. The Database class provides an API (maybe object-oriented) to some physical persistence layer (such as DB-API 2.0). The Model classes, like User, use the Database layer to load and save their state. There is no reason for the Database.py class to import all the Model classes, and in fact you wouldn't want that because you'd have to modify Database.py each time you created a new Model class - which is a code smell.
Generally, we put it all in one file. This isn't Java or C++.
Start with a single file until you get some more experience with Python. Unless your files are gargantuan, it will work fine.
For example, Django encourages this style, so copy their formula for success. One module for the model. A module for each application; each application imports a common model.
Your Database and superclass stuff can be in your __init__.py file, since it applies to the entire package. That may reduce some of the circularity.
I think you have one issue that should be straightened. Circular references often result from a failure to achieve separation of concerns. In my opinion, the database and model modules shouldn't know much about each other, working against an API instead. In this case the database shouldn't directly reference any specific model classes but instead provide the functionality the model classes will need to function. The model in turn, should get a database reference (injected or requested) that it would use to query and persist itself.
Related
I am working on a Django application that uses the SimpleGmail package to fetch mails from a Gmail inbox and I need to persist them. Normally I'd have written a model for the class, but given it's an external class, I can't figure out how to cleanly persist it.
I have come across solutions such as making it an attribute of a new model that is persisted or multiple inheritance of the desired class, but none of these seem correct to me.
How do I properly register an external class as a model for Django's persistence?
It is too long to comment. So I will try to write an answer.
One way is is to create a model class with properties which can be mapped from external class with all properties.
Another way would be just import external class in your application and create an instance of this external class. I am sorry, I am not Python guy, so code implementation would not be provided.
I initially started a small python project (Python, Tkinter amd PonyORM) and became larger that is why I decided to divide the code (used to be single file only) to several modules (e.g. main, form1, entity, database). Main acting as the main controller, form1 as an example can contain a tkinter Frame which can be used as an interface where the user can input data, entity contains the db.Enttiy mappings and database for the pony.Database instance along with its connection details. I think problem is that during import, I'm getting this error "pony.orm.core.ERDiagramError: Cannot define entity 'EmpInfo': database mapping has already been generated". Can you point me to any existing code how should be done.
Probably you import your modules in a wrong order. Any module which contains entity definitions should be imported before db.generate_mapping() call.
I think you should call db.generate_mapping() right before entering tk.mainloop() when all imports are already done.
A good approach to avoid this is rather than having your db.generate_mapping() call happening at a module's top-level code, have a function that a module exports that calls db.generate_mapping() after all other modules have been imported.
The pattern I use is to put all of my db.Entity subclasses into a single module named model, and then at the bottom of model.py is:
def setup():
""" Set up the database """
db.bind(**database_config, create_db=True)
db.generate_mapping(create_tables=True)
This function is called by my application's own startup (which is also responsible for setting up database_config). This way the correct import and setup order can be guaranteed.
The db object itself is also owned by this model module; if I need to use it somewhere else I import model and use model.db.
If you want to further separate things out (with different model classes living in different modules) you can have a module that owns db, then your separate model modules, and then a third module that imports db and the models and provides the setup function. For example, your directory structure could look like this:
model/
__init__.py -- imports all of the model sub-modules and provides a setup function
db.py -- provides the db object itself and any common entities objects that everyone else needs
form1.py, form2.py, etc. -- imports db and uses its database object to define the entities
Then your main app can do something like:
import model
model.setup()
I have a models in different files (blog/models.py, forum/models.py, article/models.py). In each of this files I have defined model classes with application prefix (BlobPost, BlogTag, ForumPost, ForumThread, Article, ArticleCategory).
Also I have appliation - comment, for adding comment attached to any model object. For example, I want to comment BlogPost, or add comment referenced to ForumPost. For this I use property with type ReferenceProperty() - without specify type of references. Any model can attached to comment.
What a problem? If I have show all comments in administration section, I see a problem with autoloading models for ReferenceProperty. I don't know, what type of model used for current comment. I need to autoload package with model, if this need.
Yes, exists simple solution - include all models from all applications. But, this is not good solution. I need load only need models. How to do this autoloading?
My idea is based on detect kind of property, and by first part of this name detect application name for load all models in this application. For example, I have comment with Reference to BlogPost model. I get name of application - Blog and load all models from blog.models import *
For implement my idea I need to understand - how to intercept creating property instances. In my case, if I loop over comments, I see that App Engine automatically (thanks, but not in my case) create instances for properties.
How to inject my logic for loading my models before creating property instance?
Thank you!
This isn't possible in the standard db framework, as there's not enough information present to find your models. The only information the framework has to work with is the kind name, which doesn't include the fully qualified package - so it has no way to figure out what package your model definition might be in.
If you're writing an admin interface, though, you probably want to use the low-level google.appengine.api.datastore interface, instead, which operates on dicts instead of model classes, and doesn't require a model definition.
Directions from my supervisor:
"I want to avoid putting any logic in the models.py. From here on out, let's use that as only classes for accessing the database, and keep all logic in external classes that use the models classes, or wrap them."
I feel like this is the wrong way to go. I feel that keeping logic out of the models just to keep the file small is a bad idea. If the logic is best in the model, that's where it really should go regardless of file size.
So is there a simple way to just use includes? In PHP-speak, I'd like to propose to the supervisor that we just have models.py include() the model classes from other places. Conceptually, this would allow the models to have all the logic we want, yet keep file size down via increasing the number of files (which leads to less revision control problems like conflicts, etc.).
So, is there a simple way to remove model classes from the models.py file, but still have the models work with all of the Django tools? Or, is there a completely different yet elegant solution to the general problem of a "large" models.py file? Any input would be appreciated.
It's natural for model classes to contain methods to operate on the model. If I have a Book model, with a method book.get_noun_count(), that's where it belongs--I don't want to have to write "get_noun_count(book)", unless the method actually intrinsically belongs with some other package. (It might--for example, if I have a package for accessing Amazon's API with "get_amazon_product_id(book)".)
I cringed when Django's documentation suggested putting models in a single file, and I took a few minutes from the very beginning to figure out how to split it into a proper subpackage.
site/models/__init__.py
site/models/book.py
__init__.py looks like:
from .book import Book
so I can still write "from site.models import Book".
The following is only required for versions prior to Django 1.7, see
https://code.djangoproject.com/ticket/3591
The only trick is that you need to explicitly set each model's application, due to a bug in Django: it assumes that the application name is the third-to-last entry in the model path. "site.models.Book" results in "site", which is correct; "site.models.book.Book" makes it think the application name is "models". This is a pretty nasty hack on Django's part; it should probably search the list of installed applications for a prefix match.
class Book(models.Model):
class Meta: app_label = "site"
You could probably use a base class or metaclass to generalize this, but I haven't bothered with that yet.
Django is designed to let you build many small applications instead of one big application.
Inside every large application are many small applications struggling to be free.
If your models.py feels big, you're doing too much. Stop. Relax. Decompose.
Find smaller, potentially reusable small application components, or pieces. You don't have to actually reuse them. Just think about them as potentially reusable.
Consider your upgrade paths and decompose applications that you might want to replace some day. You don't have to actually replace them, but you can consider them as a stand-alone "module" of programming that might get replaced with something cooler in the future.
We have about a dozen applications, each model.py is no more than about 400 lines of code. They're all pretty focused on less than about half-dozen discrete class definitions. (These aren't hard limits, they're observations about our code.)
We decompose early and often.
I can't quite get which of many possible problems you might have. Here are some possibilities with answers:
multiple models in the same file
Put them into separate files. If there are dependencies, use import to pull in the
additional models.
extraneous logic / utility functions in models.py
Put the extra logic into separate files.
static methods for selecting some model instances from database
Create a new Manager in a separate file.
methods obviously related to the model
save, __unicode__ and get_absolute_url are examples.
In Ruby on Rails, each model entity has a "validate_on_something" hook method, that will be called before the entity is actually persisted to the database. I would like similar functionality in Google App Engine. I am aware that you can do validation on individual Properties by passing arguments to them in their declarations. However, if I wish to do more validation than that, is there some place within the model class declaration within which I can do that?
Also, along the same lines, sometimes a entity needs modification before it is actually persisted to the database. I might need to modify (transform) the entity right before it is actually written to the database. Is there some place in the entity class declaration that would allow me to do so?
I am aware that I can put these transformations/validations outside of the class. Bu this hardly seems like good OO design. It really seems like there should be hook methods that would automatically be called in a model for these sort of needs.
So my question is, what is the most appropriate way to handle the validation and transformation of entities before they are persisted?
The best answer depends on what sort of transformations you need to do. There's no generalized pre-/post- put methods for models, but there are several other options:
As you mentioned, you can pass validation functions to Property class constructors
You can use a custom property class that generates values programmatically, such as this one.
You can modify entities as they are stored at the lowest level using api call hooks.
Are you using any kind of web framework on top of the raw app engine api's? Rails is a very high level framework. Have you looked into Django or any of the other web frameworks? You may find those are closer to rails than raw appengine entities.
Alternatively, if you want something lower level, have a look at this article on
hooks