I have to use PyMongo in a Django project, althought I have always used either Django's ORM or Mongoengine. With the old setup, every model had its own method which did some actual work. This time, having to use PyMongo, I don't have the models, and I don't know where to put the helper functions.
Currently I'm writing them inside the models.py file, but I'm not even halfway done that already I feel that this way I'm cluttering the models file (too many little functions).
Do you know a better solution to the problem?
You are not using django ORM, you don't have any models - you should leave models.py empty.
Here's one option you may consider to use.
Create a package called, for example, db. Divide your helper functions into separate scripts according to the entity/collection they are working with. E.g.:
db/
__init__.py
user.py
customer.py
role.py
Other generic helper function you can leave in __init__.py or create a separate module for them.
Additionally, while splitting your current models.py file into these python modules you may notice that most of your helper functions are similar - in that case, think about going with OOP approach, create a python class for each collection, define generic methods etc (and it will look like inventing your own ORM layer).
Hope that helps.
You can organize the helper methods as classmethods of "model"-like classes. After re-organization, the models file may look something like this:
# models.py
class BaseModel(object):
connection = Connection()
db = connection['test']
collection = None # override in subclasss
#classmethod
def get(cls, object_id):
return cls.collection.findOne({"_id": object_id})
class Person(BaseModel):
collection = BaseModel.db['person']
#classmethod
def get_old_people(cls):
return cls.collection.find(...)
Related
I have an existing GAE app with a reasonably small number of entities, and I would like to update the entities to use polymodel.
I currently have entities like this:
class Mammal(db.Model)
class Reptile(db.Model)
and I'd like to change it to this:
class Animal(polymodel.Polymodel)
class Mammal(Animal)
class Reptile(Animal)
My current plan is to do the following procedure:
Iterate over all of the existing entities to change the class names to some temporary class name. E.g., convert class Mammal(db.Model) to class MammalTmp(db.Model) and convert class Reptile(db.Model) to class ReptileTmp(db.Model). In doing this, I would copy all of the properties of the old class to the new class.
Delete all instances of class Mammal(db.Model) and class Reptile(db.Model).
Iterate over all of the temporary entities to change the class names to the desired class name and type. E.g., convert class MammalTmp(db.Model) to class Mammal(polymodel.Polymodel) and convert class ReptileTmp(db.Model) to class Reptile(polymodel.Polymodel). I would again copy all of the properties of the old class to the new class.
Delete all instances of class MammalTmp(db.Model) and class ReptileTmp(db.Model).
This is a laborious procedure! Is there an easier way to accomplish this?
With the way entities are built and then indexed, no unfortunately there is no other way (as far as I know) to do a similar process. I had to do it when I first wanted to implement polymodels and that's the way I did it. Luckily all of these can be done through code so you don't really have to sit at your computer and change all of that manually.
It's lengthy for sure, but think of all the speed benefits the datastore offers. That's why you have to be careful about creating your models in the first place. I know it's not necessarily easy (as I said I fell in the same hole as you and had to write code for those iterations and changes).
A very good way to do such a process programmatically would be to use MapReduce. A "mapper" could definitely do the trick and help you do that faster and more efficiently. Looking into the sample projects might give you some pointers.
I'm not familiar at all with GAE, but could you just redirect your model definitions through an intermediary? I suppose this wouldn't be any faster than just renaming the base class for all your models, though.
Create a redirect class to start with:
# redirect.py
# note: I don't really know where db comes from...
import gae.db as db
class Model(db.Model):
pass
Then add this line to your model file:
import redirect as db
class Mammal(db.Model):
pass
And since db is now the redirect version, you can change the redirect file..
class Model (db.PolyModel):
pass
But now that I've written it, it sounds like just as much work as manually updating the files, and you lose all access to db. for other basic operations. So, maybe I should just downvote my own answer. :D
I have a utility function in my Django project, it takes a queryset, gets some data from it and returns a result. I'd like to write some tests for this function. Is there anyway to 'mock' a QuerySet? I'd like to create an object that doesn't touch the database, and i can provide it with a list of values to use (i.e. some fake rows) and then it'll act just like a queryset, and will allow someone to do field lookups on it/filter/get/all etc.
Does anything like this exist already?
For an empty Queryset, I'd go simply for using none as keithhackbarth has already stated.
However, to mock a Queryset that will return a list of values, I prefer to use a Mock with a spec of the Model's manager. As an example (Python 2.7 style - I've used the external Mock library), here's a simple test where the Queryset is filtered and then counted:
from django.test import TestCase
from mock import Mock
from .models import Example
def queryset_func(queryset, filter_value):
"""
An example function to be tested
"""
return queryset.filter(stuff=filter_value).count()
class TestQuerysetFunc(TestCase):
def test_happy(self):
"""
`queryset_func` filters provided queryset and counts result
"""
m_queryset = Mock(spec=Example.objects)
m_queryset.filter.return_value = m_queryset
m_queryset.count.return_value = 97
result = func_to_test(m_queryset, '__TEST_VALUE__')
self.assertEqual(result, 97)
m_queryset.filter.assert_called_once_with(stuff='__TEST_VALUE__')
m_queryset.count.assert_called_once_with()
However, to fulfil the question, instead of setting a return_value for count, this could easily be adjusted to be a list of model instances returned from all.
Note that chaining is handled by setting the filter to return the mocked queryset:
m_queryset.filter.return_value = m_queryset
This would need to be applied for any queryset methods used in the function under test, e.g. exclude, etc.
Of course you can mock a QuerySet, you can mock anything.
You can create an object yourself, and give it the interface you need, and have it return any data you like. At heart, mocking is nothing more than providing a "test double" that acts enough like the real thing for your tests' purposes.
The low-tech way to get started is to define an object:
class MockQuerySet(object):
pass
then create one of these, and hand it to your test. The test will fail, likely on an AttributeError. That will tell you what you need to implement on your MockQuerySet. Repeat until your object is rich enough for your tests.
I am having the same issue, and it looks like some nice person has written a library for mocking QuerySets, it is called mock-django and the specific code you will need is here https://github.com/dcramer/mock-django/blob/master/mock_django/query.py I think you can then just patch you models objects function to return one of these QuerySetMock objects that you have set up to return something expected!
For this I use Django's .none() function.
For example:
class Location(models.Model):
name = models.CharField(max_length=100)
mock_locations = Location.objects.none()
This is the method used frequently in Django's own internal test cases. Based on comments in the code
Calling none() will create a queryset that never returns any objects and no
+query will be executed when accessing the results. A qs.none() queryset
+is an instance of ``EmptyQuerySet``.
Try out the django_mock_queries library that lets you mock out the database access, and still use some of the Django query set features like filtering.
Full disclosure: I contributed some features to the project.
Have you looked into FactoryBoy? https://factoryboy.readthedocs.io/en/latest/orms.html
It's a fixtures replacement tool with support for the django orm - factories basically generate orm-like objects (either in memory or in a test database).
Here's a great article for getting started: https://www.caktusgroup.com/blog/2013/07/17/factory-boy-alternative-django-testing-fixtures/
One first advice would be to split the function in two parts, one that creates the queryset
and one that manipulates the output of it. In this way testing the second part is straightforward.
For the database problem, I investigated if django uses sqlite-in-memory and I found out that
recent version of django uses the sqlite -in-memory database, from The django unittest page:
When using the SQLite database engine the tests will by default use an
in-memory database (i.e., the database will be created in memory,
bypassing the filesystem entirely!).
Mocking the QuerySet object will not make you exercise its full logic.
You can mock like this:
#patch('django.db.models.query.QuerySet')
def test_returning_distinct_records_for_city(self, mock_qs):
self.assertTrue(mock_qs.called)
Not that I know of, but why not use an actual queryset? The test framework is all set up to allow you to create sample data within your test, and the database is re-created on every test, so there doesn't seem to be any reason not to use the real thing.
This is a pretty simple django patterns question. My manager code usually lives in models.py, but what happens when models.py is really huge? Is there any other alternative pattern to letting your manager code live in models.py for maintainability and to avoid circular imports?
A question may be asked as to why models.py is so huge, but let's just assume it's size and breadth of utility is justified.
I prefer to keep my models in models.py and managers in managers.py (forms in forms.py) all within the same app. For more generic managers, I prefer to keep them in core.managers if they can be re-used for other apps. In some of our larger apps with models/modelname.py that will contains a manager and the model code which doesn't seem bad.
Your best bet with a large set of models is to use django modules to your advantage, and simply create a folder named models. Move your old models.py into this models folder, and rename it __init__.py. This will allow you to then separate each model into more specific files inside of this model folder.
You would then only need to import each model into your __init__.py's namespace.
So, for instance, you might want to separate it into:
yourapp/
models/
__init__.py # This file should import anything from your other files in this directory
basic.py # Just an example name
morespecificmodels.py # Just an example name
managers.py # Might want to separate your manager into this
Then your __init__.py can just be:
from basic import * # You should replace * with each models name, most likely.
from managers import YourManager # Whatever your manager is called.
This is the structure that I use when my model files get huge, however I try to separate things into more pluggable apps as often as possible - so this is rarely used by me.
Hope this helps.
I always place mine in managers.py. If you have a circular import issue remember that a) You can reference the model class for a manager at self.model, and b) You can do imports inside of functions.
What I did when building Django apps was to create a [modelname].py file with just the specific model code, manager code and sometimes form code and used an __init__.py file to import then all in the models directory. This helped me atleast in keeping it managable.
I'm having problems structuring classes in the Model part of an MVC pattern in my Python app. No matter how I turn things, I keep running into circular imports. Here's what I have:
Model/__init__p.y
should hold all Model class names so
I can do a "from Model import User"
e.g. from a Controller or a unit
test case
Model/Database.py
holds Database class
needs to import all Model classes to do ORM
initialization should be performed on first module import, i.e. no extra init calls or instantiations (all methods on Database class are #classmethods)
Model/User.py
contains User model class
needs access to Database class to do queries
should inherit from base class common to all Model classes to share functionality (database persistency methods, parameter validation code etc.)
I have yet to see a real world Python app employing MVC, so my approach is probably un-Pythonic (and possibly a language-agnostic mess on top of that...) - any suggestions on how to solve this?
Thanks, Simon
There is an inconsistency in your specification. You say Database.py needs to import all Model classes to do ORM but then you say the User class need access to the Database to do queries.
Think of these as layers of an API. The Database class provides an API (maybe object-oriented) to some physical persistence layer (such as DB-API 2.0). The Model classes, like User, use the Database layer to load and save their state. There is no reason for the Database.py class to import all the Model classes, and in fact you wouldn't want that because you'd have to modify Database.py each time you created a new Model class - which is a code smell.
Generally, we put it all in one file. This isn't Java or C++.
Start with a single file until you get some more experience with Python. Unless your files are gargantuan, it will work fine.
For example, Django encourages this style, so copy their formula for success. One module for the model. A module for each application; each application imports a common model.
Your Database and superclass stuff can be in your __init__.py file, since it applies to the entire package. That may reduce some of the circularity.
I think you have one issue that should be straightened. Circular references often result from a failure to achieve separation of concerns. In my opinion, the database and model modules shouldn't know much about each other, working against an API instead. In this case the database shouldn't directly reference any specific model classes but instead provide the functionality the model classes will need to function. The model in turn, should get a database reference (injected or requested) that it would use to query and persist itself.
I'd like to move my models to a separate directory, similar to the way it's done with Rails to cut down on code clutter. Is there any way to do this easily?
Thanks,
Collin
I assume you're using the basic webkit and not Django or something fancy. In that case just create a subdirectory called models. Put any python files you use for your models in here. Create also one blank file in this folder called __init__.py.
Then in your main.py or "controller" or what have you, put:
import models
at the top.
You just created a python package.
Brandon's answer is what I do. Furthermore, I rather like Rails's custom of one model per file. I don't stick to it completely but that is my basic pattern, especially since Python tends to encourage more-but-simpler lines of code than Ruby.
So what I do is I make models a package too:
models/
models/__init__.py
models/user.py
models/item.py
models/blog_post.py
In the main .py files I put my basic class definition, plus perhaps some helper functions (Python's module system makes it much safer to keep quickie helper functions coupled to the class definition). And my __init__.py stitches them all together:
"""The application models"""
from user import User
from item import Item
from blog_post import BlogPost
It's slightly redundant but I have lots of control of the namespace.