GAE: Convert model to subclass of polymodel - python

I have an existing GAE app with a reasonably small number of entities, and I would like to update the entities to use polymodel.
I currently have entities like this:
class Mammal(db.Model)
class Reptile(db.Model)
and I'd like to change it to this:
class Animal(polymodel.Polymodel)
class Mammal(Animal)
class Reptile(Animal)
My current plan is to do the following procedure:
Iterate over all of the existing entities to change the class names to some temporary class name. E.g., convert class Mammal(db.Model) to class MammalTmp(db.Model) and convert class Reptile(db.Model) to class ReptileTmp(db.Model). In doing this, I would copy all of the properties of the old class to the new class.
Delete all instances of class Mammal(db.Model) and class Reptile(db.Model).
Iterate over all of the temporary entities to change the class names to the desired class name and type. E.g., convert class MammalTmp(db.Model) to class Mammal(polymodel.Polymodel) and convert class ReptileTmp(db.Model) to class Reptile(polymodel.Polymodel). I would again copy all of the properties of the old class to the new class.
Delete all instances of class MammalTmp(db.Model) and class ReptileTmp(db.Model).
This is a laborious procedure! Is there an easier way to accomplish this?

With the way entities are built and then indexed, no unfortunately there is no other way (as far as I know) to do a similar process. I had to do it when I first wanted to implement polymodels and that's the way I did it. Luckily all of these can be done through code so you don't really have to sit at your computer and change all of that manually.
It's lengthy for sure, but think of all the speed benefits the datastore offers. That's why you have to be careful about creating your models in the first place. I know it's not necessarily easy (as I said I fell in the same hole as you and had to write code for those iterations and changes).
A very good way to do such a process programmatically would be to use MapReduce. A "mapper" could definitely do the trick and help you do that faster and more efficiently. Looking into the sample projects might give you some pointers.

I'm not familiar at all with GAE, but could you just redirect your model definitions through an intermediary? I suppose this wouldn't be any faster than just renaming the base class for all your models, though.
Create a redirect class to start with:
# redirect.py
# note: I don't really know where db comes from...
import gae.db as db
class Model(db.Model):
pass
Then add this line to your model file:
import redirect as db
class Mammal(db.Model):
pass
And since db is now the redirect version, you can change the redirect file..
class Model (db.PolyModel):
pass
But now that I've written it, it sounds like just as much work as manually updating the files, and you lose all access to db. for other basic operations. So, maybe I should just downvote my own answer. :D

Related

Django: Are Django models dataclasses?

Can we say that Django models are considered dataclasses? I don't see #dataclass annotation on them or on their base class model.Models. However, we do treat them like dataclasses because they don't have constructors and we can create new objects by naming their arguments, for example MyDjangoModel(arg1= ..., arg2=...).
On the other hand, Django models also don't have init methods (constructors) or inherit from NamedTuple class.
What happens under the hood that I create new Django model objects?
A lot of the magic that happens with models, if not nearly all of it, is from its base meta class.
This can be found in django.db.models.ModelBase specifically in the __new__ function.
Regardless of an __init__ method being defined or not (which actually, it is as per Abdul's comment), doesn't mean it can or should be considered a dataclass.
As described very eloquently in this SO post by someone else;
What are data classes and how are they different from common classes?
Despite django models quite clearly and apparently seeming to have some kind of data stored in them, the models are more like an easy to use (and reuse) set of functions which leverage a database backend, which is where the real state of an object is stored, the model just gives access to it.
It's also worth noting that models don't store data, but simply retrieves it.
Take for example this simple model:
class Person(models.Model):
name = models.CharField()
And then we did something like this in a shell:
person = Person.objects.get(...)
print(person.name)
When we access the attribute, django is actually asking the database for the information and this generates a query to get the value.
The value isn't ACTUALLY stored on the model object itself.
With that in mind, inherently, django models ARE NOT dataclasses. They are plain old regular classes.
Django does not work with data classes. You can define a custom model field. But likely this will take some development work.

django models and OOP design

I wrote a quest system for an online game. My quests are serialized into json objects for a JavaScript client that fetches those quests then from a REST backend (I use django RestFramework)
Now I'm wondering on which class or django model I should put the "behaviour" that belongs to the data.
I stored the data that belongs to a quest in several separate models:
A model QuestHistory: with models.Fields like Boolean completed, and Datetime started where I put the information belonging to a specific user (it also as a field user).
Then I have a model QuestTemplate : The part that is always the same, fields like quest_title and quest_description
I also have a model Rewards and model Task and TaskHistory that are linked to a quest with a foreign Key field.
To combine this information back to quest I created a pure python class Quest(object): and defined methods on this class like check_quest_completion. This class is the then later serialized. The Problem with this approach is that It becomes quite verbose, for example when I instantiate this class or when I define the Serializer.
Is there a python or django "shortcut" to put all fields of a django model into another class (my Quest class here), something similar to the dict.update method maybe?
Or should I try to put the methods on the models instead and get rid of the Quest class?
I have some other places in my game that look very similar to the quest system for example the inventory system so I'm hoping for a more elegant solution.
You should put the methods of the Quest class on the model itself and get rid of the Quest class.

Where to put helper functions when not using models and an ORM?

I have to use PyMongo in a Django project, althought I have always used either Django's ORM or Mongoengine. With the old setup, every model had its own method which did some actual work. This time, having to use PyMongo, I don't have the models, and I don't know where to put the helper functions.
Currently I'm writing them inside the models.py file, but I'm not even halfway done that already I feel that this way I'm cluttering the models file (too many little functions).
Do you know a better solution to the problem?
You are not using django ORM, you don't have any models - you should leave models.py empty.
Here's one option you may consider to use.
Create a package called, for example, db. Divide your helper functions into separate scripts according to the entity/collection they are working with. E.g.:
db/
__init__.py
user.py
customer.py
role.py
Other generic helper function you can leave in __init__.py or create a separate module for them.
Additionally, while splitting your current models.py file into these python modules you may notice that most of your helper functions are similar - in that case, think about going with OOP approach, create a python class for each collection, define generic methods etc (and it will look like inventing your own ORM layer).
Hope that helps.
You can organize the helper methods as classmethods of "model"-like classes. After re-organization, the models file may look something like this:
# models.py
class BaseModel(object):
connection = Connection()
db = connection['test']
collection = None # override in subclasss
#classmethod
def get(cls, object_id):
return cls.collection.findOne({"_id": object_id})
class Person(BaseModel):
collection = BaseModel.db['person']
#classmethod
def get_old_people(cls):
return cls.collection.find(...)

Is there any way that classes always inherit from some base class

In my application i have the requirement of keppling logs of all models changes and delete.
So i have created baseclass Audit and extended all classes from it.
I have overridden save , delete methods in it so that i keep old chnages as well when we do some updation.
I want to know that is there any better way of doing that rather than extending all classes fron base class. Or is it all right like that.
For this use case, you may be able to write a generic function that could be used with django signals.
https://docs.djangoproject.com/en/dev/topics/signals/

How to store a dictionary on a Django Model?

I need to store some data in a Django model. These data are not equal to all instances of the model.
At first I thought about subclassing the model, but I’m trying to keep the application flexible. If I use subclasses, I’ll need to create a whole class each time I need a new kind of object, and that’s no good. I’ll also end up with a lot of subclasses only to store a pair of extra fields.
I really feel that a dictionary would be the best approach, but there’s nothing in the Django documentation about storing a dictionary in a Django model (or I can’t find it).
Any clues?
If it's really dictionary like arbitrary data you're looking for you can probably use a two-level setup with one model that's a container and another model that's key-value pairs. You'd create an instance of the container, create each of the key-value instances, and associate the set of key-value instances with the container instance. Something like:
class Dicty(models.Model):
name = models.CharField(max_length=50)
class KeyVal(models.Model):
container = models.ForeignKey(Dicty, db_index=True)
key = models.CharField(max_length=240, db_index=True)
value = models.CharField(max_length=240, db_index=True)
It's not pretty, but it'll let you access/search the innards of the dictionary using the DB whereas a pickle/serialize solution will not.
Another clean and fast solution can be found here: https://github.com/bradjasper/django-jsonfield
For convenience I copied the simple instructions.
Install
pip install jsonfield
Usage
from django.db import models
from jsonfield import JSONField
class MyModel(models.Model):
json = JSONField()
If you don't need to query by any of this extra data, then you can store it as a serialized dictionary. Use repr to turn the dictionary into a string, and eval to turn the string back into a dictionary. Take care with eval that there's no user data in the dictionary, or use a safe_eval implementation.
For example, in the create and update methods of your views, you can add:
if isinstance(request.data, dict) == False:
req_data = request.data.dict().copy()
else:
req_data = request.data.copy()
dict_key = 'request_parameter_that_has_a_dict_inside'
if dict_key in req_data.keys() and isinstance(req_data[dict_key], dict):
req_data[dict_key] = repr(req_data[dict_key])
I came to this post by google's 4rth result to "django store object"
A little bit late, but django-picklefield looks like good solution to me.
Example from doc:
To use, just define a field in your model:
>>> from picklefield.fields import PickledObjectField
>>> class SomeObject(models.Model):
>>> args = PickledObjectField()
and assign whatever you like (as long as it's picklable) to the field:
>>> obj = SomeObject()
>>> obj.args = ['fancy', {'objects': 'inside'}]
>>> obj.save()
As Ned answered, you won't be able to query "some data" if you use the dictionary approach.
If you still need to store dictionaries then the best approach, by far, is the PickleField class documented in Marty Alchin's new book Pro Django. This method uses Python class properties to pickle/unpickle a python object, only on demand, that is stored in a model field.
The basics of this approach is to use django's contibute_to_class method to dynamically add a new field to your model and uses getattr/setattr to do the serializing on demand.
One of the few online examples I could find that is similar is this definition of a JSONField.
I'm not sure exactly sure of the nature of the problem you're trying to solve, but it sounds curiously similar to Google App Engine's BigTable Expando.
Expandos allow you to specify and store additional fields on an database-backed object instance at runtime. To quote from the docs:
import datetime
from google.appengine.ext import db
class Song(db.Expando):
title = db.StringProperty()
crazy = Song(title='Crazy like a diamond',
author='Lucy Sky',
publish_date='yesterday',
rating=5.0)
crazy.last_minute_note=db.Text('Get a train to the station.')
Google App Engine currently supports both Python and the Django framework. Might be worth looking into if this is the best way to express your models.
Traditional relational database models don't have this kind of column-addition flexibility. If your datatypes are simple enough you could break from traditional RDBMS philosophy and hack values into a single column via serialization as #Ned Batchelder proposes; however, if you have to use an RDBMS, Django model inheritance is probably the way to go. Notably, it will create a one-to-one foreign key relation for each level of derivation.
This question is old, but I was having the same problem, ended here and the chosen answer couldn't solve my problem anymore.
If you want to store dictionaries in Django or REST Api, either to be used as objects in your front end, or because your data won't necessarily have the same structure, the solution I used can help you.
When saving the data in your API, use json.dump() method to be able to store it in a proper json format, as described in this question.
If you use this structure, your data will already be in the appropriate json format to be called in the front end with JSON.parse() in your ajax (or whatever) call.
I use a textfield and json.loads()/json.dumps()
models.py
import json
from django.db import models
class Item(models.Model):
data = models.TextField(blank=True, null=True, default='{}')
def save(self, *args, **kwargs):
## load the current string and
## convert string to python dictionary
data_dict = json.loads(self.data)
## do something with the dictionary
for something in somethings:
data_dict[something] = some_function(something)
## if it is empty, save it back to a '{}' string,
## if it is not empty, convert the dictionary back to a json string
if not data_dict:
self.data = '{}'
else:
self.data = json.dumps(data_dict)
super(Item, self).save(*args, **kwargs)
Django-Geo includes a "DictionaryField" you might find helpful:
http://code.google.com/p/django-geo/source/browse/trunk/fields.py?r=13#49
In general, if you don't need to query across the data use a denormalized approach to avoid extra queries. User settings are a pretty good example!
I agree that you need to refrain stuffing otherwise structured data into a single column. But if you must do that, Django has an XMLField build-in.
There's also JSONField at Django snipplets.
Being "not equal to all instances of the model" sounds to me like a good match for a "Schema-free database". CouchDB is the poster child for that approach and you might consider that.
In a project I moved several tables which never played very nice with the Django ORM over to CouchDB and I'm quite happy with that. I use couchdb-python without any of the Django-specific CouchDB modules. A description of the data model can be found here. The movement from five "models" in Django to 3 "models" in Django and one CouchDB "database" actually slightly reduced the total lines of code in my application.
I know this is an old question, but today (2021) the cleanest alternative is to use the native JSONfield (since django 3.1)
docs: https://docs.djangoproject.com/en/3.2/ref/models/fields/#django.db.models.JSONField
you just create a field in the model called jsonfield inside the class model and voilá
Think it over, and find the commonalities of each data set... then define your model. It may require the use of subclasses or not. Foreign keys representing commonalities aren't to be avoided, but encouraged when they make sense.
Stuffing random data into a SQL table is not smart, unless it's truly non-relational data. If that's the case, define your problem and we may be able to help.
If you are using Postgres, you can use an hstore field: https://docs.djangoproject.com/en/1.10/ref/contrib/postgres/fields/#hstorefield.

Categories

Resources