Is there any plugin or 3rd party backend to manage redis connections in Django, so the methods in view.py don't have to explicitly connect to redis for every request?
If not, how would you start implementing one? A new plugin? a new backend? a new django middleware?
Thank you.
I think the emerging standard for non-rel databases is django-nonrel . I don't know if django-nonrel is production ready or if support redis, but they have a guide on writing a custom no-sql backend.
Unfortunately, i don't think that writing support for a redis on standard django is easy as writing a DatabaseBackend. There's a lot in django models mechanics and workflow that simply assumes an ACID database. What about syncdb ? And about Querysets?
However, you may try to write a poor-mans approach using models.Manager and a lot of tweaking on your model. For example:
# helper
def fill_model_instance(instance, values):
""" Fills an model instance with the values from dict values """
attributes = filter(lambda x: not x.startswith('_'), instance.__dict__.keys())
for a in attributes:
try:
setattr(instance, a, values[a.upper()])
del values[a.upper()]
except:
pass
for v in values.keys():
setattr(instance, v, values[v])
return instance
class AuthorManager( models.Manager ):
# You may try to use the default methods.
# But should be freaking hard...
def get_query_set(self):
raise NotImplementedError("Maybe you can write a Non relational Queryset()! ")
def latest(self, *args, **kwargs):
# redis Latest query
pass
def filter(self, *args, **kwargs):
# redis filter query
pass
# Custom methods that you may use, instead of rewriting
# the defaults ones.
def open_connection(self):
# Open a redis connection
pass
def search_author( self, *args, **kwargs ):
self.open_connection()
# Write your query. I don't know how this shiny non-sql works.
# Assumes it returns a dict for every matched author.
authors_list = [{'name': 'Leibniz', 'email': 'iinventedcalculus#gmail.com'},
'name': 'Kurt Godel','email': 'self.consistent.error#gmail.com'}]
return [fill_instance(Author(), author) for author in authors_list]
class Author( models.Model ):
name = models.CharField( max_length = 255 )
email = models.EmailField( max_length = 255 )
def save(self):
raise NotImplementedError("TODO: write a redis save")
def delete(self):
raise NotImplementedError(""TODO: write a delete save")
class Meta:
managed = False
Please not that i've only made an sketch of how you can tweak the django models. I have not
tested and run this code. I first suggest you to investigate django-nonrel.
Related
Question
How can I build a Model that that stores one field in the database, and then retrieves other fields from an API behind-the-scenes when necessary?
Details:
I'm trying to build a Model called Interviewer that stores an ID in the database, and then retrieves name from an external API. I want to avoid storing a copy of name in my app's database. I also want the fields to be retrieved in bulk rather than per model instance because these will be displayed in a paginated list.
My first attempt was to create a custom Model Manager called InterviewManager that overrides get_queryset() in order to set name on the results like so:
class InterviewerManager(models.Manager):
def get_queryset(self):
query_set = super().get_queryset()
for result in query_set:
result.name = 'Mary'
return query_set
class Interviewer(models.Model):
# ID provided by API, stored in database
id = models.IntegerField(primary_key=True, null=False)
# Fields provided by API, not in database
name = 'UNSET'
# Custom model manager
interviewers = InterviewerManager()
However, it seems like the hardcoded value of Mary is only present if the QuerySet is not chained with subsequent calls. I'm not sure why. For example, in the django shell:
>>> list(Interviewer.interviewers.all())[0].name
'Mary' # Good :)
>>> Interviewer.interviewers.all().filter(id=1).first().name
'UNSET' # Bad :(
My current workaround is to build a cache layer inside of InterviewManager that the model accesses like so:
class InterviewerManager(models.Manager):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.api_cache = {}
def get_queryset(self):
query_set = super().get_queryset()
for result in query_set:
# Mock querying a remote API
self.api_cache[result.id] = {
'name': 'Mary',
}
return query_set
class Interviewer(models.Model):
# ID provided by API, stored in database
id = models.IntegerField(primary_key=True, null=False)
# Custom model
interviewers = InterviewerManager()
# Fields provided by API, not in database
#property
def name(self):
return Interviewer.interviewers.api_cache[self.id]['name']
However this doesn't feel like idiomatic Django. Is there a better solution for this situation?
Thanks
why not just make the API call in the name property?
#property
def name(self):
name = get_name_from_api(self.id)
return name
If that isnt possible by manipulating a get request where you can add a list of names and recieve the data. The easy way is to do it is in a loop.
I would recommand you to build a so called proxy where you load the articles in a dataframe/dict, save this varible data ( with for example pickle ) and use it when nessary. It reduces loadtimes and is near efficient.
My flask app centers around modifying models based on SQLAlchemy. Hence, I find flask-admin a great plugin because it maps my SQLA models to forms with views already defined with a customizable interface that is tried and tested.
I understand that Flask-admin is intended to be a plugin for administrators managing their site's data. However, I don't see why I can't use FA as a framework for my users to CRUD their data as well.
To do this, I have written the following:
class AuthorizationRequiredView(BaseView):
def get_roles(self):
raise NotImplemented("Override AuthorizationRequiredView.get_roles not set.")
def is_accessible(self):
if not is_authenticated():
return False
if not current_user.has_role(*self.get_roles()):
return False
return True
def inaccessible_callback(self, name, **kwargs):
if not is_authenticated():
return current_app.user_manager.unauthenticated_view_function()
if not current_user.has_role(*self.get_roles()):
return current_app.user_manager.unauthorized_view_function()
class InstructionModelView(DefaultModelView, AuthorizationRequiredView):
def get_roles(self):
return ["admin", "member"]
def get_query(self):
"""Jails the user to only see their instructions.
"""
base = super(InstructionModelView, self).get_query()
if current_user.has_role('admin'):
return base
else:
return base.filter(Instruction.user_id == current_user.id)
#expose('/edit/', methods=('GET', 'POST'))
def edit_view(self):
if not current_user.has_role('admin'):
instruction_id = request.args.get('id', None)
if instruction_id:
m = self.get_one(instruction_id)
if m.user_id != current_user.id:
return current_app.user_manager.unauthorized_view_function()
return super(InstructionModelView, self).edit_view()
#expose('/delete/', methods=('POST',))
def delete_view(self):
return_url = get_redirect_target() or self.get_url('.index_view')
if not self.can_delete:
return redirect(return_url)
form = self.delete_form()
if self.validate_form(form):
# id is InputRequired()
id = form.id.data
model = self.get_one(id)
if model is None:
flash(gettext('Record does not exist.'), 'error')
return redirect(return_url)
# message is flashed from within delete_model if it fails
if self.delete_model(model):
if not current_user.has_role('admin') \
and model.user_id != current_user.id:
# Denial: NOT admin AND NOT user_id match
return current_app.user_manager.unauthorized_view_function()
flash(gettext('Record was successfully deleted.'), 'success')
return redirect(return_url)
else:
flash_errors(form, message='Failed to delete record. %(error)s')
return redirect(return_url)
Note: I am using Flask-User which is built on top of Flask-Login.
The code above works. However, it is difficult to abstract as a base class for other models which I would like to implement access control for CRUD operations and Index/Edit/Detail/Delete views.
Mainly, the problems are:
the API method, is_accessible, does not provide the primary key of the model. This key is needed because in almost all cases relationships between users and entities are almost always stored via relationships or in the model table directly (i.e. having user_id in your model table).
some views, such as delete_view, do not provide the instance id that can be retrieve easily. In delete_view, I had to copy the entire function just to add one extra line to check if it belongs to the right user.
Surely someone has thought about these problems.
How should I go about rewriting this to something that is more DRY and maintainable?
I have the application in Django REST as backend and Angular as frontend.
Suppose in This is my code
class ModelClass (models.Model):
name = models.CharField(max_length=100)
email = models.EmailField()
def save(self, *args, **kwargs):
#check if the row with this hash already exists.
if not self.pk:
self.hash = self.create_hash()
self.my_stuff = 'something I want to save in that field'
# call to some async task
super(ModelClass, self).save(*args, **kwargs)
In my REST i have this view
class ModelListCreateView(generics.ListCreateAPIView):
model = ModelClass
serializer_class = ModelClassSerializer
def pre_save(self, obj):
obj.created_by = obj.updated_by = self.request.user.staff
def post_save(self, obj, created=False):
# add some other child objects of other model
I don't want to do unit testing. I want to do system testing so that I need to know if I post something to that view then
Pre-save thing should work
Record gets created
Save method of Model gets called with his stuff
After save method in REST gets called
Then I can assert all that stuff.
Can I test all that . I want to know which thing I need to have that sort of test rather than small unit tests
I am confused do I need to use Django test or REST Test or selenium test or PyTEst or factory boy because i want to know if things are actually getting in database
What you are looking is some kind of a REST Client code that would then be able to run your tests and you would be able to verify if the call is successful or not. Django Rest Framework has the APIRestFactory helper class that will aid you in writing such tests. The documentation can be found here and specifically look at the Example section. This would be part of your Django tests
In developing a website for indexing system documentation I've come across a tough nut to crack regarding data "matching"/relations across databases in Django.
A simplified model for my local database:
from django.db import models
class Document(models.Model):
name = models.CharField(max_length=200)
system_id = models.IntegerField()
...
Imagined model, system details are stored in a remote database.
from django.db import models
class System(models.Model):
name = models.CharField(max_length=200)
system_id = models.IntegerField()
...
The idea is that when creating a new Document entry at my website the ID of the related system is to be stored in the local database. When presenting the data I would have to use the stored ID to retrieve the system name among other details from the remote database.
I've looked into foreign keys across databases, but this seems to be very extensive and I'm not sure if I want relations. Rather I visualize a function inside the Document model/class which is able to retrieve the matching data, for example by importing a custom router/function.
How would I go about solving this?
Note that I won't be able to alter anything on the remote database, and it's read-only. Not sure if I should create a model for System aswell. Both databases use PostgreSQL, however my impression is that it's not really of relevance to this scenario which database is used.
In the django documentation multi-db (manually-selecting-a-database)
# This will run on the 'default' database.
Author.objects.all()
# So will this.
Author.objects.using('default').all()
# This will run on the 'other' database.
Author.objects.using('other').all()
The 'default' and 'other' are aliases for you databases.
In your case it would could be 'default' and 'remote'.
of course you could replace the .all() with anything you want.
Example: System.objects.using('remote').get(id=123456)
You are correct that foreign keys across databases are a problem in Django ORM, and to some extent at the db level too.
You already have the answer basically: "I visualize a function inside the Document model/class which is able to retrieve the matching data"
I'd do it like this:
class RemoteObject(object):
def __init__(self, remote_model, remote_db, field_name):
# assumes remote db is defined in Django settings and has an
# associated Django model definition:
self.remote_model = remote_model
self.remote_db = remote_db
# name of id field on model (real db field):
self.field_name = field_name
# we will cache the retrieved remote model on the instance
# the same way that Django does with foreign key fields:
self.cache_name = '_{}_cache'.format(field_name)
def __get__(self, instance, cls):
try:
rel_obj = getattr(instance, self.cache_name)
except AttributeError:
system_id = getattr(instance, self.field_name)
remote_qs = self.remote_model.objects.using(self.remote_db)
try:
rel_obj = remote_qs.get(id=system_id)
except self.remote_model.DoesNotExist:
rel_obj = None
setattr(instance, self.cache_name, rel_obj)
if rel_obj is None:
raise self.related.model.DoesNotExist
else:
return rel_obj
def __set__(self, instance, value):
setattr(instance, self.field_name, value.id)
setattr(instance, self.cache_name, value)
class Document(models.Model:
name = models.CharField(max_length=200)
system_id = models.IntegerField()
system = RemoteObject(System, 'system_db_name', 'system_id')
You may recognise that the RemoteObject class above implements Python's descriptor protocol, see here for more info:
https://docs.python.org/2/howto/descriptor.html
Example usage:
>>> doc = Document.objects.get(pk=1)
>>> print doc.system_id
3
>>> print doc.system.id
3
>>> print doc.system.name
'my system'
>>> other_system = System.objects.using('system_db_name').get(pk=5)
>>> doc.system = other_system
>>> print doc.system_id
5
Going further you could write a custom db router:
https://docs.djangoproject.com/en/dev/topics/db/multi-db/#using-routers
This would let you eliminate the using('system_db_name') calls in the code by routing all reads for System model to the appropriate db.
I'd go for a method get_system(). So:
class Document:
def get_system(self):
return System.objects.using('remote').get(system_id=self.system_id)
This is the simplest solution. A possible solution is also to use PostgreSQL's foreign data wrapper feature. By using FDW you can abstract away the multidb handling from django and do it inside the database - now you can use queries that need to use the document -> system relation.
Finally, if your use case allows it, just copying the system data periodically to the local db can be a good solution.
I have basically a username is unique (case insensitive), but the case matters when displaying as provided by the user.
I have the following requirements:
field is CharField compatible
field is unique, but case insensitive
field needs to be searchable ignoring case (avoid using iexact, easily forgotten)
field is stored with case intact
preferably enforced on database level
preferably avoid storing an extra field
Is this possible in Django?
The only solution I came up with is "somehow" override the Model manager, use an extra field, or always use 'iexact' in searches.
I'm on Django 1.3 and PostgreSQL 8.4.2.
As of Django 1.11, you can use CITextField, a Postgres-specific Field for case-insensitive text backed by the citext type.
from django.db import models
from django.contrib.postgres.fields import CITextField
class Something(models.Model):
foo = CITextField()
Django also provides CIEmailField and CICharField, which are case-insensitive versions of EmailField and CharField.
Store the original mixed-case string in a plain text column. Use the data type text or varchar without length modifier rather than varchar(n). They are essentially the same, but with varchar(n) you have to set an arbitrary length limit, that can be a pain if you want to change later. Read more about that in the manual or in this related answer by Peter Eisentraut #serverfault.SE.
Create a functional unique index on lower(string). That's the major point here:
CREATE UNIQUE INDEX my_idx ON mytbl(lower(name));
If you try to INSERT a mixed case name that's already there in lower case you get a unique key violation error.
For fast equality searches use a query like this:
SELECT * FROM mytbl WHERE lower(name) = 'foo' --'foo' is lower case, of course.
Use the same expression you have in the index (so the query planner recognizes the compatibility) and this will be very fast.
As an aside: you may want to upgrade to a more recent version of PostgreSQL. There have been lots of important fixes since 8.4.2. More on the official Postgres versioning site.
With overriding the model manager, you have two options. First is to just create a new lookup method:
class MyModelManager(models.Manager):
def get_by_username(self, username):
return self.get(username__iexact=username)
class MyModel(models.Model):
...
objects = MyModelManager()
Then, you use get_by_username('blah') instead of get(username='blah'), and you don't have to worry about forgetting iexact. Of course that then requires that you remember to use get_by_username.
The second option is much hackier and convoluted. I'm hesitant to even suggest it, but for completeness sake, I will: override filter and get such that if you forget iexact when querying by username, it will add it for you.
class MyModelManager(models.Manager):
def filter(self, **kwargs):
if 'username' in kwargs:
kwargs['username__iexact'] = kwargs['username']
del kwargs['username']
return super(MyModelManager, self).filter(**kwargs)
def get(self, **kwargs):
if 'username' in kwargs:
kwargs['username__iexact'] = kwargs['username']
del kwargs['username']
return super(MyModelManager, self).get(**kwargs)
class MyModel(models.Model):
...
objects = MyModelManager()
As of December 2021, with the help of Django 4.0 UniqueConstraint expressions you can add a Meta class to your model like this:
class Meta:
constraints = [
models.UniqueConstraint(
Lower('<field name>'),
name='<constraint name>'
),
]
I'm by no mean a Django professional developer and I don't know technical considerations like performance issues about this solution. Hope others comment on that.
Since a username is always lowercase, it's recommended to use a custom lowercase model field in Django. For the ease of access and code-tidiness, create a new file fields.py in your app folder.
from django.db import models
from django.utils.six import with_metaclass
# Custom lowecase CharField
class LowerCharField(with_metaclass(models.SubfieldBase, models.CharField)):
def __init__(self, *args, **kwargs):
self.is_lowercase = kwargs.pop('lowercase', False)
super(LowerCharField, self).__init__(*args, **kwargs)
def get_prep_value(self, value):
value = super(LowerCharField, self).get_prep_value(value)
if self.is_lowercase:
return value.lower()
return value
Usage in models.py
from django.db import models
from your_app_name.fields import LowerCharField
class TheUser(models.Model):
username = LowerCharField(max_length=128, lowercase=True, null=False, unique=True)
End Note : You can use this method to store lowercase values in the database, and not worry about __iexact.
You can use citext postgres type instead and not bother anymore with any sort of iexact. Just make a note in model that underlying field is case insensitive.
Much easier solution.
You can use lookup='iexact' in UniqueValidator on serializer, like this:
Unique model field in Django and case sensitivity (postgres)
I liked Chris Pratt's Answer but it didn't worked for me, because the models.Manager-class doesn't have the get(...) or filter(...) Methods.
I had to take an extra step via a custom QuerySet:
from django.contrib.auth.base_user import BaseUserManager
from django.db.models import QuerySet
class CustomUserManager(BaseUserManager):
# Use the custom QuerySet where get and filter will change 'email'
def get_queryset(self):
return UserQuerySet(self.model, using=self._db)
def create_user(self, email, password, **extra_fields):
...
def create_superuser(self, email, password, **extra_fields):
...
class UserQuerySet(QuerySet):
def filter(self, *args, **kwargs):
if 'email' in kwargs:
# Probably also have to replace...
# email_contains -> email_icontains,
# email_exact -> email_iexact,
# etc.
kwargs['email__iexact'] = kwargs['email']
del kwargs['email']
return super().filter(*args, **kwargs)
def get(self, *args, **kwargs):
if 'email' in kwargs:
kwargs['email__iexact'] = kwargs['email']
del kwargs['email']
return super().get(*args, **kwargs)
This worked for me in a very simple case but is working pretty good so far.
You can also override get_prep_value() and reuse it through inheritance.
class LowerCaseField:
def get_prep_value(self, value):
value = super().get_prep_value(value)
if value:
value = value.strip().lower()
return value
class LowerSlugField(LowerCaseField, models.SlugField):
pass
class LowerEmailField(LowerCaseField, models.EmailField):
pass
class MyModel(models.Model):
email = LowerEmailField(max_length=255, unique=True)
This way, if you ever want to reuse this field in another model, you can use the same consistent strategy.
From Django Docs:
get_prep_value(value)
value is the current value of the model’s
attribute, and the method should return data in a format that has been
prepared for use as a parameter in a query.
See Converting Python objects to query values for usage.