Currently I am storing a series of objects as a dictionary of dictionaries, and within this I store calls to functions defined outside of the dictionary. These functions are specific to the objects, and cannot be generalised. In the dictionary I can refer to the function directly eg: 'some_property': function_name, and when I call that later on dictionary['some_property'](arg_1, arg_2)and the function gets called. I am looking migrate this dictionary of dictionaries to a django model, but I cannot see how I can replicate this functionality from a model.
What I currently have:
dictionaries.py
def year_camel_month(filename, **kwargs):
month = kwargs['month'].title()
return filename.format(str(kwargs['year']), month)
def year_month(filename, **kwargs):
month = kwargs['month']
return filename.format(str(kwargs['year']), month.lower())
data_source_families = {
'dataset_1': {
'source_url': 'https://example.org/url/subfolder',
'slug': 'slug_that_changes_predictably_over_time{}-{}',
'slug_treatment': year_camel_month
},
'dataset_2': {
'source_url': 'https://example2.org/url/subfolder',
'slug': 'slug_that_changes_predictably_over_time{}-{}',
'slug_treatment': year_month
},
}
Which then gets called when combined with a user-defined time frame later on:
get_data.py
from .dictionaries import data_source_families
slug = data_source_families[selected_dataset]['slug']
processed_slug = data_source_families[selected_dataset]['slug_treatment'](slug, some_kwargs)
url = data_source_families[selected_dataset]['source_url'] + processed_slug
And this is working fine. I am looking to develop functionality to improve consistency (and make these data available to another programme) by creating a django model that replicates this, something like this:
models.py
def year_camel_month(filename, **kwargs):
month = kwargs['month'].title()
return filename.format(str(kwargs['year']), month)
def year_month(filename, **kwargs):
month = kwargs['month']
return filename.format(str(kwargs['year']), month.lower())
class DataSourceFamilies(models.Model):
name = models.CharField(max_length=200, unique=True)
source_url = models.CharField(max_length=300, blank=False)
slug = models.CharField(max_length=200, blank=False)
--> slug_treatment = models._____(choices=list_of_functions) <--
def __str___(self):
return self.name.name
Does something like this exist? How would I go about doing this?
You cannot store functions (Python functions I mean) in a SQL database, indeed. But you can store any text value, and you can have a dict of 'key:func' in your model, ie:
class DataSourceFamilies(models.Model):
name = models.CharField(max_length=200, unique=True)
source_url = models.CharField(max_length=300, blank=False)
slug = models.CharField(max_length=200, blank=False)
SLUG_TREATEMENTS = [
# key, label, function
('year_camel_month', "Year, Camel month", year_camel_month),
('year_month': "Year month", year_month),
]
SLUG_TREATEMENTS_ACTIONS = {
k: func
for k, label, func in SLUG_TREATEMENTS
}
SLUG_TREATEMENTS_CHOICES = [
(k, label)
for k, label, func in SLUG_TREATEMENTS
]
slug_treatment = models.CharField(
max_length=50 # let's have a little headroom,
choices=SLUG_TREATMENT_CHOICES
)
def get_slug_treatment_func(self):
return self.SLUG_TREATEMENTS_ACTIONS[self.slug_treatment]
One thing you could do is to use a CharField and then eval it. However, using eval is usually a huge security risk. Any python code that enters it will be executed, and you do not want anything like that in a web application.
Another option is to have a lookup system. You could, say, have a CharField with choices that corresponds to a dictionary like so:
models.py
...
slug_treatment = models.CharField(max_length=100, choices=function_choices)
...
And then:
get_data.py
function_lookup = {
"year_month": year_month,
"year_camel_month": year_camel_month
}
processed_slug = function_lookup[data_source.slug_treatment](slug, some_kwargs)
Sry its a bit confusing to me, but based on what i understood, maybe you can declare all function in your class and use slug_treament as parameter to which function will be called when you need.
Lets draw it a bit
YEAR_CAMEL_MONTH=1
YEAR_MONTH=2
SLUG_TREATEMENTS_CHOICES = [
(YEAR_CAMEL_MONTH: 'year_camel_month'),
(YEAR_MONTH: 'year_month'),
]
class DataSourceFamilies(models.Model):
...
slug = models.CharField(max_length=200, blank=False)
slug_treatment = models.IntegerField(choices=SLUG_TREATMENT_CHOICES)
def year_camel_month(self):
... # Your logic
return formated_slug
def year_month(self):
... # Your logic
return formated_slug
def save(self *args **kwargs):
if self.slug_treatment == YEAR_CAMEL_MONTH:
self.slug = self.year_camel_month()
elif self.slug_treatment == YEAR_MONTH:
self.slug = self.year_month()
super(DataSourceFamilies, self).save(*args, **kwargs)
Or you can use it as prorperty method instead of persisted data (so the slug will be evaluated everytime your call your queryset, so its is dinamic instead of persisted" Obs.: Property methods works like columns from your database, but its not persisted, its like CAST in database
class DataSourceFamilies(models.Model):
...
slug_treatment = models.IntegerField(choices=SLUG_TREATMENT_CHOICES)
#property
def slug(self):
if self.slug_treatment == YEAR_CAMEL_MONTH:
return slug = self.year_camel_month()
elif self.slug_treatment == YEAR_MONTH:
return slug = self.year_month()
https://docs.djangoproject.com/en/2.0/topics/db/models/
Obs.: If you trying to get code from text and evaluate it in python i guess its possible, but is highly unsafe, and i do not recommend it
Related
I want to serialize a python object, after saved it into mysql(based on Django ORM) I want to get it and pass this object to a function which need this kind of object as a param.
Following two parts are my main logic code:
1 save param part :
class Param(object):
def __init__(self, name=None, targeting=None, start_time=None, end_time=None):
self.name = name
self.targeting = targeting
self.start_time = start_time
self.end_time = end_time
#...
param = Param()
param.name = "name1"
param.targeting= "targeting1"
task_param = {
"task_id":task_id, # string
"user_name":user_name, # string
"param":param, # Param object
"save_param":save_param_dict, # dictionary
"access_token":access_token, # string
"account_id": account_id, # string
"page_id": page_id, # string
"task_name":"sync_create_ad" # string
}
class SyncTaskList(models.Model):
task_id = models.CharField(max_length=128, blank=True, null=True)
ad_name = models.CharField(max_length=128, blank=True, null=True)
user_name = models.CharField(max_length=128, blank=True, null=True)
task_status = models.SmallIntegerField(blank=True, null=True)
task_fail_reason = models.CharField(max_length=255, blank=True, null=True)
task_name = models.CharField(max_length=128, blank=True, null=True)
start_time = models.DateTimeField()
end_time = models.DateTimeField(blank=True, null=True)
task_param = models.TextField(blank=True, null=True)
class Meta:
managed = False
db_table = 'sync_task_list'
SyncTaskList(
task_id=task_id,
ad_name=param.name,
user_name=user_name,
task_status=0,
task_param = task_param,
).save()
2 use param part
def add_param(param, access_token):
pass
task_list = SyncTaskList.objects.filter(task_status=0)
for task in task_list:
task_param = json.loads(task.task_param)
add_param(task_param["param"], task_param["access_token"]) # pass param object to function add_param
If I directly use Django ORM to save task_param into mysql, I get error,
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
for after ORM operation, I get string who's property name enclosed in single quotes like :
# in mysql it saved as
task_param: " task_param: {'task_id': 'e4b8b240cefaf58fa9fa5a591221c90a',
'user_name': 'jimmy',
'param': Param(name='name1',
targeting='geo_locations',
),
'save_param': {}}"
I am now confused with serializing an python object, then how to load this original object and pass it to a function?
Any commentary is very welcome. great thanks.
update my solution so far
task_param = {
# ...
"param":vars(param), # turn Param object to dictionary
# ...
}
SyncTaskList(
#...
task_param = json.dumps(task_param),
#...
).save()
#task_list = SyncTaskList.objects.filter(task_status=0)
#for task in task_list:
task_param = json.loads(task.task_param)
add_param(Param(**task_param["param"]), task_param["access_token"])
update based on #AJS's answer
directly pickle dumps and saved it as an binary field, then pickle loadsit also works
Any better solution for this?
Try looking into msgpack
https://msgpack.org/index.html
unlike pickle, which is python-specific, msgpack is supported by many languages (so the language you use to write to mysql can be different than the language used to read).
There are also some projects out there that integrate these serializer-libraries into Django model fields:
Pickle: https://pypi.org/project/django-picklefield/
MsgPack: https://github.com/vakorol/django-msgpackfield/blob/master/msgpackfield/msgpackfield.py
You can use pickle basically you are serializing your python object and save it as bytes in your MySQL db using BinaryField as your model field type in Django, as i don't think JSON serialization would work in your case as you have a python object as a value as well in your dict, when you fetch your data from db simpily unpickle it syntax is similar to json library see below.
import pickle
#to pickle
data = pickle.dumps({'name':'testname'})
# to unpickle just do
pickle.loads(data)
so in your case when you unpickle your object you should get your data in same form as it was before you did pickle.
Hope this helps.
I'm building a simple recipe storage application that uses the Graphene package for GraphQL. I've been able to use Django Forms so far very easily in my mutations, however one of my models fields is really an Enum and I'd like to expose it in Graphene/GraphQL as such.
My enum:
class Unit(Enum):
# Volume
TEASPOON = "teaspoon"
TABLESPOON = "tablespoon"
FLUID_OUNCE = "fl oz"
CUP = "cup"
US_PINT = "us pint"
IMPERIAL_PINT = "imperial pint"
US_QUART = "us quart"
IMPERIAL_QUART = "imperial quart"
US_GALLON = "us gallon"
IMPERIAL_GALLON = "imperial gallon"
MILLILITER = "milliliter"
LITER = "liter"
# Mass and Weight
POUND = "pound"
OUNCE = "ounce"
MILLIGRAM = "milligram"
GRAM = "gram"
KILOGRAM = "kilogram"
My Model:
class RecipeIngredient(TimeStampedModel):
recipe = models.ForeignKey(Recipe, on_delete=models.CASCADE, related_name='ingredients')
direction = models.ForeignKey(RecipeDirection, on_delete=models.CASCADE, null=True, related_name='ingredients')
quantity = models.DecimalField(decimal_places=2, max_digits=10)
unit = models.TextField(choices=Unit.as_tuple_list())
My form:
class RecipeIngredientForm(forms.ModelForm):
class Meta:
model = RecipeIngredient
fields = (
'recipe',
'direction',
'quantity',
'unit',
)
My Mutation:
class CreateRecipeIngredientMutation(DjangoModelFormMutation):
class Meta:
form_class = RecipeIngredientForm
exclude_fields = ('id',)
I've created this graphene enum UnitEnum = Enum.from_enum(Unit) however I haven't been able to get graphene to pick it up. I've tried adding it to the CreateRecipeIngredientMutation as a regular field like unit = UnitEnum() as well as an Input class on that mutation. So far, the closest I've gotten is this Github issue from awhile ago. After playing around with the class in an iPython shell, I think I could just do CreateRecipeIngredientMutation.Input.unit.type.of_type = UnitEnum() but this feels awful.
I came up with a solution that works but is not pretty. I used the https://github.com/hzdg/django-enumfields package to help with this.
I created my own form field:
class EnumChoiceField(enumfields.forms.EnumChoiceField):
def __init__(self, enum, *, coerce=lambda val: val, empty_value='', **kwargs):
if isinstance(enum, six.string_types):
self.enum = import_string(enum)
else:
self.enum = enum
super().__init__(coerce=coerce, empty_value=empty_value, **kwargs)
And used it in my Django form. Then in my custom AppConfig I did this:
class CoreAppConfig(AppConfig):
name = 'myapp.core'
def ready(self):
registry = get_global_registry()
#convert_form_field.register(EnumChoiceField)
def convert_form_field_to_enum(field: EnumChoiceField):
converted = registry.get_converted_field(field.enum)
if converted is None:
raise ImproperlyConfigured("Enum %r is not registered." % field.enum)
return converted(description=field.help_text, required=field.required)
And finally in my schema:
UnitEnum = Enum.from_enum(Unit)
get_global_registry().register_converted_field(Unit, UnitEnum)
I really don't like this, but couldn't think of a better way to handle this. I came across this idea when searching down another graphene django issue here https://github.com/graphql-python/graphene-django/issues/481#issuecomment-412227036.
I feel like there has to be a better way to do this.
In my model I have two fields for the title, one for language. I want name the post with the title in the user language if there is, else in the other language.
My model.py:
class Post(models.Model):
title_it = Model.CharField(_('title'), max_length=64, blank=True)
title_en = Model.CharField(_('title'), max_length=64, blank=True)
def __str__(self):
name_traslated={'title_it': self.title_it, 'title_en': self.title_en}
name_verbose=_('title_it')
name=name_traslated[name_verbose]
if name=='':
name=name_traslated['title_it']
if name=='':
name=name_traslated['title_en']
if name=='':
name=ugettext('No Title')
There is some faster way to do so? For example what about this:
def __str__(self):
name_traslated={'title_it': self.title_it, 'title_en': self.title_en}
name_verbose=_('title_it')
name=self.title_it
if name_traslated[name_verbose]: #!=''
name=name_traslated[name_verbose]
elif name==''
name=ugettext('No Title')
return name
You can use the python version of a ternary operation to shorten the clauses.
class Post:
title_it = 'ciao mondo'
title_en = 'hello world'
def __str__(self):
name_translated = {'title_it': self.title_it, 'title_en': self.title_en}
name_verbose = 'title_it'
out = name_translated[name_verbose] if name_translated[name_verbose] else name_translated['title_en']
return out if out else 'No Title'
Depending on your frequency of use and desired extension to more languages, you might want to maintain a preference list. In your three-way case above, I'd generalize this with:
pref_list = [
name,
name_translated[user_language],
name_translated[default_language]
]
Now, you simply pull out the first one that has text within it. You can do this by checking string contents or length. The entire operation can be wrapped up in a single expression, if you like.
I have a simple search in my Django project. I want to search through documents using their type and part of factory info in addition to search by name.
Here is my models.py:
class Docs(models.Model):
Date = models.DateField(default=date.today)
Name = models.CharField(max_length=50)
Type = models.ForeignKey(DocTypes)
Part = models.ForeignKey(Parts)
Link = models.FileField(upload_to='Docs/%Y/%m/%d')
class Parts(models.Model):
Name = models.CharField(max_length=50)
def __str__(self):
return str(self.Name)
class DocTypes(models.Model):
Type = models.CharField(max_length=50)
def __str__(self):
return str(self.Type)
My forms.py:
class DocsSearchForm(ModelForm):
class Meta:
model = Docs
fields = [ 'Name', 'Type', 'Part']
And this is part of my views.py, if no search was done then all documents are given
def showdocs(request):
if request.method == 'POST':
form = DocsSearchForm(request.POST)
documents = Docs.objects.filter(Name__contains=request.POST['Name']|
Type==request.POST['Type']|
Part==request.POST['Part'])
else:
form = DocsSearchForm()
documents = Docs.objects.all()
return render(
request,
'showdocs.html',
{'documents': documents, 'form':form}
So, the problem is the following: if I try to use a search then I have
NameError at /showdocs
name 'Type' is not defined.
POST values are:Part '1', Name 'Example', Type '1'.
If I delete
Type==request.POST['Type']|
Part==request.POST['Part']
then search by name works well. So I have a guess that problem is about searching by foreign key values, but have no ideas more. Will appreciate any help.
Try replacing the line with this
Docs.objects.filter(Name__contains=request.POST['Name'],
Type=request.POST['Type'],
Part=request.POST['Part']
)
It seems you have misunderstood the syntax. I don't know why you are trying to use | operator here.
That's not how Django filters work. You can't | them because they are not actually expressions, just keyword arguments. In this case, correct syntax would be:
Docs.objects.filter(
Name__contains=request.POST['Name'],
Type_Type=request.POST['Type'],
Part_Name=request.POST['Part'],
)`
I have two models :
class Category(models.Model):
name = models.CharField(max_length=50)
and
class SubCategory(models.Model):
sex = models.CharField(choices=SEX, blank=True, max_length=5)
name = models.TextField(blank=True)
category = models.ForeignKey(Category)
def __unicode__(self):
return u'%s' % (self.name)
I'm creating an api using tastypie that returns me "SubCategory" objects in a JSON. I want to add a custom field "start_counter_of_category" in every result set which contains the counter of the subcategory where the category id changed (when ordered on "category_id" field)
The algorithm is fairly straightforward, something like this in the "dehydrate" function:
API_SKU_VARS = {
'COUNTER' : 1,
'FIRST_ELEMENT' : 1,
'PREV_CATEGORY' : 1
}
class SubCategoryResource(ModelResource):
start_counter_of_category = fields.IntegerField(readonly=True)
category = fields.ForeignKey(CategoryResource,'category')
class Meta:
queryset = SubCategory.objects.all()
resource_name = 'subcategory'
filtering = {
'id' : ALL,
'name' : ALL,
'category': ALL_WITH_RELATIONS,
}
ordering = ['id','name','category']
serializer = Serializer(formats=['json'])
def dehydrate(self, bundle):
if API_SKU_VARS['PREV_CATEGORY'] != bundle.data['category']: #if the category of the current bundle is not equal to the category of the previous bundle, we update the ['PREV_CATEGORY']
API_SKU_VARS['FIRST_ELEMENT']=API_SKU_VARS['COUNTER'] #update the ['FIRST_ELEMENT'] with the counter of the current bundle
API_SKU_VARS['PREV_CATEGORY'] = bundle.data['category']
API_SKU_VARS['COUNTER'] = API_SKU_VARS['COUNTER']+1 #for every bundle passed, we update the counter
bundle.data['start_counter_of_category']=API_SKU_VARS['FIRST_ELEMENT']
return bundle.data
serializer = Serializer(formats=['json'])
It works perfectly for the first run after I start the server. Predictably the issue of course is that the second time I make the api call, the variables retain the values they had in the previous run.
Any ideas how to re-initiate the variables every time the api call is made?
SOLUTION:
re-initialte the variables in
build_filters if the api called is a filtering API
get_detail if the api called is a detail API
example (in my case):
def build_filters(self, filters=None):
if filters is None:
filters = {}
orm_filters = super(SubCategoryResource, self).build_filters(filters) #get the required response using the function's behavior from the super class
self.API_SKU_VARS = {
'PREV_CATEGORY':1,
'COUNTER':1,
'FIRST_ELEMENT':1,
}
return orm_filters
(These functions are over-ridden if you want to apply any custom logic into the API response)
BETTER AND MOST OBVIOUS SOLUTION
re-instantiate the variables in the init function, something like this:
def __init__(self,api_name=None):
self.API_SKU_VARS = {.....}
super(SKUResource,self).__init__(api_name)
Yes, you can reinitialize (not "reinitiate") the variables just by running this code at the start of each call:
API_SKU_VARS['COUNTER'] = 1
API_SKU_VARS['PREV_CATEGORY'] = 1
API_SKU_VARS['FIRST_ELEMENT'] = 1
But this is a bad idea. Why is this variable global in the first place? The whole point of a global variable is that it's shared by all objects in the module, and lives for the lifetime of the module. If you want something that's local to a single API call and lives for the lifetime of that call, make it a member of something that has those characteristics. Initialize the member as part of initializing the appropriate object. Then there's no need to reinitialize it.
To see why this is a bad idea: What happens if two clients connect up concurrently and both make the same API call?