Develop Reference

Store Subtitles in a Database

I'm working on a project that uses AI to recognise the speech of an audio file. The output of this AI is a huge JSON object with tons of values. I'll remove some keys, and the final structure will look as follows.
{
text: "<recognised text>",
language: "<detected language>"
segments: [
{startTimestamp: "00:00:00", endTimestamp: "00:00:10", text: "<some text>"},
{startTimestamp: "00:00:10", endTimestamp: "00:00:17", text: "<some text>"},
{startTimestamp: "00:00:17", endTimestamp: "00:00:26", text: "<some text>"},
{ ... },
{ ... }
]
}
Now, I wish to store this new trimmed object in a SQL database because I wish to be able to edit it manually. I'll create a React application to edit segments, delete segments, etc. Additionally, I want to add this feature to the React application, where the information will be saved every 5 seconds using an AJAX call.
Now, I don't understand how I should store this object in the SQL database. Initially, I thought I would store the whole object as a string in a database. Whenever some change is made to the object, I'll send a JSON object from the React application, the backend will sanitize it and then replace the old stringified object in the database with the new sanitised string object. This way updating and deletion will happen with ease but there can be issues in case of searching. But I'm wondering if there are any better approaches to do this.
Could someone guide me on this?
Tech Stack
Frontend - React
Backend - Django 3.2.15
Database - PostgreSQL
Thank you

Now, I don't understand how I should store this object in the SQL database. Initially, I thought I would store the whole object as a string in a database.
If the data has a clear structure, you should not store it as a JSON blob in a relational database. While relational databases have some support for JSON nowadays, it is still not very effective, and normally it means you can not effectively filter, aggregate, and manipulate data, nor can you check referential integrity.
You can work with two models that look like:
from django.db import models
from django.db.models import F, Q
class Subtitle(models.Model):
text = models.CharField(max_length=128)
language = models.CharField(max_length=128)
class Segment(models.Model):
startTimestamp = models.DurationField()
endTimestamp = models.DurationField()
subtitle = models.ForeignKey(
Subtitle, on_delete=models.CASCADE, related_name='segments'
)
text = models.CharField(max_length=512)
class Meta:
ordering = ('subtitle', 'startTimestamp', 'endTimestamp')
constraints = [
models.CheckConstraint(
check=Q(startTimestamp__gt=F('endTimestamp')),
name='start_before_end',
)
]
This will also guarantee that the startTimestamp is before the endTimestamp for example, that these fields store durations (and not "foo" for example).
You can convert from and to JSON with serializers [drf-doc]:
from rest_framework import serializers
class SegmentSerializer(serializers.ModelSerializer):
class Meta:
model = Segment
fields = ['startTimestamp', 'endTimestamp', 'text']
class SubtitleSerializer(serializers.ModelSerializer):
segments = SegmentSerializer(many=True)
class Meta:
model = Subtitle
fields = ['text', 'language', 'segments']

MongoEngine (python) - Extending the document class not working

Im trying to extend the base functionality of the Document class like the following:
class DocumentExtended(Document):
meta = {'allow_inheritance': True}
class User(DocumentExtended):
name = StringField()
User(name="John Smith").save()
The purpose is that I want to add some extra methods to DocumentExtended (but I've omitted those for brevity)
The problem is that the document does not get saved.
If I do
class User(Document):
name = StringField()
User(name="John Smith").save()
it does get saved so I know it should work
Is there some weird trick I need to do to be able to extend the mongoengine.Document class and be able to save the models to the database?

After 2 hours of not understanding I finally read the docs
The DocumentExtended class must set meta = {'abstract': True}
class DocumentExtended(Document):
meta = { 'abstract': True }

Django Model load choices from a json file

I am writing a Django User model which contains a mobile_country_code field. This field needs to be populated from a list of ISD codes, pre-populated in a json file. What is the best pythonic way to do the same?
My current implementation, which is working:
json_data/countries.json
[
...
{
"name": "Malaysia",
"dial_code": "+60",
"code": "MY"
},
...
]
project/app/models.py
import json, os
class User(models.Model):
with open(os.path.dirname(__file__)+'/json_data/countries.json') as f:
countries_json = json.load(f)
COUNTRIES_ISD_CODES = [(str(country["dial_code"]), str(country["name"])) for country in countries_json]
mobile_country_code = models.CharField(choices=COUNTRIES_ISD_CODES, help_text="Country ISD code loaded from JSON file")
Other Possible options listed below. Which one is better to use?
Using a model's __init__ method to create COUNTRIES_ISD_CODES
Importing a library method, like:
from library import import_countries_isd_codes
class User(models.Model):
mobile_country_code = models.CharField(choices=import_countries_isd_codes())

Try this,
Since Django only takes tuples
def jsonDjangoTupple(jsonData):
"""
Django only takes tuples (actual value, human readable name) so we need to repack the json in a dictionay of tuples
"""
dicOfTupple = dict()
for key, valueList in jsonData.items():
dicOfTupple[str(key)]=[(value,value) for value in valueList]
return dicOfTupple
json_data = jsonDjangoTupple(jsonData)
in the model pass something like
class User(models.Model):
code = models.CharField(max_length=120,choices=json_data['code'])
Works for when there's more than one value per key.

Django Doesn't Serialize id of the model

I have a simple class :
class Item(models.Model):
name = models.CharField(max_length=250)
I need to serialize it to JSON with the ID as part of the JSON, but the ID never gets serialized when called in the view:
items_json = serializers.serialize('json', item_list)
The item_list json as received by the browser contains only name, not the ID.
I need the ID for the processing on the client side which returns to the server later.
1) I'd like to avoid creating another ID if possible
2) I haven't created a Serializer Class. I am not sure where and how to do that.

I found the solution. The pk field that is sent over is the id. The id will now show up in the fields. So in javascript you should do:
$.each( $.parseJSON(data), function(key, val) {
console.log("ID: " + val.pk)
});

Google App Engine return object with reference set

Hi I am kind of trying to get the concept behind DataStore as a No-SQL database, what I am trying to fetch is a list of object wich have been "reference" by another. As this
class Person(db.Model):
name = db.StringProperty(required=True)
class Contact(db.Model):
name = db.StringProperty(required=True)
email = db.StringProperty()
trader = db.ReferenceProperty(Person)
This works fine and they get to be saved when I use person.put() without any problem. But when I try to retrieve it and encoded as json it nevers shows me the contact as a list in fact it totally ignores it.
persons_query = Person.all()
persons = persons_query.fetch(50)
data = json.encode(persons)
I would expect person to have a collection of Contact but it doesn't any ideas on how to solve this problem?
To make it clearer currently i am getting something like this:
[
{
name: "John Doe"
}
]
I would like to be
[
{
name: "John Doe"
contacts: [{name:"Alex", email:'alex#gmail.com'}]
}
]
Edit
Thanks all you were right I needed to fetch the collection of contacts there was only one issue for this is that when Contact was being encoded it recursively tried to encode the Trader object and this it's contact and so on.
So I got an obvious error recursive error, the solution to this was clearly to remove the trader object from the Contact when it's being encoded.

Make a custom toJson function in your class
class Person(db.Model):
name = db.StringProperty(required=True)
def toJson(self):
contact = self.contact_set #this is the default collection name for your class
d = {"name":self.name,"contact":contact}
return json.dumps(d)
class Contact(db.Model):
name = db.StringProperty(required=True)
email = db.StringProperty()
trader = db.ReferenceProperty(Person)
then you may do the ff:
persons_query = Person.all()
persons = persons_query.fetch(50)
data = person.toJson()

To fetch all the contacts you will need to write a custom json encoder, which fetches all of the reverse of the reference property.
ReferenceProperties automatically get a reverse query. From the docs "collection_name is the name of the property to give to the referenced model class. The value of the property is a Query for all entities that reference the entity. If no collection_name is set, then modelname_set (with the name of the referenced model in lowercase letters and _set added) is used."
So you would add a method to resolve the reverse reference set query.
class Person(db.Model):
name = db.StringProperty(required=True)
def contacts(self):
return self.contact_set.fetch(50) # should be smarter than that
Then use it in your custom json encoder.

If you want to find all the contacts that include a person you will need to issue a query for it.
contacts = Contact.all().filter("trader =", person)