Develop Reference

Python + ElasticSearch: Mapper Parsing Exceptions for join field

I'm using ElasticSearch 8.3.2 to store some data I have. The data consists of metabolites and several "studies" for each metabolite, with each study in turn containing concentration values. I am also using the Python ElasticSearch client to communicate with the backend, which works fine.
To associate metabolites with studies, I was considering using a join field as described here.
I have defined this index mapping:
INDEXMAPPING_MET = {
"mappings": {
"properties": {
"id": {"type": "keyword"},
"entry_type": {"type": "text"},
"pc_relation": {
"type": "join",
"relations": {
"metabolite": "study"
}
},
"concentration": {
"type": "nested",
}
}
}
}
pc_relation is the join field here, with metabolites being the parent documents of each study document.
I can create metabolite entries (the parent documents) just fine using the Python client, for example
self.client.index(index="metabolitesv2", id=metabolite, body=json.dumps({
#[... some other fields here]
"pc_relation": {
"name": "metabolite",
},
}))
However, once I try adding child documents, I get a mapping_parser_exception. Notably, I only get this exception when trying to add the pc_relation field, any other fields work just fine and I can create documents if I omit the join field. Here is an example for a study document I am trying to create (on the same index):
self.client.index(index="metabolitesv2", id=study, body=json.dumps({
#[... some other fields here]
"pc_relation": {
"name": "study",
"parent": metabolite_id
},
}))
At first I thought there might be some typing issues, but casting everything to a string sadly does not change the outcome. I would really appreciate any help with regards to where the error could be as I am not really sure what the issue is - From what I can tell from the official ES documentation and other Python+ES projects I am not really doing anything differently.
Tried: Creating an index with a join field, creating a parent document, creating a child document with a join relation to the parent.
Expectation: Documents get created and can be queried using has_child or has_parent tags.
Result: MappingParserException when trying to create the child document

Tldr;
You need to provide a routing value at indexing time for the child document.
The routing value is mandatory because parent and child documents must be indexed on the same shard
By default the routing value of a document is its _id, so in practice you need to provide the _id of the parent document when indexing the child.
Solution
self.client.index(index="metabolitesv2", id=study, routing=metabolite, body=json.dumps({
#[... some other fields here]
"pc_relation": {
"name": "study",
"parent": metabolite_id
},
}))
To reproduce
PUT 75224800
{
"settings": {
"number_of_shards": 4
},
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"pc_relation": {
"type": "join",
"relations": {
"metabolite": "study"
}
}
}
}
}
PUT 75224800/_doc/1
{
"id": "1",
"pc_relation": "metabolite"
}
# No routing Id this is going to fail
PUT 75224800/_doc/2
{
"id": 2,
"pc_relation":{
"name": "study",
"parent": "1"
}
}
PUT 75224800/_doc/3
{
"id": "3",
"pc_relation": "metabolite"
}
PUT 75224800/_doc/4?routing=3
{
"id": 2,
"pc_relation":{
"name": "study",
"parent": "3"
}
}

Extracting fields from a json file

I am provided with the following json file.
{
"results": [
{
"id": "1234",
"useless_field1": null,
"useless_field2": null,
"data": [
{
"type": "type_1",
"useless_field3": null,
"volumne": "5"
}
]
}
]
}
I would like to extract only the following fields: id, type, volume. I did it in the following way.
def extract(json_response):
return [{
item['id'],
item['data'][0]['type'],
item['data'][0]['volume']
} for item in json_response['results']]
It is possible for the type to be a list. Is my solution efficient? Are there any alternatives of solving such problems?

SugarCRM response ordered dict key _hash

What is
_hash
that is received with the API request?
My request url,
url = "https://" + sugar_instance + "/rest/v10/Leads"
Is there a unique user_id for each Lead/Employee/Module in SugarCRM? And if yes, how can I obtain is using a request. I am using Python.

There are a few different questions within your question. I'll try to answer all of them.
What is _hash?
Have a look at this subset of an API response:
"modified_user_id": "e8b433d5-5d17-456c-8506-fe56452fcce8",
"modified_by_name": "Reisclef",
"modified_user_link": {
"full_name": "Administrator",
"id": "1",
"_acl": {
"fields": [],
"delete": "no",
"_hash": "8e11bf9be8f04daddee9d08d44ea891e"
}
},
"created_by": "1",
"created_by_name": "Administrator",
"created_by_link": {
"full_name": "Administrator",
"id": "1",
"_acl": {
"fields": [],
"delete": "no",
"_hash": "8e11bf9be8f04daddee9d08d44ea891e"
}
},
The "_hash" in the above response is a hash of the related acl record, representing the user's access control limits to the record in question.
We can prove this by looking further down my response. You will notice that the hash changes, but is consistent with each object with the same criteria:
"member_of": {
"name": "",
"id": "",
"_acl": {
"fields": [],
"_hash": "654d337e0e912edaa00dbb0fb3dc3c17"
}
},
"campaign_id": "",
"campaign_name": "",
"campaign_accounts": {
"name": "",
"id": "",
"_acl": {
"fields": [],
"_hash": "654d337e0e912edaa00dbb0fb3dc3c17"
}
},
What we can gather from this is that the _hash is a hash of the _acl object. You can confirm this by looking at include/MetaDataManager/MetaDataManager.php, line 1035.
Therefore, it's not a hash of the user record, it's a hash of the ACL settings of the record.
Is there a unique user_id?
Strictly speaking, no, there won't be a unique user id for every record (unless one user only ever created/edited one record).
If you refer back to my first block of JSON, you'll see there are two user relationships:
modified_user_id
and
created_by
These indicate what the unique id is of the user record, which we can guarantee to be unique (as far as GUIDs are).
How can I obtain it?
It's technically already in the request, but if you just wanted to retrieve the created by user id and modified by user id, you can do the call using this:
https://{INSTANCE}/rest/v10/{MODULE}?fields=created_by,modified_user_id

Serialize complex fields in django rest framework

I have a model that has a JSON field extra_data that contains other fields that might be added to the model. From the beginning it is not known how many fields will be added apart from the compulsory ones, which is why I introduced the extra_data field.
With the usual rest framework serialization I currently have something like this:
[
{
"code": "1",
"name": "Moscow",
"extra_data": {
"type": "Region"
}
},
{
"code": "2",
"name": "Tatarstan",
"extra_data": {
"type": "Republic",
"capital": "Kazan"
}
}
]
But what I need something is like this:
[
{
"code": "1",
"name": "Moscow",
"type": "City"
},
{
"code": "2",
"name": "Tatarstan",
"type": "Republic",
"capital": "Kazan"
}
]
Please I need help, I'm new to django

The serializer itself I don't think can do this, since you do not know how many fields are there. But once you get the serializer.data, you may update your dict like:
serializer_data = serializer.data
extra_data = serializer_data.pop('extra_data')
serializer_data.update(extra_data)
return serializer_data
I'm no expert on Django so I'm not telling you for sure that there is no way of doing that withing the serializer, but none that I can think of

Where to change the form of json response in django rest framework?

Lets say I have a model:
class MyModel(models.Model):
name = models.CharField(max_length=100)
description= models.TextField()
...
Then I created ModelViewSet with HyperLinkedSerializer, so when I call my /api/mymodels endpint I get responses like this:
{
"count": 2,
"next": null,
"previous": null,
"results": [
{ "name": "somename", "description": "desc"},
{ "name": "someothername", "description": "asdasd"},
]
}
and when I call /api/mymodels/1 I get:
{ "name": "somename", "description": "asdasd"}
but what I would like to get is:
{
"metadata":{ ...},
"results": { "name": "somename", "description": "desc"}
}
And I would like to use this format for all models at my website, so I dont want to change every viewset, I want to implement it in (most likely) one class and then use it for all my viewsets.
So my question is: which renderer or serializer or other class (Im really not sure) should I alter or create to get this behavior of json response?

The first response appears to be a paginated response, which is determined by the pagination serializer. You can create a custom pagination serializer that will use a custom format. You are looking for something similar to the following:
class MetadataSerialier(pagination.BasePaginationSerializer):
count = serializers.Field(source='paginator.count')
next = NextPageField(source='*')
previous = PreviousPageField(source='*')
class CustomPaginationSerializer(pagination.BasePaginationSerializer):
metadata = MetadataSerializer(source='*')
This should give you an output similar to the following:
{
"metadata": {
"count": 2,
"next": null,
"previous": null
},
"results": [
{ "name": "somename", "description": "desc"},
{ "name": "someothername", "description": "asdasd"}
]
}
The pagination serializer can be set globally through your settings, as described in the documentation.
REST_FRAMEWORK = {
'DEFAULT_PAGINATION_SERIALIZER_CLASS': {
'full.path.to.CustomPaginationSerializer',
}
}