Develop Reference

Loading irregular json into Elasticsearch index with mapping using Python client

I have some .json where not all fields are present in all records, for e.g. caseclass.json looks like:
[{
"name" : "john smith",
"age" : 12,
"cars": ["ford", "toyota"],
"comment": "i am happy"
},
{
"name": "a. n. other",
"cars": "",
"comment": "i am panicking"
}]
Using Elasticsearch-7.6.1 via python client elasticsearch:
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
import json
import os
from elasticsearch_dsl import Document, Text, Date, Integer, analyzer
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
class Person(Document):
class Index:
using = es
name = 'person_index'
name = Text()
age = Integer()
cars = Text()
comment = Text(analyzer='snowball')
Person.init()
with open ("caseclass.json") as json_file:
data = json.load(json_file)
for indexid in range(len(data)):
document = Person(name=data[indexid]['name'], age=data[indexid]['age'], cars=data[indexid]['cars'], comment=data[indexid]['comment'])
document.meta.id = indexid
document.save()
Naturally I get KeyError: 'age' when the second record is trying to be read. My question is: it is possible to load such records onto a Elasticsearch index using the Python client and a pre-defined mapping, instead of dynamic mapping? Above code works if all fields are present in all records but is there a way to do this without checking presence of each field per record as the actual records have complex structure and there are millions of them? Thanks

The error has nothing to do w/ your mapping -- it's just telling you that age could not be accessed in one of your caseclasses.
The index mapping is created when you call Person.init() -- you can verify that by calling print(es.indices.get_mapping(Person.Index.name)) right after Person.init().
I've cleaned up your code a bit:
import json
import os
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search, Document, Text, Date, Integer, analyzer
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
class Person(Document):
class Index:
using = es
name = 'person_index'
name = Text()
age = Integer()
cars = Text()
comment = Text(analyzer='snowball')
Person.init()
print(es.indices.get_mapping(Person.Index.name))
with open("caseclass.json") as json_file:
data = json.load(json_file)
for indexid, case in enumerate(data):
document = Person(**case)
document.meta.id = indexid
document.save()
Notice how I used **case to spread all key-value pairs inside of a case instead of using data[property_key].
The generated mapping is as follows:
{
"person_index" : {
"mappings" : {
"properties" : {
"age" : {
"type" : "integer"
},
"cars" : {
"type" : "text"
},
"comment" : {
"type" : "text",
"analyzer" : "snowball"
},
"name" : {
"type" : "text"
}
}
}
}
}

MongoEngine EmbeddedDocument query with array value

I have an object that looks something like this in the database, with corresponding MongoEngine models:
{
...
"config" : {
"inner_group" : {
"individuals" : [
{
"entity_id" : "54321",
}
],
},
...
}
...
}
I am trying to query this data using the entity_id field in the object which is part of the individuals collection.
I have tried querying according to the MongoEngine docs but I have not been able to pull the data using the following:
data = Model.objects(config__inner_group__individuals__S__entity_id="54321")
How can I query the entire parent based on the entity_id?

mongoengine embedded field update

this is my schema
class Url_per_date(EmbeddedDocument):
date = DateTimeField()
count = IntField(default=0)
class Daily_visit(Document):
domain = StringField()
count = IntField(default=0)
per_date = ListField(EmbeddedDocumentField('Url_per_date'))
i have collection like this:
{
"_id" : ObjectId("51c97e685aa3b3414c7e406a"),
"_types" : "Daily_visit",
"count" : 1,
"domain" : "yahoo.com",
"per_date" : {
"count" : 1,
"date" : ISODate("2013-05-20T00:00:00Z")
}
}
i need to update yahoo.com by date range. if per_date not have ISODate("2013-05-20T00:00:00Z") i need to create it, if i have it inc__count=1.

In MongoEngine, embedded fields are referenced by replacing dot-notation with double underscores:
Fields on embedded documents may also be referred to using field lookup syntax by using a double-underscore in place of the dot in object attribute access syntax:
http://mongoengine-odm.readthedocs.org/en/v0.6.8/guide/querying.html#filtering-queries

at first your result must be like this
{
"_id" : ObjectId("51c97e685aa3b3414c7e406a"),
"_types" : "Daily_visit",
"count" : 1,
"domain" : "yahoo.com",
"per_date" : [{
"count" : 1,
"date" : ISODate("2013-05-20T00:00:00Z")
}]
}
how create date? create new one for today and select database like this
today = datetime.today()
try:
yahoo_obj = Daily_visit.objects.get(domain="yahoo.com", per_date__date=date)
yahoo_obj.per_date[-1].count += 1
except:
yahoo_obj = Daily_visit.objects.get(domain="yahoo.com")
yahoo_obj.per_date = .... # just append new list

tastypie mongoengine rest api for DynamicDocument

I am trying to save dynamic document in mongo using testypie and mongoengine my code is like this:
class Test(DynamicDocument):
pass
class TestResource(resources.MongoEngineResource):
class Meta:
queryset = models.Test.objects.all()
allowed_methods = ('get', 'post', 'put', 'delete')
authorization = authorization.Authorization()
and i am trying to save this object using rest api
{
"fname":"Shon",
"lname":"Lil"
}
I am geting this document in mongo
{
"_id" : ObjectId("50bdf1a0d6b9bb12db267be4"),
"_types" : [
"Test"
],
"_cls" : "Test"
}
I need document like this
{
"_id" : ObjectId("50be06fcd6b9bb14d07f8866"),
"_types" : [
"Test"
],
"lname" : "Lil",
"_cls" : "Test",
"fname" : "Shon"
}
can anybody help me on this?

adding a list field in mongo (python)

I have an id which some tags are associated with and I want to fill a collection with those tags. How do I do this?
Suppose I have the following:
key: 12345, tags = ["foo","bar","foobar"]
So I am hoping to get a document in a collection named "Tags" like this:
{
"_id" : ObjectId("4fbaa6076a56b2700d000000"),
"key" : 12345,
"tags" : ["foo","bar","foobar"]
}
How do I do this in python?

mongo_conn.Tags.insert({
"key": 12345,
"tags": ["foo","bar","foobar"]
})