Develop Reference

cant do case insensitive search in elastic search

I'm new to elastic search and trying to do this query right.
So I'm having a document like this:
{
"id": 1,
"name": "Văn Hiến"
}
I want to get that document in 3 cases:
1/ User input is: "v" or "h" or "i",...
2/ User input is: "Văn" or "văn" or "hiến",...
3/ User input is: "va" or "van" or "van hi",...
I'm currently can search for case 1 and 2, but not case 3, where the user input don't have the 'tonal' of the Vietnamese language
This is my query, I'm using Python:
query = {
"bool": {
"should": [
{
"match": {
"name": name.lower()
}
},
{
"wildcard": {
"name": {
"value": f"*{name.lower()}*"
}
}
}
]
}
}
Can anyone help me with this? Any helps will be apperciated

Use the lowercase_filter and mapping_character_filter functions in your mapping.
the following mapping and query will work for all the three usecases you mentioned
Mapping Example:
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"type": "custom",
"tokenizer": "my_tokenizer",
"filter": [
"lowercase"
],
"char_filter": [
"my_mappings_char_filter"
]
}
},
"char_filter": {
"my_mappings_char_filter": {
"type": "mapping",
"mappings": [
"ă => a",
"ế => e"
]
}
},
"tokenizer": {
"my_tokenizer": {
"type": "ngram",
"min_gram": 1,
"max_gram": 10,
"token_chars": [
"letter"
]
}
}
},
"max_ngram_diff" : "9"
},
"mappings": {
"properties": {
"name": {
"type": "text",
"analyzer": "my_analyzer",
"fields": {
"facet": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
Example Query:
{
"query" : {
"query_string" :{
"query":"van hi",
"type": "best_fields",
"default_field": "name"
}
}
}

python parsing strange JSON data

How should I parse (with Python3) data in this "unusual" format?
As you can see inside the "variables" dictionary the data that is in capitals has no label, it is provided as a literal. Therefore when I loop over the entries inside "variables" all I get is the strings in capitals, nothing else. I need, obviously, to get the capitals plus the value inside it.
{
"variables": {
"ABSENCE_OSL_PROD": {
"value": "REZWWnBTejN5Ng=="
},
"ACTION_OSL_INT": {
"value": "S0RXSVNTbmFhNw=="
},
"ACTION_OSL_PROD": {
"value": "RUJCaDJGnmFnUg=="
},
"API_STORE_OSL_INT": {
"value": "U3lxaVhogWtIcg=="
}
},
"id": 4,
"type": "Vsts"
}

To load the variables inside variables in the local variable space:
data = {
"variables": {
"ABSENCE_OSL_PROD": {
"value": "REZWWnBTejN5Ng=="
},
"ACTION_OSL_INT": {
"value": "S0RXSVNTbmFhNw=="
},
"ACTION_OSL_PROD": {
"value": "RUJCaDJGnmFnUg=="
},
"API_STORE_OSL_INT": {
"value": "U3lxaVhogWtIcg=="
}
},
"id": 4,
"type": "Vsts"
}
for variable_name, variable_content in data['variables'].items():
locals()[variable_name] = variable_content['value']
print(ABSENCE_OSL_PROD)
# prints "REZWWnBTejN5Ng=="

With dict comprehension you can get a time efficient manner:
ugly = {
"variables": {
"ABSENCE_OSL_PROD": {
"value": "REZWWnBTejN5Ng=="
},
"ACTION_OSL_INT": {
"value": "S0RXSVNTbmFhNw=="
},
"ACTION_OSL_PROD": {
"value": "RUJCaDJGnmFnUg=="
},
"API_STORE_OSL_INT": {
"value": "U3lxaVhogWtIcg=="
}
},
"id": 4,
"type": "Vsts"
}
proper = {elt: ugly["variables"][elt]["value"] for elt in ugly["variables"]}
print(proper)
returns
{'ABSENCE_OSL_PROD': 'REZWWnBTejN5Ng==', 'ACTION_OSL_INT': 'S0RXSVNTbmFhNw==', 'ACTION_OSL_PROD': 'RUJCaDJGnmFnUg==', 'API_STORE_OSL_INT': 'U3lxaVhogWtIcg=='}```

MongoDB - How to aggregate data for each record

I have some stored data like this:
{
"_id" : 1,
"serverAddresses" : {
"name" : "0.0.0.0:8000",
"name2": "0.0.0.0:8001"
}
}
I need aggregated data to this:
[
{
"gameId": "1",
"name": "name1",
"url": "0.0.0.0:8000"
},
{
"gameId": "1",
"name": "name2",
"url": "0.0.0.0:8001"
}
]
What is the solution without using for loop?

$project - Add addresses field by converting $serverAddress to (key-value) array.
$unwind - Descontruct addresses field to multiple documents.
$replaceRoot - Decorate the output document based on (2).
db.collection.aggregate([
{
"$project": {
"addresses": {
"$objectToArray": "$serverAddresses"
}
}
},
{
$unwind: "$addresses"
},
{
"$replaceRoot": {
"newRoot": {
gameId: "$_id",
name: "$addresses.k",
address: "$addresses.v"
}
}
}
])
Sample Mongo Playground

Navigating through a JSON with multiples arrays in Python

I'm trying to go through a JSON by using python but I can't access the "mbid" node. I want to print only the first "mbid" node.
Here is my function :
def get_data():
newJsonx = dict()
for item in data["resultsPage"]["results"]["calendarEntry"]:
mbid = item["event"]["performance"][0]["artist"]["identifier"][0]["mbid"]
With this function i get this error : IndexError: list index out of range
but when I'm doing
def get_data():
newJsonx = dict()
for item in data["resultsPage"]["results"]["calendarEntry"]:
mbid = item["event"]["performance"][0]["artist"]["identifier"]
And print(mbid), I'm getting a correct answer :
"identifier": [
{
"mbid": "6655955b-1c1e-4bcb-84e4-81bcd9efab30"
},
{
"mbid": "1b1b1b1b-1c1d"
}
]
So means I don't have a problem with the data. Maybe I'm doing something wrong with the second array?
Here is an example of the JSON structure :
{
"resultsPage": {
"status": "ok",
"results": {
"calendarEntry": [
{
"reason": {
},
"event": {
"performance": [
{
"id": 72641494,
"displayName": "Arnalds",
"artist": {
"id": 590465,
"identifier": [
{
"mbid": "6655955b-1c1e-4bcb-84e4-81bcd9efab30"
},
{
"mbid": "1b1b1b1b-1c1d"
}
]
}
}
]
}
}
]
}
}
}
Thanks for your time

def get_data():
newJsonx = dict()
for item in data["resultsPage"]["results"]["calendarEntry"]:
performance=item["event"]["performance"]
if performace:
identifier=performace[0]["artist"]["identifier"]
if identifier:
mbid=identifier[0]["mbid"]

How to specify Stopwords in Elasticsearch mapping using python

I have this python code where I first create a Elasticsearch mapping and then after data is inserted I do searching for that data:
# Create Data mapping
data_mapping = {
"mappings": {
(doc_type): {
"properties": {
"data_id": {
"type": "string",
"fields": {
"stemmed": {
"type": "string",
"analyzer": "english"
}
}
},
"data":{
"type": "array",
"fields": {
"stemmed": {
"type": "string",
"analyzer": "english"
}
}
},
"resp": {
"type": "string",
"fields": {
"stemmed": {
"type": "string",
"analyzer": "english"
}
}
},
"update": {
"type": "integer",
"fields": {
"stemmed": {
"type": "integer",
"analyzer": "english"
}
}
}
}
}
}
}
#Search
data_search = {
"query": {
"function_score": {
"query": {
"match": {
'data': question
}
},
"field_value_factor": {
"field": "update",
"modifier": "log2p"
}
}
}
}
response = es.search(index=doc_type, body=data_search)
Now what I am unable to figure out where and how to specify stopwords in the above code? This link gives an example of using stopwords but I am unable to relate it to my code. Do I need to specify in the data mapping section, search section or both? And how do I specify it?
Any example help would be appreciated!
UPDATE: Based on some comments suggestion is to add either analysis section or settings sections but I am not sure how should I add those to the mapping section I have written above.