Elasticsearch [5.6] Sum Aggregations Losing Precision? - python

I'm trying to do a sum aggregation of amounts ($21.28 for example). But in the aggregation result, it only shows (21.0)
I've also tried changing the type of the mapping to float, and I got the same results.
The query itself looks like:
'aggs': {
'total': {
'sum': {
'field': 'amount'
}
}
}
And the mapping looks like:
'amount': {
'type': 'double',
'index': 'not_analyzed',
'store': False
},
And finally, here's the result, I've omitted some data but the important bits are the amounts:
{
'took': 3,
'aggregations': {'total': {'value': 21.0}},
'hits': {'total': 1, 'max_score': 0.51623213,
'hits': [
{
'_score': 0.51623213,
'_index': 'some_index',
'_type': 'donation',
'_source': {
'amount': 21.28,
'created_on': '2019-06-15T01:37:42.451249+00:00'
}
}
]},
'timed_out': False,
'_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0}
}
I'd expect to see 21.28 in the results, and not 21.0.

My solution was to update to Elasticsearch 6.x

Related

How to filter nested json array on Python

I have a json array on python which I want to filter based on the value of : my_json_array['models']['variants']['condition']['type']
my json array looks like the following :
my_json_array = [
{'id': 1,
'models': [
{'color': {'code': '#626262', 'name': 'Gray'},
'variants': [{'id': 1,
'condition': [{'id': 1,
'type': 'type1',
'value': 14900},
{'id': 2,
'type': 'type2',
'value': 14000}]]
]
}]
I'm looking for a method to remove condition items with type = type2. The output should look like this :
my_json_array = [{
'id': 1,
'models': [
{'color': {'code': '#626262', 'name': 'Gray'},
'variants': [{'id': 1,
'condition': [{'id': 1,
'type': 'type1',
'value': 14900}]]
]
}]
Do you mean this?
my_json_array = [
{
'id': 1,
'models': [
{
'color': {'code': '#626262', 'name': 'Gray'},
'variants': [
{
'id': 1,
'condition': [
{
'id': 1,
'type': 'type1',
'value': 14900
},
{
'id': 2,
'type': 'type2',
'value': 14000
}
]
}
]
}
]
}
]
for mydict in my_json_array:
for model in mydict['models']:
for variant in model['variants']:
for condition in variant['condition']:
if condition['type']=="type2":
variant['condition'].remove(condition)
print(my_json_array) # [{'id': 1, 'models': [{'color': {'code': '#626262', 'name': 'Gray'}, 'variants': [{'id': 1, 'condition': [{'id': 1, 'type': 'type1', 'value': 14900}]}]}]}]

Why index keep on creating while hitting the api in elastic search

code is below
r = [{'eid': '1', 'data': 'Health'},
{'eid': '2', 'data': 'countries'},
{'eid': '3', 'data': 'countries currency'},
{'eid': '4', 'data': 'countries language'}]
from elasticsearch import Elasticsearch
es = Elasticsearch()
es.cluster.health()
es.indices.create(index='my-index_1', ignore=400)
for e in enumerate(r):
es.index(index="my-index_1", body=e[1])
search1 = es.search(index="my-index_1", body={'query': {'term' : {'data.keyword': 'Health'}}})
search1
First time out is below
{'took': 0,
'timed_out': False,
'_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0},
'hits': {'total': {'value': 0, 'relation': 'eq'},
'max_score': None,
'hits': []}}
Second time
{'took': 0,
'timed_out': False,
'_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0},
'hits': {'total': {'value': 1, 'relation': 'eq'},
'max_score': 1.2039728,
'hits': [{'_index': 'my-index_1',
'_type': '_doc',
'_id': 'Rov4UHMBpo0uANDoY2_5',
'_score': 1.2039728,
'_source': {'eid': '1', 'data': 'Health'}}]}}
Third time
{'took': 0,
'timed_out': False,
'_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0},
'hits': {'total': {'value': 2, 'relation': 'eq'},
'max_score': 1.2809337,
'hits': [{'_index': 'my-index_1',
'_type': '_doc',
'_id': 'Rov4UHMBpo0uANDoY2_5',
'_score': 1.2809337,
'_source': {'eid': '1', 'data': 'Health'}},
{'_index': 'my-index_1',
'_type': '_doc',
'_id': 'aov4UHMBpo0uANDonm_E',
'_score': 1.2809337,
'_source': {'eid': '1', 'data': 'Health'}}]}}
​Below tag are keep on repating while hitting again and again
{'_index': 'my-index_1',
'_type': '_doc',
'_id': 'aov4UHMBpo0uANDonm_E',
'_score': 1.2809337,
'_source': {'eid': '1', 'data': 'Health'}}
Is it because of enumerate?. My input is list of dictionary then which having multiple keys, otherwise how to parse this?
My expected out is it should show only one time for every hit
?

Interactive Plotly Barchart

I am attempting to adapt the Python Plotly example on 'Adding Sliders to Animations in Python' to a barchart but am getting a 'Figure field is invalid' message.
I am trying to adapt the input data to reflect that of a bar chart rather than a scatter chart (used in the example). I have created a grid:
Which I am using with the following code:
years = ['2007','2008','2009']
items = ['Name_1','Name_2']
col_name_template = '{column}'
for year in years:
frame = {'data': [], 'name': str(year)}
x_list = []
y_list = []
for item in items:
x_list.append(grid.get_column_reference(col_name_template.format(column='name')))
y_list.append(grid.get_column_reference(col_name_template.format(column=year)))
frame['data'].append(go.Bar(
x=x_list,
y=y_list
))
figure['frames'].append(frame)
slider_step = {'args': [
[year],
{'frame': {'duration': 300, 'redraw': False},
'mode': 'immediate',
'transition': {'duration': 300}}
],
'label': year,
'method': 'animate'}
sliders_dict['steps'].append(slider_step)
figure['layout']['sliders'] = [sliders_dict]
py.icreate_animations(figure, 'barchart example')
When trying to plot I get the following error:
Figure field is invalid. Reason: Raw data arrays are not allowed at this endpoint. Use grid references instead. Raw data found at the following paths in the figure...
How do I use only grid references but also ensure that a bar chart is plotted rather than a scatter chart?
I used the offline method of plotting plotly.offline.iplot which doesn't require grid references. The only downside of animations with bar charts is that transitions aren't currently supported.
Code below (including buttons and slider):
from plotly.offline import init_notebook_mode, iplot
from IPython.display import display, HTML
init_notebook_mode(connected = True)
years = ['2010', '2011', '2012']
items = ['A', 'B', 'C', 'D']
count = [
[1, 2, 3, 4],
[2, 3, 4, 1],
[3, 4, 1, 2]
]
figure = {
'data': [{
'type': 'bar',
'x': items,
'y': count[0]
}],
'layout': {
'xaxis': {
'title': 'X',
'gridcolor': '#FFFFFF',
'linecolor': '#000',
'linewidth': 1,
'zeroline': False,
'autorange': False
},
'yaxis': {
'title': 'Y',
'gridcolor': '#FFFFFF',
'linecolor': '#000',
'linewidth': 1,
'range': [0, 5],
'autorange': False
},
'title': 'Example Title',
'hovermode': 'closest',
'updatemenus': [{
'type': 'buttons',
'buttons': [{
'label': 'Play',
'method': 'animate',
'args': [None, {
'frame': {
'duration': 500,
'redraw': True
},
'fromcurrent': True,
'transition': {
'duration': 300,
'easing': 'quadratic-in-out'
}
}]
},
{
'label': 'End',
'method': 'animate',
'args': [None, {
'frame': {
'duration': 0,
'redraw': True
},
'fromcurrent': True,
'mode': 'immediate',
'transition': {
'duration': 0
}
}]
}
],
'direction': 'left',
'pad': {
'r': 10,
't': 87
},
'showactive': False,
'type': 'buttons',
'x': 0.1,
'xanchor': 'right',
'y': 0,
'yanchor': 'top'
}]
},
'frames': []
}
sliders_dict = {
'active': 0,
'yanchor': 'top',
'xanchor': 'left',
'currentvalue': {
'font': {
'size': 20
},
'prefix': 'Year:',
'visible': True,
'xanchor': 'right'
},
'transition': {
'duration': 300,
'easing': 'cubic-in-out'
},
'pad': {
'b': 10,
't': 50
},
'len': 0.9,
'x': 0.1,
'y': 0,
'steps': []
}
for index, year in enumerate(years):
frame = {
'data': [{
'type': 'bar',
'x': items,
'y': count[index]
}],
'name': str(year)
}
figure['frames'].append(frame)
slider_step = {
'args': [
[year],
{
'frame': {
'duration': 300,
'redraw': True
},
'mode': 'immediate',
'transition': {
'duration': 300
}
}
],
'label': year,
'method': 'animate'
}
sliders_dict['steps'].append(slider_step)
figure['layout']['sliders'] = [sliders_dict]
iplot(figure)

List indices must be integers, not dict

I'm making a script for the game and i need help about grabbing dictionary information out of json.
So far I've dealt with list and I didn't had problem grabbing that information.
req['result']['trades'][key]['output']
This is my try of grabbing the info after which i get error list indices must be integers, not dict.
Any help greatly appreciated.
json print, i have removed some account sensitive data.
{
'fue_step': 30,
'type': 'm',
'rubies': xxx,
'might': xxx,
'race': 'xxx',
'result': {
'trades': [
{
'id': 13,
'requirement': ['VenomTrapPart', 1],
'token_cost': 1,
'output': ['VulcansBlessingRare', 1],
'name': 'VenomVulcan',
'enabled': True
},
{
'id': 14,
'requirement': ['EternalRune', 50],
'token_cost': 1,
'output': ['ColossalMite500TroopPrizeItem', 1],
'name': 'EternalColossal',
'enabled': True
},
{
'id': 86,
'requirement':['RenameProclamation', 25],
'token_cost': 1,
'output': ['TimeTrickstersBag', 1],
'name': 'RenameTrick',
'enabled': True
},
{
'id': 95,
'requirement': ['FireDragonBlaze', 1],
'token_cost': 1,
'output': ['FireTroop1000TroopPrizeItem', 10],
'name': 'BlazeFireTroop',
'enabled': True
},
{
'id': 100,
'requirement': ['StoneDragonQuake', 1],
'token_cost': 1,
'output': ['StoneTroop10kTroopPrizeItem', 1],
'name': 'StoneDQStoneTroop',
'enabled': True
},
{
'id': 113,
'requirement': ['VulcansBlessing15Elite', 3],
'token_cost': 1,
'output': ['VulcansBlessing16Elite', 1],
'name': '15 --> 16 Elite',
'enabled': True
},
{
'id': 114,
'requirement': ['LunaPowder', 25],
'token_cost': 1,
'output': ['IceTroop50kTroopPrizeItem', 1],
'name': 'Luna Powder --> Soul Reapers',
'enabled': True
},
{
'id': 115,
'requirement': ['GreaterCrystalDefense', 5],
'token_cost': 1,
'output': ['MasterCrystalDefense', 1],
'name': 'Greater Defense --> Master Defense',
'enabled': True
},
{
'id': 116,
'requirement': ['NanoCanisters', 3],
'token_cost': 1,
'output': ['TestroniusDeluxe', 1],
'name': 'Nano Canisters for Deluxes',
'enabled': True
},
{
'id': 117,
'requirement': ['VulcansBlessing16Common', 3],
'token_cost': 1,
'output': ['VulcansBlessing17Common', 1],
'name': "Common +16's for Common +17",
'enabled': True
}
],
'trades_today': 0,
'doubloons_left': 9567,
'free_trades_left': 1,
'success': True,
'auto_confirm': True,
'trade_reset_date':1450771200
},
'client_cachebreaker': '1449879464'
I edited your edit to include indentation in your JSON so that it is more readable. As soon as the edit is approved, you'll be able to see it.
Here is an example of how to access a value deep within the object:
print req['result']['trades'][4]['name']
Output:
StoneDQStoneTroop
If you want to access dicts within a list by their values, here is a post that explains a few ways to do so. This would allow you, for example, to find a dict in req['result']['trades'] by its 'id' value instead of by its index within the 'trades' list.

While creating a Dictionary: TypeError: unhashable type: 'dict'

I'm trying to script a simple Character Generator that I can use for Pen and Paper RPG's. I was thinking about storing all my information in a nested dictionary and saving it into a JSON file.
However, while creating the following dictionary, I receive as error:
nhashable type: 'dict', focussing on {'cha': 1}}}
core_phb = {
'races': {
'Human': {
{'abilities': 'None'},
{'alignment': 'Neutral'},
{'size': 'Medium'},
{'speed': 6},
{'languages': 'Common'},
{'ability_modifiers': {
{'str': 1},
{'dex': 1},
{'con': 1},
{'int': 1},
{'wis': 1},
{'cha': 1}}}
},
'Dwarf': {
{'abilities': [
'ability1',
'ability2'
]},
{'alignment': 'Lawful Good'},
{'size': 'Medium'},
{'speed': 5},
{'languages': [
'Common',
'Dwarven'
]},
{'ability_modifiers': [
{'con': 2},
{'wis': 1}
]}
},
'Elf': {
{'abilities': [
'ability1',
'ability2'
]},
{'alignment': 'Chaotic Good'},
{'size': 'Medium'},
{'speed': 6},
{'languages': [
'Common',
'Elven'
]},
{'ability_modifiers': [
{'dex': 2},
{'int': 1}
]}
}
},
'classes': {
{'Fighter': {}},
{'Ranger': {}},
{'Wizard': {}}
},
'ability_scores': [
{'Str': 'str'},
{'Dex': 'dex'},
{'Con': 'con'},
{'Int': 'int'},
{'Wis': 'wis'},
{'Cha': 'cha'}]
}
I am simply trying to create the dictionary, not calling any keys from it.
As I understand from TypeError: unhashable type: 'dict' , I can use frozenset() to get keys.
Is there a better way to do what I am trying to do?
You seem to be making dictionaries {...} incorrectly for Python.
Lists look like this:
[ {'a': 1}, {'b': 1}, {'c': 1} ]
Dictionaries look like this:
{ 'a': 1, 'b': 2, 'c': 3 }
If I'm guessing the behavior you want correctly, then you probably wanted something like this:
human = {
'abilities': 'None',
'alignment': 'Neutral',
'size': 'Medium',
'speed': 6,
'languages': 'Common',
'ability_modifiers': {
'str': 1,
'dex': 1,
'con': 1,
'int': 1,
'wis': 1,
'cha': 1
}
}
The problem is not with the dicts, but with the sets. The elements of a set must be hashable. In
core_phb = {
'races': {
'Human': {
{'abilities': 'None'},
{'alignment': 'Neutral'},
{'size': 'Medium'},
{'speed': 6},
{'languages': 'Common'},
{'ability_modifiers': {
{'str': 1},
{'dex': 1},
{'con': 1},
{'int': 1},
{'wis': 1},
{'cha': 1}}}
},
'Dwarf': {
{'abilities': [
'ability1',
'ability2'
]},
{'alignment': 'Lawful Good'},
{'size': 'Medium'},
{'speed': 5},
{'languages': [
'Common',
'Dwarven'
]},
{'ability_modifiers': [
{'con': 2},
{'wis': 1}
]}
},
'Elf': {
{'abilities': [
'ability1',
'ability2'
]},
{'alignment': 'Chaotic Good'},
{'size': 'Medium'},
{'speed': 6},
{'languages': [
'Common',
'Elven'
]},
{'ability_modifiers': [
{'dex': 2},
{'int': 1}
]}
}
},
'classes': {
{'Fighter': {}},
{'Ranger': {}},
{'Wizard': {}}
},
'ability_scores': [
{'Str': 'str'},
{'Dex': 'dex'},
{'Con': 'con'},
{'Int': 'int'},
{'Wis': 'wis'},
{'Cha': 'cha'}]
}
the key is fine, but the value is an illegal set, because its elements are dicts. You could make frozensets from the sets and you'd be OK.
{frozenset({1})}
{frozenset({1})}
{{1}}
Traceback (most recent call last):
Python Shell, prompt 7, line 1
builtins.TypeError: unhashable type: 'set'
I think this :
'Human': {
{'abilities': 'None'},
{'alignment': 'Neutral'},
{'size': 'Medium'},
{'speed': 6},
{'languages': 'Common'},
{'ability_modifiers': {
{'str': 1},
{'dex': 1},
{'con': 1},
{'int': 1},
{'wis': 1},
{'cha': 1}}}
},
should be a list. Otherwise, each of the comma-separated elements is a mutable element which you are trying to store in a set. You are already doing it right with the very last entry:
'ability_scores': [
{'Str': 'str'},
{'Dex': 'dex'},
{'Con': 'con'},
{'Int': 'int'},
{'Wis': 'wis'},
{'Cha': 'cha'}]
so why not all the others?

Categories

Resources