I would like to modify the value of a field on a specific index of a nested type depending on another value of the same nested object or a field outside of the nested object.
As example, I have the current mapping of my index feed:
{
"feed": {
"mappings": {
"properties": {
"attacks_ids": {
"type": "keyword"
},
"created_by": {
"type": "keyword"
},
"date": {
"type": "date"
},
"groups_related": {
"type": "keyword"
},
"indicators": {
"type": "nested",
"properties": {
"date": {
"type": "date"
},
"description": {
"type": "text"
},
"role": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"value": {
"type": "keyword"
}
}
},
"malware_families": {
"type": "keyword"
},
"published": {
"type": "boolean"
},
"references": {
"type": "keyword"
},
"tags": {
"type": "keyword"
},
"targeted_countries": {
"type": "keyword"
},
"title": {
"type": "text"
},
"tlp": {
"type": "keyword"
}
}
}
}
}
Take the following document as example:
{
"took": 194,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "feed",
"_type": "_doc",
"_id": "W3CS7IABovFpcGfZjfyu",
"_score": 1,
"_source": {
"title": "Test",
"date": "2022-05-22T16:21:09.159711",
"created_by": "finch",
"tlp": "white",
"published": true,
"references": [
"test",
"test"
],
"tags": [
"tag1",
"tag2"
],
"targeted_countries": [
"Italy",
"Germany"
],
"malware_families": [
"family1",
"family2"
],
"groups_related": [
"group1",
"griup2"
],
"attacks_ids": [
""
],
"indicators": [
{
"value": "testest",
"description": "This is a test",
"type": "sha256",
"role": "file",
"date": "2022-05-22T16:21:09.159560"
},
{
"value": "testest2",
"description": "This is a test 2",
"type": "ipv4",
"role": "c2",
"date": "2022-05-22T16:21:09.159699"
}
]
}
}
]
}
}
I would like to make this update: indicators[0].value = 'changed'
if _id == 'W3CS7IABovFpcGfZjfyu'
or if title == 'some_title'
or if indicators[0].role == 'c2'
I already tried with a script, but it seems I can't manage to get it work, I hope the explanation is clear, ask any question if not, thank you.
Edit 1:
I managed to make it work, however it needs the _id, still looking for a way to do that without it.
My partial solution:
update = Pulse.get(id="XHCz7IABovFpcGfZWfz9") #Pulse is my document
update.update(script="for (indicator in ctx._source.indicators) {if (indicator.value=='changed2') {indicator.value='changed3'}}")
# Modify depending on the value of a field inside the same nested object
Let's suppose I have an incoming data in the following format:
{
"Name": "Test"
"Location": "Whatever",
"customerServices": [
{
"id": "test",
"cusId": "test",
"adr": "Adr 1",
"serviceCounty": "Center",
"area": "village"
},
{
"id": "test",
"cusId": "test",
"adr": "adr2",
"serviceCounty": "West",
"area": "city"
},
{
"id": "test",
"cusId": "test",
"adr": "test",
"serviceCounty": "West",
"area": "test"
}
]
}
Any ideas, how to write an aggregation query which would:
Create a new field called 'serviceAreas'. Type: list
For each item in the 'customerServices'. It would select: adr, serviceCounty and area fields.
Append them together into a single string and add to the newly created serviceAreas field.
It would select and do the operation only for distinct serviceCounty items
So the end result would be like this:
{
"Name": "Test"
"Location": "Whatever",
"customerServices": [
{
"id": "test",
"cusId": "test",
"adr": "Adr 1",
"serviceCounty": "Center",
"area": "village"
},
{
"id": "test",
"cusId": "test",
"adr": "adr2",
"serviceCounty": "West",
"area": "city"
},
{
"id": "test",
"cusId": "test",
"adr": "test",
"serviceCounty": "West",
"area": "test"
}
],
"serviceAreas": [
"Adr 1, Center, village", "adr2, West, city"
]
}
Any help is appreciated!
Here is what I have tried, but it didn't work out:
'serviceAreas': {
'$reduce': {
'input': '$serviceImpactHistory.event.services',
'initialValue': [],
'in': {
'$setUnion': [
'$$value', {'$concat': ['$$this.serviceCounty', '$$this.adr', '$$this.area']}
]
}
}
},
You don't need to use $setUnion, just create your string with $concat
db.collection.aggregate([
{
$addFields: {
serviceAreas: {
"$reduce": {
"input": "$customerServices",
"initialValue": "",
"in": {
"$concat": [
"$$value",
"$$this.serviceCounty",
", ",
"$$this.adr",
", ",
"$$this.area",
", "
]
}
}
}
}
}
])
Try it here
I have the following schema doc
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "foo",
"definitions": {
"stuff": {
"type": "object",
"properties": {
"id": {
"description": "the id",
"type": "string"
},
"type": {
"description": "the type",
"type": "string"
}
},
"oneOf": [{
"required": ["id"]
}, {
"required": ["type"]
}],
},
},
"type": "object",
"properties": {
"bar": {
"description": "blah",
"type": "object",
"additionalProperties": {
"$ref": "#/definitions/stuff"
}
}
},
"required": ["bar"]
}
and I'm trying to create the flask restx model, but I'm not sure how to model the required OneOf fields?
I think the following will work, but then it ignores the oneof requirement.
stuff_model = (
"Stuff Model",
"id": fields.String(description="the id", required=False),
"type": fields.String(description="the type", required=False)
)
bar_model = (
"Bar Model",
"bar": fields.Nested(stuff_model, required=True)
)
Using Python I need to extract all the API (just the API endpoint names) endpoints by reading from a JSON file.
Below is the sample JSON code,
{
"swagger": "2.0",
"info": {
"title": "None",
"description": "some thing over here sample content\n",
"version": "0.1"
},
"produces": [
"application/json"
],
"basePath": "/busrouting",
"schemes": [
"https"
],
"definitions": {
"OAuth2": {
"description": "some thing over here sample content\n",
"type": "coauthor",
"flow": "implicit",
"authorization": "NONE",
"scopes": {
"Zero": "three",
"One": "two"
}
}
},
"paths": {
"/service-provider/details": {
"get": {
"description": "some thing over here sample content",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "service provider",
"in": "query",
"description": "List of service-provider ID's",
"required": false,
"type": "array",
"items": {
"type": "string"
}
},
{
"name": "limit",
"in": "query",
"description": "Record limit. Default is 20",
"required": false,
"type": "integer"
},
{
"name": "offset",
"in": "query",
"description": "Record offset. Default is 0.",
"required": false,
"type": "integer"
}
],
"tags": [
"Service Providers"
],
"responses": {
"200": {
"description": "OK.",
"headers": {
"Link": {
"description": "some thing over here sample content\n",
"type": "array",
"items": {
"type": "string"
}
}
},
"schema": {
"type": "object",
"properties": {
"names": {
"type": "array",
"items": {
"$ref": "#/definitions/service-provider"
}
}
} #Sample add
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Forbidden"
},
"404": {
"description": "Zero"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
},
"/Bus/{bus-id}/names": {
"get": {
"description": "some thing over here sample content\n",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "Bus-id",
"in": "path",
"required": true,
"type": "string"
}
],
"tags": [
"BusNames"
],
"responses": {
"200": {
"description": "OK.",
"schema": {
"$ref": "#/definitions/busnames"
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Forbidden"
},
"404": {
"description": "Balances not found"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
},
"/busoperator/{bus-id}/names/routes": {
"get": {
"description": "some thing over here sample content.\n",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "bus-id",
"in": "path",
"required": true,
"type": "string"
},
{
"name": "limit",
"in": "query",
"description": "Record limit. Default is 10",
"required": false,
"type": "integer"
},
{
"name": "offset",
"in": "query",
"description": "Record offset. Default is 0.",
"required": false,
"type": "integer"
}
],
"tags": [
"bus route mapping"
],
"responses": {
"200": {
"description": "Recent Route",
"headers": {
"Link": {
"description": "some thing over here sample content.\n",
"type": "array",
"items": {
"type": "string"
}
}
},
"schema": {
"type": "object",
"properties": {
"Route details": {
"type": "array",
"items": {
"$ref": "#/definitions/Route"
}
}
}
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Not entitled to this account and its transactions"
},
"404": {
"description": "User has no recent transactions"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
},
"/operator/{id}/route/path": {
"get": {
"description": "some thing over here sample content.\n",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "bus-id",
"in": "path",
"required": true,
"type": "string"
},
{
"name": "limit",
"in": "query",
"description": "Record limit. Default is 10",
"required": false,
"type": "integer"
},
{
"name": "offset",
"in": "query",
"description": "Record offset. Default is 0.",
"required": false,
"type": "integer"
}
],
"tags": [
"Sample Reporting"
],
"responses": {
"200": {
"description": "sample actions",
"headers": {
"Link": {
"description": "some thing over here sample content\n",
"type": "array",
"items": {
"type": "string"
}
}
},
"schema": {
"type": "object",
"properties": {
"Boisterous": {
"type": "array",
"items": {
"$ref": "#/definitions/salesperson"
}
}
}
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Not entitled to this account and its transactions"
},
"404": {
"description": "User has no recent transactions"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
}
},
"definitions": {
"sample": {
"required": [
"demo",
"request"
],
"description": "Holds identifying attributes for an sample\n",
"properties": {
"request": {
"description": "some thing over here sample content\n",
"type": "string"
},
"sample one": {
"type": "string"
},
"sample two": {
"type": "string"
},
"number": {
"description": "some thing over here sample content",
"type": "string"
},
"name": {
"type": "string",
"description": "some thing over here sample content"
}
}
},
"dummy": {
"description": "some thing over here sample content\n",
"properties": {
"dummy": {
"$ref": "#/definitions/samples"
},
"data": {
"type": "string",
"description": "sample data "
},
"country": {
"type": "string",
"description": "some thing over here sample content"
},
"dataone": {
"$ref": "#/definitions/samples"
},
"error": {
"$ref": "#/definitions/Error"
},
"links": {
"description": "some thing over here sample content",
"type": "array",
"items": {
"$ref": "#/definitions/Links"
}
}
}
},
"reference": {
"description": "only description\n",
"properties": {
"available": {
"type": "number",
"format": "double",
"description": "some thing over here sample content"
},
"availableFormatted": {
"type": "string",
"description": "some thing over here sample content"
},
"held": {
"type": "number",
"format": "double",
"description": "some thing over here sample content."
},
"formatted": {
"type": "string",
"description": "some thing over here sample content"
},
"asst": {
"type": "string",
"format": "date-time",
"description": "Timestamp of , in UTC"
},
"sampler": {
"type": "string",
"format": "date-time",
"description": "Timestamp of , in users preferred TZ"
}
}
},
"dummy": {
"description": "Transaction done on an account",
"required": [
"because",
"versioned",
"postdates"
],
"properties": {
"tintype": {
"description": "sample content",
"type": "string"
},
"cringed": {
"description": "some thing over here sample content",
"type": "string",
"menu": [
"D",
"C"
]
},
"reference": {
"description": "Reference information",
"type": "string"
},
"amount": {
"description": "some thing over here sample content\n",
"type": "number",
"format": "double"
},
"formatted": {
"description": "formatted according to user's preferences\n"
},
"post Date": {
"type": "string",
"format": "date-time",
"description": "Date-time in UTC"
},
"postdate": {
"type": "string",
"format": "date-time",
"description": "Post Date-time in user-preferred TZ"
},
"narrative": {
"type": "string"
}
}
},
"Links": {
"description": "Related Links for the resource\n",
"properties": {
"rel": {
"description": "relationship to the resource",
"type": "string"
},
"ref": {
"description": "URL of the related link",
"type": "string"
}
}
},
"Error": { Sample
"description": "Error descriptor",
"properties": {
"code": {
"type": "integer",
"format": "intent"
},
"message": {
"type": "string"
},
"description": {
"type": "string"
},
"severity": {
"type": "string"
},
"location": {
"type": "string"
}
}
}
}
}
I have tried the below code:
import json
with open("example.json", "r") as reading:
data = json.load(reading)
print(data["paths"])
From here i need to go further the code to capture all the API endpoint name only.
In the sample json file, Under paths i need to capture all the endpoints and method type of an API as below,
In addition I would also like to capture the values for the below keys from the sample JSON code.
Under "parameters" i also need to capture the below keys,
name:
in:
required:
Expected output (just an example):
/service-provider/details
get
bus (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
/Bus/{bus-id}/names
get
bus-id (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
/busoperator/{bus-id}/names/routes
get
routes (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
/operator/{id}/route/path
get
id (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
Try this:
with open('example.json', 'r') as f:
data = json.load(f)
for path, values in data['paths'].items():
print(path)
for value in values:
print(value)
This should also get all the endpoints functions in a path if there are multiple.
To get a list of two-tuples (endpoint, type) use:
[(endpoint, value.keys[0]) for endpoint, value in data["paths"].items()]
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5006", "type": "Chocolate with Sprinkles" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
}
Concider above json string, I want to get all keys from this json string with proper hierarchy like
id ,
type,
name,
ppu,
batter,
then in batters :
batter,
topping
in batter: id, type
in topping: id, type
I am using python, is there any way out to do the purpose?
If this is a string just try:
import json
json.loads(your_variable)
If this is a dict parse it into string and use method from above.