Develop Reference

CDK WAF Python Multiple Statement velues error

I have AWS WAF CDK that is working with rules, and now I'm trying to add a rule in WAF with multiple statements, but I'm getting this error:
Resource handler returned message: "Error reason: You have used none or multiple values for a field that requires exactly one value., field: STATEMENT, parameter: Statement (Service: Wafv2, Status Code: 400, Request ID: 6a36bfe2-543c-458a-9571-e929142f5df1, Extended Request ID: null)" (RequestToken: b751ae12-bb60-bb75-86c0-346926687ea4, HandlerErrorCode: InvalidRequest)
My Code:
{
'name': 'ruleName',
'priority': 3,
'statement': {
'orStatement': {
'statements': [
{
'iPSetReferenceStatement': {
'arn': 'arn:myARN'
}
},
{
'iPSetReferenceStatement': {
'arn': 'arn:myARN'
}
}
]
}
},
'action': {
'allow': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': 'ruleName'
}
},

There are two things going on there:
Firstly, your capitalization is off. iPSetReferenceStatement cannot be parsed and creates an empty statement reference. The correct key is ipSetReferenceStatement.
However, as mentioned here, there is a jsii implementation bug causing some issues with the IPSetReferenceStatementProperty. This causes it not to be parsed properly resulting in a jsii error when synthesizing.
You can fix it by using the workaround mentioned in the post.
Add to your file containing the construct:
import jsii
from aws_cdk import aws_wafv2 as wafv2 # just for clarity, you might already have this imported
#jsii.implements(wafv2.CfnRuleGroup.IPSetReferenceStatementProperty)
class IPSetReferenceStatement:
#property
def arn(self):
return self._arn
#arn.setter
def arn(self, value):
self._arn = value
Then define your ip reference statement as follows:
ip_set_ref_stmnt = IPSetReferenceStatement()
ip_set_ref_stmnt.arn = "arn:aws:..."
ip_set_ref_stmnt_2 = IPSetReferenceStatement()
ip_set_ref_stmnt_2.arn = "arn:aws:..."
Then in the rules section of the webacl, you can use it as follows:
...
rules=[
{
'name': 'ruleName',
'priority': 3,
'statement': {
'orStatement': {
'statements': [
wafv2.CfnWebACL.StatementProperty(
ip_set_reference_statement=ip_set_ref_stmnt
),
wafv2.CfnWebACL.StatementProperty(
ip_set_reference_statement=ip_set_ref_stmnt_2
),
]
}
},
'action': {
'allow': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': 'ruleName'
}
}
]
...
This should synthesize your stack as expected.

Is there a way to get the particular Values from JSON Array using robot or Python code?

JSON OUTPUT:
${response}= [
{
"Name":"7122Project",
"checkBy":[
{
"keyId":"NA",
"target":"1232"
}
],
"Enabled":false,
"aceess":"123"
},
{
"Name":"7122Project",
"checkBy":[
{
"keyId":"_GU6S3",
"target":"123"
}
],
"aceess":"11222",
"Enabled":false
},
{
"Name":"7122Project",
"checkBy":[
{
"keyId":"-1lLUy",
"target":"e123"
}
],
"aceess":"123"
}
]
Need to get the keyId values from json without using hardcoded index using robot?
I did
${ID}= set variable ${response[0]['checkBy'][0]['keyId']}
But I need to check the length get all keyID values and store the values that dose not contain NA
How can I do check length and use for loop using robot framework?

I suppose you can have more elements in checkBy arrays, like so:
response = [
{
"Name":"7122Project",
"checkBy": [
{
"keyId": "NA",
"target": "1232"
}
],
"Enabled": False,
"aceess": "123"
},
{
"Name": "7122Project",
"checkBy": [
{
"keyId": "_GUO6g6S3",
"target": "123"
}
],
"aceess": "11222",
"Enabled": False
},
{
"Name": "7122Project",
"checkBy": [
{
"keyId": "-1lLlZOUy",
"target": "e123"
},
{
"keyId": "test",
"target": "e123"
}
],
"aceess": "123"
}
]
then you can key all keyIds in Python with this code:
def get_key_ids(response):
checkbys = [x["checkBy"] for x in response]
key_ids = []
for check_by in checkbys:
for key_id in check_by:
key_ids.append(key_id["keyId"])
return key_ids
for the example above, it will return: ['NA', '_GUO6g6S3', '-1lLlZOUy', 'test_NA'].
You want to get both ids with NA and without NA, so perhaps you can change the function a bit:
def get_key_ids(response, predicate):
checkbys = [x["checkBy"] for x in response]
key_ids = []
for check_by in checkbys:
for key_id in check_by:
if predicate(key_id["keyId"]):
key_ids.append(key_id["keyId"])
return key_ids
and use it like so:
get_key_ids(response, lambda id: id == "NA") # ['NA']
get_key_ids(response, lambda id: id != "NA") # ['_GUO6g6S3', '-1lLlZOUy', 'test_NA']
get_key_ids(response, lambda id: "NA" in id) # ['NA', 'test_NA']
get_key_ids(response, lambda id: "NA" not in id) # ['_GUO6g6S3', '-1lLlZOUy']
Now it's just a matter of creating a library and importing it into RF. You can get inspiration in the official documentation.
But I need to check the length get all keyID values and store the values that dose not contain NA
I don't completely understand what you are up to. Do you mean length of keyId strings, like "NA" and its length of 2, or the number of keyIds in the response?
How can I do check length and use for loop using robot framework?
You can use keyword Should Be Equal * from BuiltIn library. Some examples of for loops could be found in the user guide here.
Now you should have all the parts you need to accomplish your task, you can try to put it all together.

Pymongo include only the fields which are starting with a name

For example, if this is my record
{
"_id":"123",
"name":"google",
"ip_1":"10.0.0.1",
"ip_2":"10.0.0.2",
"ip_3":"10.0.1",
"ip_4":"10.0.1",
"description":""}
I want to get only those fields starting with 'ip_'. Consider I have 500 fields & only 15 of them start with 'ip_'
Can we do something like this to get the output -
db.collection.find({id:"123"}, {'ip*':1})
Output -
{
"ip_1":"10.0.0.1",
"ip_2":"10.0.0.2",
"ip_3":"10.0.1",
"ip_4":"10.0.1"
}

The following aggregate query, using PyMongo, returns documents with the field names starting with "ip_".
Note the various aggregation operators used: $filter, $regexMatch, $objectToArray, $arrayToObject. The aggregation pipeline the two stages $project and $replaceWith.
pipeline = [
{
"$project": {
"ipFields": {
"$filter" : {
"input": { "$objectToArray": "$$ROOT" },
"cond": { "$regexMatch": { "input": "$$this.k" , "regex": "^ip" } }
}
}
}
},
{
"$replaceWith": { "$arrayToObject": "$ipFields" }
}
]
pprint.pprint(list(collection.aggregate(pipeline)))

I am unaware of a way to specify an expression that would decide which hash keys would be projected. MongoDB has projection operators but they deal with arrays and text search.
If you have a fixed possible set of ip fields, you can simply request all of them regardless of which fields are present in a particular document, e.g. project with
{ip_1: true, ip_2: true, ...}

Improve the performce of a MongoDB query which uses a "$where" expression

I need to run the following query on a MongoDB server:
QUERY = {
"$and" : [
{"x" : {'$gt' : 1.0}},
{"y" : {'$gt' : 0.1}},
{"$where" : 'this.s1.length < this.s2.length+3'}
]
}
This query is very slow, due to the JavaScript expression which the server needs to execute on every document in the collection.
Is there any way for me to optimize it?
I thought about using the $size operator, but I'm not really sure that it works on strings, and I'm even less sure on how to compare its output on a pair of strings (as is the case here).
Here is the rest of my script, in case needed:
from pymongo import MongoClient
USERNAME = ...
PASSWORD = ...
SERVER_NAME = ...
DATABASE_NAME = ...
COLLECTION_NAME = ...
uri = 'mongodb://{}:{}#{}/{}'.format(USERNAME,PASSWORD,SERVER_NAME,DATABASE_NAME)
mongoClient = MongoClient(uri)
collection = mongoClient[DATABASE_NAME][COLLECTION_NAME]
cursor = collection.find(QUERY)
print cursor.count()
The pymongo version is 3.4.

You can use aggregation framework, which provides $strLenCP to get length of a string and $cmp to compare them:
db.collection.aggregate(
[
{
$match: {
"x" : {'$gt' : 1.0},
"y" : {'$gt' : 0.1}
}
},
{
$addFields: {
str_cmp: { $cmp: [ { $strLenCP: "$s1" }, { $add: [ { $strLenCP: "$s2" }, 3 ] } ] }
}
},
{
$match: {
"str_cmp": -1,
}
}
]
)

MongoDB pipeline unwind and check for empty array

I'm unwinding one field which is an array of date objects, however in some cases there are empty array's which is fine. I'd like the same treatment using a pipeline, but in some cases, I want to filter the results which have an empty array.
pipeline = []
pipeline.append({"$unwind": "$date_object"})
pipeline.append({"$sort": {"date_object" : 1}})
I want to use the pipeline format, however the following code does not return any records:
pipeline.append({"$match": {"date_object": {'$exists': False }}})
nor does the following work:
pipeline.append({"$match": {"date_object": []}})
and then:
results = mongo.db.xxxx.aggregate(pipeline)
I'm also trying:
pipeline.append({ "$cond" : [ { "$eq" : [ "$date_object", [] ] }, [ { '$value' : 0 } ], '$date_object' ] } )
But with this I get the following error:
.$cmd failed: exception: Unrecognized pipeline stage name: '$cond'
However if I query using find such as find({"date_object": []}), I can get these results. How can I make this work with the pipeline.

I've done in MongoDB shell, but it can be translated into Python easily in python language.
Is it your requirements?
I suppose you have such structure:
db.collection.save({foo:1, date_object:[new Date(), new Date(2016,1,01,1,0,0,0)]})
db.collection.save({foo:2, date_object:[new Date(2016,0,16,1,0,0,0),new Date(2016,0,5,1,0,0,0)]})
db.collection.save({foo:3, date_object:[]})
db.collection.save({foo:4, date_object:[new Date(2016,1,05,1,0,0,0), new Date(2016,1,06,1,0,0,0)]})
db.collection.save({foo:5, date_object:[]})
// Get empty arrays after unwind
db.collection.aggregate([
{$project:{_id:"$_id", foo:"$foo",
date_object:{
$cond: [ {"$eq": [{ $size:"$date_object" }, 0]}, [null], "$date_object" ]
}
}
},
{$unwind:"$date_object"},
{$match:{"date_object":null}}
])
// Get empty arrays before unwind
db.collection.aggregate([
{$match:{"date_object.1":{$exists:false}}},
{$project:{_id:"$_id", foo:"$foo",
date_object:{
$cond: [ {"$eq": [{ $size:"$date_object" }, 0]}, [null], "$date_object" ]
}
}
},
{$unwind:"$date_object"}
])
Only empty date_object
[
{
"_id" : ObjectId("56eb0bd618d4d09d4b51087a"),
"foo" : 3,
"date_object" : null
},
{
"_id" : ObjectId("56eb0bd618d4d09d4b51087c"),
"foo" : 5,
"date_object" : null
}
]
At the end, if you need only empty date_object, you don't need to aggregate, you can easely achieve it with find:
db.collection.find({"date_object.1":{$exists:false}},{date_object:0})
Output
{
"_id" : ObjectId("56eb0bd618d4d09d4b51087a"),
"foo" : 3
}
{
"_id" : ObjectId("56eb0bd618d4d09d4b51087c"),
"foo" : 5
}