I'm using Tornado_JSON which is based on jsonschema and there is a problem with my schema definition. I tried fixing it in an online schema validator and the problem seems to lie in "additionalItems": True. True with capital T works for python and leads to an error in the online validator (Schema is invalid JSON.). With true the online validator is happy and the example json validates against the schema, but my python script doesn't start anymore (NameError: name 'true' is not defined). Can this be resolved somehow?
#schema.validate(
"""input_schema={
'type': 'object',
'properties': {
'DB': {
'type': 'number'
},
'values': {
'type': 'array',
'items': [
{
'type': 'array',
'items': [
{
'type': 'string'
},
{
'type': [
'number',
'string',
'boolean',
'null'
]
}
]
}
],
'additionalItems': true
}
}
},
input_example={
'DB': 22,
'values': [['INT', 44],['REAL', 33.33],['CHAR', 'b']]
}"""
)
I changed it according to your comments ( external file with json.loads() ). Perfect. Thank you.
Put the schema in a triple-quoted string or an external file, then parse it with json.loads(). Use the lower-case spelling.
The error stems from trying to put a builtin Python datatype into a JSON schema. The latter is a template syntax that is used to check type consistency and should not hold actual data. Instead, under input_schema you'll want to define "additionalItems" to be of { "type": "boolean" } and then add it to the test JSON in your input_example with a boolean after for testing purposes.
Also, I'm not too familiar with Tornado_JSON but it looks like you aren't complying with the schema definition language by placing "additionalItems" inside of the "values" property. Bring that up one level.
More specifically, I think what you're trying to do should look like:
"values": {
...value schema definition...
}
"additionalItems": {
"type": "boolean"
}
And the input examples would become:
input_example={
"DB": 22,
"values": [['INT', 44],['REAL', 33.33],['CHAR', 'b']],
"additionalItems": true
}
Related
OK, I'm sure there's something wrong with this jsonschema, but I just can't seem to wrap my head around the problem.
I'm not going to post the actual code, but a minimal example that reproduces the issue.
Here's what I had before, which worked fine:
person = {
'type': 'object',
'properties': {
'name': {
'type': 'string'
}
'auth_token': {
'type': 'string',
},
'username': {
'type': 'string'
},
'password': {
'type': 'string'
}
},
'oneOf': [
{
'required': ['auth_token']
},
{
'required': ['username', 'password']
}
],
'required': ['name']
}
The idea here was that you always need to provide the name of the person, and then either an auth token or a username and password pair. As I said above, this validation worked fine, since we have parametrized tests that send all posible combinations of invalid JSON and evaluate the resulting error message, and those tests pass.
But then a new requirement came in and I needed to add a second mutually exclusive required pair of fields, which I did in this way:
person = {
'type': 'object',
'properties': {
'name': {
'type': 'string'
}
'auth_token': {
'type': 'string',
},
'username': {
'type': 'string'
},
'password': {
'type': 'string'
},
'project_id': {
'type': 'number'
},
'contract_date_from': {
'type': 'string'
}
'contract_date_to': {
'type': 'string'
}
},
'allOf': [
{
'oneOf': [
{
'required': ['auth_token']
},
{
'required': ['username', 'password']
}
]
},
{
'oneOf': [
{
'required': ['project_id']
},
{
'required': ['contract_date_from', 'contract_date_to']
}
]
}
],
'required': ['name']
}
But now the second validation always fails, whether the json provided is valid or invalid. The error message I get is:
{'name': 'John Doe', 'auth_token': '9d9a324b-26de-4ac3-85eb-05566e4a7204', 'username': None, 'password': None, 'project_id': 2785, 'contract_date_from': None, 'contract_date_to': None} is valid under each of {'required': ['contract_date_from', 'contract_date_to']}, {'required': ['project_id']}
No matter what values I send in those three fields (ie. project id, contract date from and contract date to), it fails with the same error. I've tried leaving all three empty, completing all three, and all permutations in between, but the error stays the same.
I've been reading the documentation for json schema but I can't seem to grasp what's going on with this example. I'm considering trying different approaches for this, but I'd really like to understand why this is not working. Any help is appreciated!
{'name': 'John Doe', 'auth_token': '9d9a324b-26de-4ac3-85eb-05566e4a7204', 'username': None, 'password': None, 'project_id': 2785, 'contract_date_from': None, 'contract_date_to': None} is valid under each of {'required': ['contract_date_from', 'contract_date_to']}, {'required': ['project_id']}
Read the error message more carefully: you requested that project_id be provided, OR contract_date_from and contract_date_to are provided, but you are providing all three of these. Providing a null value in a property is still providing a property. The error message is confusing, but you'd be failing validation anyway because null is not a string. Your evaluator is simply running the allOf->anyOfs first, so that's the error that comes back first. You should still get the type violation errors as well, though (if you don't, that's a bug: evaluators are required to provide ALL errors, not just the first.)
You can make the errors better at the expense of brevity by adding the "type" checks to live next to the "required" keywords. That will ensure the oneOf keywords produce failures rather than successes and maybe make the error messages more obvious.
I have AWS WAF CDK that is working with rules, and now I'm trying to add a rule in WAF with multiple statements, but I'm getting this error:
Resource handler returned message: "Error reason: You have used none or multiple values for a field that requires exactly one value., field: STATEMENT, parameter: Statement (Service: Wafv2, Status Code: 400, Request ID: 6a36bfe2-543c-458a-9571-e929142f5df1, Extended Request ID: null)" (RequestToken: b751ae12-bb60-bb75-86c0-346926687ea4, HandlerErrorCode: InvalidRequest)
My Code:
{
'name': 'ruleName',
'priority': 3,
'statement': {
'orStatement': {
'statements': [
{
'iPSetReferenceStatement': {
'arn': 'arn:myARN'
}
},
{
'iPSetReferenceStatement': {
'arn': 'arn:myARN'
}
}
]
}
},
'action': {
'allow': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': 'ruleName'
}
},
There are two things going on there:
Firstly, your capitalization is off. iPSetReferenceStatement cannot be parsed and creates an empty statement reference. The correct key is ipSetReferenceStatement.
However, as mentioned here, there is a jsii implementation bug causing some issues with the IPSetReferenceStatementProperty. This causes it not to be parsed properly resulting in a jsii error when synthesizing.
You can fix it by using the workaround mentioned in the post.
Add to your file containing the construct:
import jsii
from aws_cdk import aws_wafv2 as wafv2 # just for clarity, you might already have this imported
#jsii.implements(wafv2.CfnRuleGroup.IPSetReferenceStatementProperty)
class IPSetReferenceStatement:
#property
def arn(self):
return self._arn
#arn.setter
def arn(self, value):
self._arn = value
Then define your ip reference statement as follows:
ip_set_ref_stmnt = IPSetReferenceStatement()
ip_set_ref_stmnt.arn = "arn:aws:..."
ip_set_ref_stmnt_2 = IPSetReferenceStatement()
ip_set_ref_stmnt_2.arn = "arn:aws:..."
Then in the rules section of the webacl, you can use it as follows:
...
rules=[
{
'name': 'ruleName',
'priority': 3,
'statement': {
'orStatement': {
'statements': [
wafv2.CfnWebACL.StatementProperty(
ip_set_reference_statement=ip_set_ref_stmnt
),
wafv2.CfnWebACL.StatementProperty(
ip_set_reference_statement=ip_set_ref_stmnt_2
),
]
}
},
'action': {
'allow': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': 'ruleName'
}
}
]
...
This should synthesize your stack as expected.
I am trying to validate JSON, the schema for which specifies a list of dicts with arbitrary string keys, the corresponding values of which are dicts with a strict schema (i.e, the keys of the inner dict are strictly some string, here 'a'). From the Cerberus docs, I think that what I want is the 'keysrules' rule. The example in the docs seems to only show how to use 'keysrules' to validate arbitrary keys, but not their values. I wrote the below code as an example; the best I could do was assume that 'keysrules' would support a 'schema' argument for defining a schema for these values.
keysrules = {
'myDict': {
'type': 'dict',
'keysrules': {
'type': 'string',
'schema': {
'type': 'dict',
'schema': {
'a': {'type': 'string'}
}
}
}
}
}
keysRulesTest = {
'myDict': {
'arbitraryStringKey': {
'a': 'arbitraryStringValue'
},
'anotherArbitraryStringKey': {
'shouldNotValidate': 'arbitraryStringValue'
}
}
}
def test_rules():
v = Validator(keysrules)
if not v.validate(keysRulesTest):
print(v.errors)
assert(0)
This example does validate, and I would like it to not validate on 'shouldNotValidate', because that key should be 'a'. Does the flexibility implied by 'keysrules' (i.e, keys governed by 'keysrules' have no constraint other than {'type': 'string'}) propagate down recursively to all schemas underneath it? Or have I made some different error? How can I achieve my desired outcome?
I didn't want keysrules, I wanted valuesrules:
keysrules = {
'myDict': {
'type': 'dict',
'valuesrules': {
'type': 'dict',
'schema': {
'a': {'type': 'string'}
}
}
}
}
keysRulesTest = {
'myDict': {
'arbitraryStringKey': {
'a': 'arbitraryStringValue'
},
'anotherArbitraryStringKey': {
'shouldNotValidate': 'arbitraryStringValue'
}
}
}
def test_rules():
v = Validator(keysrules)
if not v.validate(keysRulesTest):
print(v.errors)
assert(0)
This produces my desired outcome.
When writing .ebextensions .config files, Amazon allows for long and shortform entries, for example these two configurations are identical:
Long form:
"option_settings": [
{
'Namespace': 'aws:rds:dbinstance',
'OptionName': 'DBEngine',
'Value': 'postgres'
},
{
'Namespace': 'aws:rds:dbinstance',
'OptionName': 'DBInstanceClass',
'Value': 'db.t2.micro'
}
]
Shortform:
"option_settings": {
"aws:rds:dbinstance": {
"DBEngine": "postgres",
"DBInstanceClass": "db.t2.micro"
}
}
However, all of the configurations I've seen only specify using a long form with boto3:
response = eb_client.create_environment(
... trimmed ...
OptionSettings=[
{
'Namespace': 'aws:rds:dbinstance',
'OptionName': 'DBEngineVersion',
'Value': '5.6'
},
... trimmed ...
)
Is it possible to use a dictionary with shortform entries with boto3?
Bonus: If not, why not?
Trial and error suggests no, you can not use the shortform config type.
However, if you are of that sort of persuasion you can do this:
def short_to_long(_in):
out = []
for namespace,key_vals in _in.items():
for optname,value in key_vals.items():
out.append(
{
'Namespace': namespace,
'OptionName': optname,
'Value': value
}
)
return out
Then elsewhere:
response = eb_client.create_environment(
OptionSettings=short_to_long({
"aws:rds:dbinstance": {
"DBDeletionPolicy": "Delete", # or snapshot
"DBEngine": "postgres",
"DBInstanceClass": "db.t2.micro"
},
})
So say i have the following document:
test_obj = {
'my_things':{
'id17': {
'blah': 3,
'weird': 'yay',
'thechallenge': ObjectId('5712d06fdb4d0856551300d2')
},
'id32': {
'blah': 62,
'weird': 'hoorah',
'thechallenge': ObjectId('5712d06fdb4d0856551300d4')
}
},
'_id': 12,
'an_extra_field': 'asdf'
}
for this document i have the following schema:
API.config['DOMAIN']['test_obj']['schema'] = {
'id': {'type': 'int'},
'an_extra_field': {'type': 'string'},
'my_things': {
'type': 'dict',
'valueschema': {
'type': 'dict',
'schema': {
'blah': {'type': 'dict'},
'weird': {'type': 'string'},
'thechallenge': {'type': 'objectid'}
}
}
}
}
Now say i make a patch with the following pseudocode:
data = {
'mythings': {
'id17': {
'thechallenge': '5712d06fdb4d0856551300d8'
}
}
}
PATCH(url='/v1/test_objs/12', data=data)
When I make this patch Cerberus raises an error during validation, saying "value '5712d06fdb4d0856551300d8' cannot be converted to a ObjectId". Now this is a valid object id, and i find that if I make a patch to other non-valueschema fields it does not raise this error. It seems like valueschema was not meant to have a value of dict, and adding an extra 'schema' attribute was the only way i could get around cerberus raising a schemaerror/having cerberus actually validate my fields. But eve does not appear to actually be serializing my fields in my dictionary correctly. It should be of type ObjectId when it gets passed to Cerberus.
The way i'm temporarily getting around this is by manipulating my the code in Eve. In common.py (module) in serialize (function) in line 398 i added, where it checks if the field schema is a 'valueschema':
elif field_type == 'dict' and 'schema' in field_schema['valueschema']:
for subdocument in document[field].values():
serialize(subdocument, schema=field_schema['valueschema']['schema'])
Should i not be using type dict for the valueschema? If not how else should i handle this scenario? I would like to not have to maintain my own fork of Eve, so if others do want the ability to have valueschema be of type dict should i submit a pull-request for this change?
This has been fixed with Eve v0.6.4, which has just been released.