I'm using Django 3.0.2.
I have a serializer defined:
class ValueNestedSerializer(request_serializer.Serializer):
lower = request_serializer.DateTimeField(required=True, allow_null=False, format=None, input_formats=['%Y-%m-%dT%H:%M:%SZ',])
upper = request_serializer.DateTimeField(required=True, allow_null=False, format=None, input_formats=['%Y-%m-%dT%H:%M:%SZ',])
class DateRangeSerializer(request_serializer.Serializer):
attribute = request_serializer.CharField(default="UPLOAD_TIME")
operator = request_serializer.CharField(default="between_dates")
value = ValueNestedSerializer(required=True)
timezone = request_serializer.CharField(default="UTC")
timezoneOffset = request_serializer.IntegerField(default=0)
class BaseQueryPayload(request_serializer.Serializer):
appid = request_serializer.CharField(required=True, validators=[is_valid_appid])
filters = request_serializer.ListField(
required=True, validators=[is_validate_filters],
min_length=1
)
date_range = DateRangeSerializer(required=True)
And the payload :
{
"appid": "6017cef554df4124274ef36d",
"filters": [
{
"table": "session",
"label": "1month"
}
],
"date_range": {
"value": {
"lower": "2023-01-01T01:00:98Z",
"upper": "2023-01-20T01:00:98Z"
}
},
"page": 1
}
But I get this validation error:
{
"error": {
"date_range": {
"value": {
"lower": [
"Datetime has wrong format. Use one of these formats instead: YYYY-MM-DDThh:mm:ssZ."
],
"upper": [
"Datetime has wrong format. Use one of these formats instead: YYYY-MM-DDThh:mm:ssZ."
]
}
}
}
}
The suggested format YYYY-MM-DDThh:mm:ssZ is similar to what is passed.
Am I missing anything here?
Format is wrong because seconds must be in 0-59 range (while you try to post 98s)
Related
Here is my Pydantic model:
from enum import Enum
from pydantic import BaseModel
class ProfileField(str, Enum):
mobile = "mobile"
email = "email"
address = "address"
interests ="interests" # need list of strings
class ProfileType(str, Enum):
primary = "primary"
secondary = "secondary"
class ProfileDetail(BaseModel):
name: ProfileField
value: str
type: ProfileType
My API is accepting this type of JSON and its working fine.
{
"data": [
{
"name": "email",
"value": "abcd#gmail.com",
"type": "primary"
}
]
}
The requirement is email is string type and needs a regex, mobile is integer type and also needs a regex, and address is a string and needs to be restricted to 50 characters.
Is it possible to add corresponding validations?
Discriminated union and built-in types/validators
If I understand correctly, the actual JSON data you receive has the top-level data key and its value is an array of objects that you currently represent with your ProfileDetail schema.
If that is the case, you may be better served by not using an Enum at all for your name field and instead defining a discriminated union based on the value of the name field. You can write a separate model for each case (mobile, email, and address) and delegate validation to each of them for their own case.
Since all three of them share a base schema, you can define a base model for them to inherit from to reduce repetition. The type field for example can stay an Enum (Pydantic handles validation of those out of the box) and can be inherited by the three submodels.
For mobile and address it sounds like you can just use constr to define your constraints via the regex and max_length parameters respectively.
For email, you can use the built-in Pydantic type EmailStr (subtype of str). You'll just need to install the optional dependency with pip install 'pydantic[email]'.
That way you should not even need to write any custom validators.
Here is the setup I suggest:
from enum import Enum
from typing import Annotated, Literal, Union
from pydantic import BaseModel, EmailStr, Field, constr
class ProfileType(str, Enum):
primary = "primary"
secondary = "secondary"
class BaseProfileFieldData(BaseModel):
value: str
type: ProfileType
class MobileData(BaseProfileFieldData):
value: constr(regex=r"\d{5,}") # your actual regex here
name: Literal["mobile"]
class EmailData(BaseProfileFieldData):
value: EmailStr
name: Literal["email"]
class AddressData(BaseProfileFieldData):
value: constr(max_length=50)
name: Literal["address"]
ProfileField = Annotated[
Union[MobileData, EmailData, AddressData],
Field(discriminator="name")
]
class ProfileDetails(BaseModel):
data: list[ProfileField]
Tests
Let's test it with some fixtures:
test_data_mobile_valid = {
"name": "mobile",
"value": "123456",
"type": "secondary",
}
test_data_mobile_invalid = {
"name": "mobile",
"value": "12",
"type": "secondary",
}
test_data_email_valid = {
"name": "email",
"value": "abcd#gmail.com",
"type": "primary",
}
test_data_email_invalid = {
"name": "email",
"value": "abcd#gmail#..",
"type": "primary",
}
test_data_address_valid = {
"name": "address",
"value": "some street 42, 12345 example",
"type": "secondary",
}
test_data_address_invalid = {
"name": "address",
"value": "x" * 51,
"type": "secondary",
}
test_data_invalid_name = {
"name": "foo",
"value": "x",
"type": "primary",
}
test_data_invalid_type = {
"name": "mobile",
"value": "123456",
"type": "bar",
}
The first six should be self explanatory. test_data_invalid_name should cause an error because "foo" is not a valid discriminator value for name. test_data_invalid_type should demonstrate the built-in enum validator catching the invalid type value "bar".
Let's test the valid data first:
if __name__ == "__main__":
from pydantic import ValidationError
obj = ProfileDetails.parse_obj({
"data": [
test_data_mobile_valid,
test_data_email_valid,
test_data_address_valid,
]
})
print(obj.json(indent=4))
...
Output:
{
"data": [
{
"value": "123456",
"type": "secondary",
"name": "mobile"
},
{
"value": "abcd#gmail.com",
"type": "primary",
"name": "email"
},
{
"value": "some street 42, 12345 example",
"type": "secondary",
"name": "address"
}
]
}
No surprises here. Now test those that should not pass the value validation:
if __name__ == "__main__":
...
try:
ProfileDetails.parse_obj({
"data": [
test_data_mobile_invalid,
test_data_email_invalid,
test_data_address_invalid,
]
})
except ValidationError as exc:
print(exc.json(indent=4))
...
Output:
[
{
"loc": [
"data",
0,
"MobileData",
"value"
],
"msg": "string does not match regex \"\\d{5,}\"",
"type": "value_error.str.regex",
"ctx": {
"pattern": "\\d{5,}"
}
},
{
"loc": [
"data",
1,
"EmailData",
"value"
],
"msg": "value is not a valid email address",
"type": "value_error.email"
},
{
"loc": [
"data",
2,
"AddressData",
"value"
],
"msg": "ensure this value has at most 50 characters",
"type": "value_error.any_str.max_length",
"ctx": {
"limit_value": 50
}
}
]
Caught all the wrong values. Now just to be sure, the last two fixtures:
if __name__ == "__main__":
...
try:
ProfileDetails.parse_obj({
"data": [
test_data_invalid_name,
test_data_invalid_type,
]
})
except ValidationError as exc:
print(exc.json(indent=4))
Output:
[
{
"loc": [
"data",
0
],
"msg": "No match for discriminator 'name' and value 'foo' (allowed values: 'mobile', 'email', 'address')",
"type": "value_error.discriminated_union.invalid_discriminator",
"ctx": {
"discriminator_key": "name",
"discriminator_value": "foo",
"allowed_values": "'mobile', 'email', 'address'"
}
},
{
"loc": [
"data",
1,
"MobileData",
"type"
],
"msg": "value is not a valid enumeration member; permitted: 'primary', 'secondary'",
"type": "type_error.enum",
"ctx": {
"enum_values": [
"primary",
"secondary"
]
}
}
]
Seems like we get the desired behavior from our model.
Caveat
If you really want a separate model like the ProfileDetail you showed in your question, that will not be possible with discriminated unions because those rely on being defined for a field on a separate model. In that case you'll actually have to write a custom validator (probably a root_validator) to ensure consistency between name and value.
I have a collection of documents that looks like this
{
"_id": "4",
"contacts": [
{
"email": "mail#mail.com",
"name": "A1",
"phone": "00",
"crashNotificationEnabled": false,
"locationShared": true,
"creationDate": ISODate("2020-10-19T15:19:04.498Z")
},
{
"email": "mail#mail.com",
"name": "AG2",
"phone": "00",
"crashNotificationEnabled": false,
"locationShared": false,
"creationDate": ISODate("2020-10-19T15:19:04.498Z")
}
],
"creationDate": ISODate("2020-10-19T15:19:04.498Z"),
"_class": ".model.UserContacts"
}
And i would like to iterate through all documents to check if either crashNotificationEnabled or locationShared is true and add +1 to a counter if its the case, im quite new to python and mongosql so i actually have a hard time trying to do that, i tried a lot of things but there is my last try :
def users_with_guardian_angel(mongoclient):
try:
mydb = mongoclient["main"]
userContacts = mydb["userContacts"]
users = userContacts.find()
for user in users:
result = userContacts.find_one({contacts : { $in: [true]}})
if result:
count_users = count_users + 1
print(f"{count_users} have at least one notificiation enabled")
But the result variable stays empty all the time, so if somebody could help me to accomplish what i want to do and tell what i did wrong here ?
Thanks !
Here's one way you could do it by letting the MongoDB server do all the work.
N.B.: This doesn't consider the possibility of multiple entries of the same user.
db.userContacts.aggregate([
{
"$unwind": "$contacts"
},
{
"$match": {
"$expr": {
"$or": [
"$contacts.crashNotificationEnabled",
"$contacts.locationShared"
]
}
}
},
{
"$count": "userCountWithNotificationsEnabled"
}
])
Try it on mongoplayground.net.
Example output:
[
{
"userCountWithNotificationsEnabled": 436
}
]
I'm a beginner in Elasticsearch and Python and I have an index created in Elasticsearch with some data, and I want to perform a query request on those data with python. This is my data mapping created in Kibana's Dev tools:
PUT /main-news-test-data
{
"mappings": {
"properties": {
"content": {
"type": "text"
},
"title": {
"type": "text"
},
"lead": {
"type": "text"
},
"agency": {
"type": "keyword"
},
"date_created": {
"type": "date"
},
"url": {
"type": "keyword"
},
"image": {
"type": "keyword"
},
"category": {
"type": "keyword"
},
"id":{
"type": "keyword"
}
}
}
}
and here is my Python code, in which we give it a keyword and a category number and it has to check in title, lead and content fields of the elastic data for the matching keyword and also check the entered category number with the data category number and return/print out any object that matches this criteria:
from elasticsearch import Elasticsearch
import json,requests
es = Elasticsearch(HOST="http://localhost", PORT=9200)
es = Elasticsearch()
def QueryMaker (keyword,category):
response = es.search(index="main-news-test-data",body={"from":0,"size":5,"query":{"multi_match":{
"content":keyword,"category":category,"title":keyword,"lead":keyword}}})
return(response)
if __name__ == '__main__':
keyword = input('Enter Keyword: ')
category = input('Enter Category: ')
#startDate = input('Enter StartDate: ')
#endDate = input('Enter EndDate: ')
data = QueryMaker(keyword,category)
print(data)
but I receive this error when I give the data to the input:
elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', '[multi_match] query does not support [content]')
What am I doing wrong?
Edit: the keyword has to be included in the title, lead and content but it doesn't have to be the same as them
Your multi_match query syntax is wrong here, also I think you need something like this, See more: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html
{
"from":0,
"size":5,
"query": {
"bool": {
"should": [
{
"multi_match" : {
"query": keyword,
"fields": [ "content", "title","lead" ]
}
},
{
"multi_match" : {
"query": category,
"fields": [ "category" ]
}
}
]
}
}
}
I am using a site's REST API's and have been primarily using Python's 'requests' module to GET json responses. The goal of the GET requests are to ultimately pull a user's form response which ends up being a complex json document. To deal with this:
user_form_submission = requests.get('https://www.url/doc.json',
auth = (api_key, secret),
params = params)
python_obj = json.loads(user_form_submission.text)
trimmed_dict = python_obj['key'][0]['keys']
For context, this is what trimmed_dict would look like formatted as .json:
{
"Date": { "value": "2020-04-26", "type": "date" },
"Location": {
"value": "Test ",
"type": "text",
"geostamp": "lat=34.00000, long=-77.00000, alt=17.986118, hAccuracy=65.000000, vAccuracy=10.000000, timestamp=2020-04-26T23:39:56Z"
},
"form": {
"value": [
{
"form_Details": {
"value": [
{
"code": {
"value": "0000000000",
"type": "barcode"
},
"Name": { "value": "bob", "type": "text" }
}
],
"type": "group"
},
"Subtotal": { "value": "4", "type": "decimal" },
"form_detail2": {
"value": [
{
"name": {
"value": "billy",
"type": "text"
},
"code": {
"value": "00101001",
"type": "barcode"
},
"Classification": {
"value": "person",
"type": "select1"
},
"Start_Time": { "value": "19:43:00", "type": "time" },
"time": { "value": "4", "type": "decimal" }
}
],
"type": "subform"}
}
]
}
}
Now I have a portion of the json that contains both the useful and useless. From this point, can I pass this obj in a POST? I've tried every way that I can think of approaching it, and have been shut down.
Understanding how I want to go about this, this is how I thought it would go:
json_post = requests.post(' https://url/api/doc.json',
auth = (api_key, secret),
json = {
"form_id" : 'https://url.form.com/formid',
'payload':{
json.dumps(trimmed_dict)
}})
But, when I do this, I get the following error --
TypeError: Object of type set is not JSON serializable
How can I push this dict through this POST? If there's a more effective way of going about it, I am very open to suggestion.
Try removing the curly braces around json.dumps(trimmed_dict). json.dumps turns your trimmed_dict into a string, which becomes a python set when surrounded with braces.
Additionally you could remove json.dumps and plug the trimmed_dict into the structure directly as the value associated with payload.
Remove the extra {} from the payload. payload itself is a key and json.dumps(trimmed_dict) as a value is enough
json_post = requests.post(' https://url/api/doc.json',
auth = (api_key, secret),
json = {
"form_id" : 'https://url.form.com/formid',
"payload": json.dumps(trimmed_dict)
})
I am looking to add validation to my api endpoint using Marshmallow.
I am running into the problem of how to get this chunk properly validated. The end goal is to make sure impressions is a positive number.
I would greatly appreciate any help or insight you can provide. First time using Marshmallow.
Sample Json:
{
"mode": [
{
"type": "String",
"values": {
"visits": 1000,
"budget": 400
},
"active": true
}
]
}
Sample code attempting to validate
class ValidateValues(BaseSchema):
visits = fields.Int(allow_none=True, validate=[validate.Range(min=0, error="Value must be greater than 0")])
budget = fields.Int(allow_none=True, validate=[validate.Range(min=0, error="Value must be greater than 0")])
class ModeSchema(BaseSchema):
type = fields.String(required=True)
active = fields.Boolean(required=True)
values = fields.Nested(ValidateValues)
class JsonSchema(BaseSchema):
mode = fields.List(fields.Dict(fields.Nested(ModeSchema, many=True)))
Current result
{
"mode": {
"0": {
"type": {
"key": [
"Invalid type."
]
},
"values": {
"key": [
"Invalid type."
]
},
"active": {
"key": [
"Invalid type."
]
}
}
}
}
You're just using a list of Nested fields. No need for Dict, here.
And no need for many=True since you're putting the Nested field in a List field.
Try this:
class JsonSchema(BaseSchema):
mode = fields.List(fields.Nested(ModeSchema))