Develop Reference

Make Pydantic BaseModel fields optional including sub-models for PATCH

As already asked in similar questions, I want to support PATCH operations for a FastApi application where the caller can specify as many or as few fields as they like, of a Pydantic BaseModel with sub-models, so that efficient PATCH operations can be performed, without the caller having to supply an entire valid model just in order to update two or three of the fields.
I've discovered there are 2 steps in Pydantic PATCH from the tutorial that don't support sub-models. However, Pydantic is far too good for me to criticise it for something that it seems can be built using the tools that Pydantic provides. This question is to request implementation of those 2 things while also supporting sub-models:
generate a new DRY BaseModel with all fields optional
implement deep copy with update of BaseModel
These problems are already recognised by Pydantic.
There is discussion of a class based solution to the optional model
And there two issues open on the deep copy with update
A similar question has been asked one or two times here on SO and there are some great answers with different approaches to generating an all-fields optional version of the nested BaseModel. After considering them all this particular answer by Ziur Olpa seemed to me to be the best, providing a function that takes the existing model with optional and mandatory fields, and returning a new model with all fields optional: https://stackoverflow.com/a/72365032
The beauty of this approach is that you can hide the (actually quite compact) little function in a library and just use it as a dependency so that it appears in-line in the path operation function and there's no other code or boilerplate.
But the implementation provided in the previous answer did not take the step of dealing with sub-objects in the BaseModel being patched.
This question therefore requests an improved implementation of the all-fields-optional function that also deals with sub-objects, as well as a deep copy with update.
I have a simple example as a demonstration of this use-case, which although aiming to be simple for demonstration purposes, also includes a number of fields to more closely reflect the real world examples we see. Hopefully this example provides a test scenario for implementations, saving work:
import logging
from datetime import datetime, date
from collections import defaultdict
from pydantic import BaseModel
from fastapi import FastAPI, HTTPException, status, Depends
from fastapi.encoders import jsonable_encoder
app = FastAPI(title="PATCH demo")
logging.basicConfig(level=logging.DEBUG)
class Collection:
collection = defaultdict(dict)
def __init__(self, this, that):
logging.debug("-".join((this, that)))
self.this = this
self.that = that
def get_document(self):
document = self.collection[self.this].get(self.that)
if not document:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Not Found",
)
logging.debug(document)
return document
def save_document(self, document):
logging.debug(document)
self.collection[self.this][self.that] = document
return document
class SubOne(BaseModel):
original: date
verified: str = ""
source: str = ""
incurred: str = ""
reason: str = ""
attachments: list[str] = []
class SubTwo(BaseModel):
this: str
that: str
amount: float
plan_code: str = ""
plan_name: str = ""
plan_type: str = ""
meta_a: str = ""
meta_b: str = ""
meta_c: str = ""
class Document(BaseModel):
this: str
that: str
created: datetime
updated: datetime
sub_one: SubOne
sub_two: SubTwo
the_code: str = ""
the_status: str = ""
the_type: str = ""
phase: str = ""
process: str = ""
option: str = ""
#app.get("/endpoint/{this}/{that}", response_model=Document)
async def get_submission(this: str, that: str) -> Document:
collection = Collection(this=this, that=that)
return collection.get_document()
#app.put("/endpoint/{this}/{that}", response_model=Document)
async def put_submission(this: str, that: str, document: Document) -> Document:
collection = Collection(this=this, that=that)
return collection.save_document(jsonable_encoder(document))
#app.patch("/endpoint/{this}/{that}", response_model=Document)
async def patch_submission(
document: Document,
# document: optional(Document), # <<< IMPLEMENT optional <<<
this: str,
that: str,
) -> Document:
collection = Collection(this=this, that=that)
existing = collection.get_document()
existing = Document(**existing)
update = document.dict(exclude_unset=True)
updated = existing.copy(update=update, deep=True) # <<< FIX THIS <<<
updated = jsonable_encoder(updated)
collection.save_document(updated)
return updated
This example is a working FastAPI application, following the tutorial, and can be run with uvicorn example:app --reload. Except it doesn't work, because there's no all-optional fields model, and Pydantic's deep copy with update actually overwrites sub-models rather than updating them.
In order to test it the following Bash script can be used to run curl requests. Again I'm supplying this just to hopefully make it easier to get started with this question.
Just comment out the other commands each time you run it so that the command you want is used.
To demonstrate this initial state of the example app working you would run GET (expect 404), PUT (document stored), GET (expect 200 and same document returned), PATCH (expect 200), GET (expect 200 and updated document returned).
host='http://127.0.0.1:8000'
path="/endpoint/A123/B456"
method='PUT'
data='
{
"this":"A123",
"that":"B456",
"created":"2022-12-01T01:02:03.456",
"updated":"2023-01-01T01:02:03.456",
"sub_one":{"original":"2022-12-12","verified":"Y"},
"sub_two":{"this":"A123","that":"B456","amount":0.88,"plan_code":"HELLO"},
"the_code":"BYE"}
'
# method='PATCH'
# data='{"this":"A123","that":"B456","created":"2022-12-01T01:02:03.456","updated":"2023-01-02T03:04:05.678","sub_one":{"original":"2022-12-12","verified":"N"},"sub_two":{"this":"A123","that":"B456","amount":123.456}}'
method='GET'
data=''
if [[ -n data ]]; then data=" --data '$data'"; fi
curl="curl -K curlrc -X $method '$host$path' $data"
echo $curl >&2
eval $curl
This curlrc will need to be co-located to ensure the content type headers are correct:
--cookie "_cookies"
--cookie-jar "_cookies"
--header "Content-Type: application/json"
--header "Accept: application/json"
--header "Accept-Encoding: compress, gzip"
--header "Cache-Control: no-cache"
So what I'm looking for is the implementation of optional that is commented out in the code, and a fix for existing.copy with the update parameter, that will enable this example to be used with PATCH calls that omit otherwise mandatory fields.
The implementation does not have to conform precisely to the commented out line, I just provided that based on Ziur Olpa's previous answer.

When I first posed this question I thought that the only problem was how to turn all fields Optional in a nested BaseModel, but actually that was not difficult to fix.
The real problem with partial updates when implementing a PATCH call is that the Pydantic BaseModel.copy method doesn't attempt to support nested models when applying it's update parameter. That's quite an involved task for the generic case, considering you may have fields that are dicts, lists, or sets of another BaseModel, just for instance. Instead it just unpacks the dict using **: https://github.com/pydantic/pydantic/blob/main/pydantic/main.py#L353
I haven't got a proper implementation of that for Pydantic, but since I've got a working example PATCH by cheating, I'm going to post this as an answer and see if anyone can fault it or provide better, possibly even with an implementation of BaseModel.copy that supports updates for nested models.
Rather than post the implementations separately I am going to update the example given in the question so that it has a working PATCH and being a full demonstration of PATCH hopefully this will help others more.
The two additions are partial and merge. partial is what's referred to as optional in the question code.
partial:
This is a function that takes any BaseModel and returns a new BaseModel with all fields Optional, including sub-object fields. That's enough for Pydantic to allow through any sub-set of fields without throwing an error for "missing fields". It's recursive - not really popular - but given these are nested data models the depth is not expected to exceed single digits.
merge:
The BaseModel update on copy method operates on an instance of BaseModel - but supporting all the possible type variations when descending through a nested model is the hard part - and the database data, and the incoming update, are easily available as plain Python dicts; so this is the cheat: merge is an implementation of a nested dict update instead, and since the dict data has already been validated at one point or other, it should be fine.
Here's the full example solution:
import logging
from typing import Optional, Type
from datetime import datetime, date
from functools import lru_cache
from pydantic import BaseModel, create_model
from collections import defaultdict
from pydantic import BaseModel
from fastapi import FastAPI, HTTPException, status, Depends, Body
from fastapi.encoders import jsonable_encoder
app = FastAPI(title="Nested model PATCH demo")
logging.basicConfig(level=logging.DEBUG)
class Collection:
collection = defaultdict(dict)
def __init__(self, this, that):
logging.debug("-".join((this, that)))
self.this = this
self.that = that
def get_document(self):
document = self.collection[self.this].get(self.that)
if not document:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Not Found",
)
logging.debug(document)
return document
def save_document(self, document):
logging.debug(document)
self.collection[self.this][self.that] = document
return document
class SubOne(BaseModel):
original: date
verified: str = ""
source: str = ""
incurred: str = ""
reason: str = ""
attachments: list[str] = []
class SubTwo(BaseModel):
this: str
that: str
amount: float
plan_code: str = ""
plan_name: str = ""
plan_type: str = ""
meta_a: str = ""
meta_b: str = ""
meta_c: str = ""
class SubThree(BaseModel):
one: str = ""
two: str = ""
class Document(BaseModel):
this: str
that: str
created: datetime
updated: datetime
sub_one: SubOne
sub_two: SubTwo
# sub_three: dict[str, SubThree] = {} # Hah hah not really
the_code: str = ""
the_status: str = ""
the_type: str = ""
phase: str = ""
process: str = ""
option: str = ""
#lru_cache
def partial(baseclass: Type[BaseModel]) -> Type[BaseModel]:
"""Make all fields in supplied Pydantic BaseModel Optional, for use in PATCH calls.
Iterate over fields of baseclass, descend into sub-classes, convert fields to Optional and return new model.
Cache newly created model with lru_cache to ensure it's only created once.
Use with Body to generate the partial model on the fly, in the PATCH path operation function.
- https://stackoverflow.com/questions/75167317/make-pydantic-basemodel-fields-optional-including-sub-models-for-patch
- https://stackoverflow.com/questions/67699451/make-every-fields-as-optional-with-pydantic
- https://github.com/pydantic/pydantic/discussions/3089
- https://fastapi.tiangolo.com/tutorial/body-updates/#partial-updates-with-patch
"""
fields = {}
for name, field in baseclass.__fields__.items():
type_ = field.type_
if type_.__base__ is BaseModel:
fields[name] = (Optional[partial(type_)], {})
else:
fields[name] = (Optional[type_], None) if field.required else (type_, field.default)
# https://docs.pydantic.dev/usage/models/#dynamic-model-creation
validators = {"__validators__": baseclass.__validators__}
return create_model(baseclass.__name__ + "Partial", **fields, __validators__=validators)
def merge(original, update):
"""Update original nested dict with values from update retaining original values that are missing in update.
- https://github.com/pydantic/pydantic/issues/3785
- https://github.com/pydantic/pydantic/issues/4177
- https://docs.pydantic.dev/usage/exporting_models/#modelcopy
- https://github.com/pydantic/pydantic/blob/main/pydantic/main.py#L353
"""
for key in update:
if key in original:
if isinstance(original[key], dict) and isinstance(update[key], dict):
merge(original[key], update[key])
elif isinstance(original[key], list) and isinstance(update[key], list):
original[key].extend(update[key])
else:
original[key] = update[key]
else:
original[key] = update[key]
return original
#app.get("/endpoint/{this}/{that}", response_model=Document)
async def get_submission(this: str, that: str) -> Document:
collection = Collection(this=this, that=that)
return collection.get_document()
#app.put("/endpoint/{this}/{that}", response_model=Document)
async def put_submission(this: str, that: str, document: Document) -> Document:
collection = Collection(this=this, that=that)
return collection.save_document(jsonable_encoder(document))
#app.patch("/endpoint/{this}/{that}", response_model=Document)
async def patch_submission(
this: str,
that: str,
document: partial(Document), # <<< IMPLEMENTED partial TO MAKE ALL FIELDS Optional <<<
) -> Document:
collection = Collection(this=this, that=that)
existing_document = collection.get_document()
incoming_document = document.dict(exclude_unset=True)
# VVV IMPLEMENTED merge INSTEAD OF USING BROKEN PYDANTIC copy WITH update VVV
updated_document = jsonable_encoder(merge(existing_document, incoming_document))
collection.save_document(updated_document)
return updated_document

FastApi urls path and 2 values into a query

im using FastApi and get some troubles with url.
i have a root url
#app.get("/myurl")
http://host/myurl
and
http://host/myurl?id=2
and here function returns all info from needed table.
on url like http://host/myurl?id=2&type=3 i need to get another query from table. how i need to create function because now this http://host/myurl?id=2 overlapping this function http://host/myurl?id=2&type=3
how i can use multiple urls with different values in it in fastapi?
and i want to know how to make url like http://host/myurl?id=2&type=3,2 to return result from table for two types (query example is SELECT * from mytable WHERE id=%(id)s and type IN (1,2) but type IN (,) should be parameters which i need to inpout

how i can use multiple urls with different values in it in fastapi?
As far as I know, you can't. But fortunately, you don't need to. What you can do is define only one route ("/myurl") with both parameters id and type, and set the second as optional.
Then, if you don't receive type, you process a different query.
By the way, don't use id and type as parameter names, that will mess with the name of the in-built function id()and type().
Here a working example:
from fastapi import FastAPI, Query
app = FastAPI()
#app.get("/myurl")
async def my_url(my_id: int = Query(...), my_type: int = Query(None)):
if my_type:
return f"You gave an id ({my_id}) and a type ({my_type})."
return f"You gave only an id ({my_id}) but no type."
i want to know how to make url like http://host/myurl?id=2&type=3,2
Not sure you can do it at all. What you can do is add the type parameter several times, like this:
http://host/myurl?my_id=2&my_type=3&my_type=2
In this case, you need to slightly change your code:
from fastapi import FastAPI, Query
from typing import List
app = FastAPI()
#app.get("/myurl")
async def my_url(my_id: int = Query(...), my_type: List[int] = Query(None)):
if my_type:
if len(my_type) > 1:
return f"You gave an id ({my_id}) and a list of types ({my_type})."
else:
return f"You gave an id ({my_id}) and a type ({my_type})."
return f"You gave only an id ({my_id}) but no type."
Note that you'll always receive my_type as a list then, even if you pass it only one time.

Using python and suds, data not read by server side because element is not defined as an array

I am a very inexperienced programmer with no formal education. Details will be extremely helpful in any responses.
I have made several basic python scripts to call SOAP APIs, but I am running into an issue with a specific API function that has an embedded array.
Here is a sample excerpt from a working XML format to show nested data:
<bomData xsi:type="urn:inputBOM" SOAP-ENC:arrayType="urn:bomItem[]">
<bomItem>
<item_partnum></item_partnum>
<item_partrev></item_partrev>
<item_serial></item_serial>
<item_lotnum></item_lotnum>
<item_sublotnum></item_sublotnum>
<item_qty></item_qty>
</bomItem>
<bomItem>
<item_partnum></item_partnum>
<item_partrev></item_partrev>
<item_serial></item_serial>
<item_lotnum></item_lotnum>
<item_sublotnum></item_sublotnum>
<item_qty></item_qty>
</bomItem>
</bomData>
I have tried 3 different things to get this to work to no avail.
I can generate the near exact XML from my script, but a key attribute missing is the 'SOAP-ENC:arrayType="urn:bomItem[]"' in the above XML example.
Option 1 was using MessagePlugin, but I get an error because my section is like the 3 element and it always injects into the first element. I have tried body[2], but this throws an error.
Option 2 I am trying to create the object(?). I read a lot of stack overflow, but I might be missing something for this.
Option 3 looked simple enough, but also failed. I tried setting the values in the JSON directly. I got these examples by an XML sample to JSON.
I have also done a several other minor things to try to get it working, but not worth mentioning. Although, if there is a way to somehow do the following, then I'm all ears:
bomItem[]: bomData = {"bomItem"[{...,...,...}]}
Here is a sample of my script:
# for python 3
# using pip install suds-py3
from suds.client import Client
from suds.plugin import MessagePlugin
# Config
#option 1: trying to set it as an array using plugin
class MyPlugin(MessagePlugin):
def marshalled(self, context):
body = context.envelope.getChild('Body')
bomItem = body[0]
bomItem.set('SOAP-ENC:arrayType', 'urn:bomItem[]')
URL = "http://localhost/application/soap?wsdl"
client = Client(URL, plugins=[MyPlugin()])
transact_info = {
"username":"",
"transaction":"",
"workorder":"",
"serial":"",
"trans_qty":"",
"seqnum":"",
"opcode":"",
"warehouseloc":"",
"warehousebin":"",
"machine_id":"",
"comment":"",
"defect_code":""
}
#WIP - trying to get bomData below working first
inputData = {
"dataItem":[
{
"fieldname": "",
"fielddata": ""
}
]
}
#option 2: trying to create the element here and define as an array
#inputbom = client.factory.create('ns3:inputBOM')
#inputbom._type = "SOAP-ENC:arrayType"
#inputbom.value = "urn:bomItem[]"
bomData = {
#Option 3: trying to set the time and array type in JSON
#"#xsi:type":"urn:inputBOM",
#"#SOAP-ENC:arrayType":"urn:bomItem[]",
"bomItem":[
{
"item_partnum":"",
"item_partrev":"",
"item_serial":"",
"item_lotnum":"",
"item_sublotnum":"",
"item_qty":""
},
{
"item_partnum":"",
"item_partrev":"",
"item_serial":"",
"item_lotnum":"",
"item_sublotnum":"",
"item_qty":""
}
]
}
try:
response = client.service.transactUnit(transact_info,inputData,bomData)
print("RESPONSE: ")
print(response)
#print(client)
#print(envelope)
except Exception as e:
#handle error here
print(e)
I appreciate any help and hope it is easy to solve.

I have found the answer I was looking for. At least a working solution.
In any case, option 1 worked out. I read up on it at the following link:
https://suds-py3.readthedocs.io/en/latest/
You can review at the '!MessagePlugin' section.
I found a solution to get message plugin working from the following post:
unmarshalling Error: For input string: ""
A user posted an example how to crawl through the XML structure and modify it.
Here is my modified example to get my script working:
#Using MessagePlugin to modify elements before sending to server
class MyPlugin(MessagePlugin):
# created method that could be reused to modify sections with similar
# structure/requirements
def addArrayType(self, dataType, arrayType, transactUnit):
# this is the code that is key to crawling through the XML - I get
# the child of each parent element until I am at the right level for
# modification
data = transactUnit.getChild(dataType)
if data:
data.set('SOAP-ENC:arrayType', arrayType)
def marshalled(self, context):
# Alter the envelope so that the xsd namespace is allowed
context.envelope.nsprefixes['xsd'] = 'http://www.w3.org/2001/XMLSchema'
body = context.envelope.getChild('Body')
transactUnit = body.getChild("transactUnit")
if transactUnit:
self.addArrayType('inputData', 'urn:dataItem[]', transactUnit)
self.addArrayType('bomData', 'urn:bomItem[]', transactUnit)

How to chain validations with pydantic

Let's say I have webhook where I get json data. This json is recursively converted by pydantic.
#app.route("/", methods=['POST'])
async def telegram_webhook(request):
update = Update.parse_obj(request.json)
/* do something with update */
I check this json is minimal valid object with Update model (which internally contains Message model):
class Update(BaseModel):
update_id: int
message: Message
...
class Message(BaseModel):
message_id: int
text: Optional[str]
But later in the code I want to extend validation, so to check that message is not only Message, but TextMessage:
// text field now is required
class TextMessage(Message):
text: str
#validator('text')
def check_text_length(cls, value):
length = len(value)
if length > 4096:
raise ValueError(f'text length {length} is too large')
return value
So I pass message to validation function
def process_text_message(message):
text_message = TextMessage.parse_obj(message)
But I get error that pydantic requires not Message type, but dict.
How would I do that?
How could I apply additional validation on already validated (basically) data?

The short answer is: use message.dict():
def process_text_message(message):
text_message = TextMessage.parse_obj(message.dict())
The longer answer is that parse_obj should be fixed to cope with "dict-like" things not just dicts, I'll explain that on the issues you created.

Verify Tornado arguments

I have a server which can accept 0, 1 or many of the following url arguments:
/api/cases?id={id}&name={name}&owner={owner}&status={status}
So these, amongst other, are correct:
/api/cases?owner=me
/api/cases
/api/cases?name=bob&status=waiting
Currently, my code looks like this
routes = [(r'/cases?([^/]+)', MyHandler)]
tornado.web.Application.__init__(self, routes, settings={})
class MyHandler(APIHandler):
ACCEPTED_URL_ARGS = ["id", "name", "owner", "status"]
def get(self, i):
for key in self.request.arguments:
if key not in self.ACCEPTED_URL_ARGS:
# error
Is there a better way to check for the url arguments?

What you have is correct. In Tornado there is no other way to verify that you only got the arguments that you expect than to iterate over self.request.arguments.

As Ben says, this is the correct way to do it in Tornado. That being said, the better place for your test would be the prepare method; also, a strictly more "pythonic" approach would be to use sets:
class MyHandler(APIHandler):
ACCEPTED_URL_ARGS = {"id", "name", "owner", "status"}
def prepare(self):
unwanted_args = self.ACCEPTED_URL_ARGS - set(self.request.arguments)
if unwanted_args:
# error