I have a pydantic object definition that includes an optional field. I am looking to be able to configure the field to only be serialised if it is not None.
class MyObject(BaseModel):
id: str
msg: Optional[str] = None
pri: Optional[int] = None
MyObject(id="123").json() # ideal output: {"id": "123", "pri": null}
MyObject(id="123", msg="hello").json() # ideal output: {"id": "123", "msg": "hello", "pri": null}
I would like to be able to specify the field precisely, as this object will be nested, and there are other optional fields that should be returned, regardless of whether they are None or not.
The solution to set json option exclude_none to True won't work for this purpose.
you can't do such thing with pydantic and even with more powerfull lib like attrs. The why may be because it is not a good way of returning json object, it is realy confusing for you, the api client and your test suite.
you may get some inspiration from elegant-way-to-remove-fields-from-nested-dictionaries.
you would be able to achieve something (not recommanded at all) by parsing your object jsoned and remove fiels folowing a logic.
exemple of key/value manipulation in nested dict:
import re
def dict_key_convertor(dictionary):
"""
Convert a dictionary from CamelCase to snake_case
:param dictionary: the dictionary given
:return: return a dict
"""
if not isinstance(dictionary, (dict, list)):
return dictionary
if isinstance(dictionary, list):
return [dict_key_convertor(elem) for elem in dictionary]
return {to_snake(key): dict_key_convertor(data) for key, data in dictionary.items()}
def to_snake(word) -> str:
"""
Convert all word from camel to snake case
:param word: the word given to be change from camelCase to snake_case
:return: return word variable in snake_case
"""
return re.sub(r'([A-Z]{2,}(?=[a-z]))', '\\1_', re.sub(r'([a-z])([A-Z]+)', '\\1_\\2', word)).lower()
with a bit of work you may achive something with this:
from typing import List
def dict_key_cleaner(dictionary):
if not isinstance(dictionary, (dict, list)):
return dictionary
if isinstance(dictionary, list):
return [dict_key_cleaner(elem) for elem in dictionary]
# change this return to work with dict
return {poper(key, dictionary): dict_key_cleaner(data) for key, data in dictionary.items()}
def poper(key, dictionary):
special_keys: List[str] = ["field_name","field_name1","field_name2"]
# do some stuff here
for spe_key in special_keys:
if key == spe_key and key.key_value is None:
dictionary.pop(key)
# add return of modified dict
Related
Ok I'm trying to define a dataclass to enqueue a job in redis for a sidekiq worker, the specification of the sidekiq payload requires some attributes something with this format:
{
"class": "SomeWorker",
"queue": "default"
"jid": "b4a577edbccf1d805744efa9", // 12-byte random number as 24 char hex string
"args": [......],
"created_at": 1234567890,
"enqueued_at": 1234567890
}
So I define a dataclass in my python code:
#dataclass
class PusherNotificationJob:
args: Any = None
queue: str = "default"
jid: str = secrets.token_hex(12)
retry: bool = True
created_at: datetime.datetime = time.time()
enqueued_at: datetime.datetime = time.time()
def asdict(self):
return {** self.__dict__, "class": "SomeWorker"}
My problem is that I can't define "class" as an attribute of PusherNotificationJob because it's a reserved word. So I need to define the asdict method to serialize as a dict and add the "class" attribute I added here.
There is a better way to do this?
A few things to pay attention before the answer: created_at and enqueued_at will have the same value for as long as your code is running... if you want the time to be when the instance was created, use:
created_at: datetime.datetime = field(default_factory=time.time)
moving on:
You can override __iter__ to use dict(instance). It would be something like below if you have an attribute named class_: str = 'SomeWorker'
def __iter__(self):
for key in self.__dict__:
if key == 'class_':
key = 'class'
yield key, self.__dict__
If you need __iter__ to behave normally, you use the following:
def convert_keys(self):
for key in self.__dict__:
value = self.__dict__[key]
if key == 'class_':
key = 'class'
yield key, value
def asdict(self):
return {key: value for key, value in self.convert_keys()}
I created a dict source = {'livemode': False}. I thought it's possible to access the livemode value via source.livemode. But it doesn't work. Is there a way to access it that way?
As a not source['livemode'] works, but I need source.livemode as that's already used in my code and I have to handle it as an alternative to the Stripe return value charge.
I want to give a bit more context
Here I create a charge via Stripe:
def _create_charge(self, request, order_reference, order_items_dict, token):
try:
charge = stripe.Charge.create(
amount=order_items_dict['total_gross'],
application_fee=order_items_dict['application_fee'],
currency=order_items_dict['event'].currency,
source=token,
stripe_account=order_items_dict['event'].organizer.stripe_account,
expand=['balance_transaction', 'application_fee'],
)
except stripe.error.StripeError as e:
body = e.json_body
err = body.get('error', {})
messages.error(
request,
err.get('message')
)
else:
if charge.paid and charge.status == 'succeeded':
return charge
I can access this with e.g. charge_or_source.livemode
def _create_order(self, request, charge_or_source, order_status):
order_reference = request.session.get('order_reference')
new_order = self.order_form.save(commit=False)
print(charge_or_source.livemode, "charge_or_source.livemode")
new_order_dict = {
'total_gross': self.order_items_dict['total_gross'],
'livemode': charge_or_source.livemode,
}
Now there is a case (when the order is Free) where I have to 'skip' the _create_charge function but still, I have to send information about charge_or_source.livemode. Therefore I tried to create the above-mentioned dictionary.
You can implement a custom dict wrapper (either a subclass of dict or something that contains a dict) and implement __getattr__ (or __getattribute__) to return data from the dict.
class DictObject(object):
def __init__(self, data):
self.mydict = data
def __getattr__(self, attr):
if attr in self.mydict: return self.mydict[attr]
return super(self, DictObject).__getattr__(attr)
I'm a beginner myself, but let me try and answer:
Say you have a dictionary:
dictionary = {"One": 1, "Two": 2, "Three": 3}
You can create a class with its keys like:
class DictKeys:
One = 'One'
Two = 'Two'
Three = 'Three'
Here, One, Two and Three are class variables or attributes, which means if you create an object for this class:
key = DictKeys()
You can access all of those keys using the '.' (dot) operator.
key.One
>>'One'
Now just plug it where ever you want to access your dictionary!
dictionary[key.One]
>>1
I'm sure this isn't the best way, and class access is a tiny bit slower than dict access, but if you really want to, you can access all your keys with a dot using this method.
The correct way to access a dictionary is how you proposed it:
source['livemode']
i am trying convert json string to model
then it is easy to get value with .
i have checked another question
but different, my json sting looks like,
{
"id":"123",
"name":"name",
"key":{
"id":"345",
"des":"des"
},
}
i prefer to use 2 class like,
class A:
id = ''
name = ''
key = new B()
class B:
id = ''
des = ''
There are few libraries that might help:
marshmallow is nice
colander from Pylons
schematics
For easier cases you can also use something from standard library like
named tuples and one from collections which is available also in py2
SimpleNamespace
In order to do that you should provide your custom callback as an object_hook argument to the json.loads function.
object_hook is an optional function that will be called with the
result of any object literal decode (a dict). The return value of
object_hook will be used instead of the dict. This feature
can be used to implement custom decoders (e.g. JSON-RPC class hinting).
Consider using collections.namestuple subclasses:
json_str = '''
{
"id":"123",
"name":"name",
"key":{
"id":"345",
"des":"des"
}
}'''
B = collections.namedtuple('B', 'id des')
A = collections.namedtuple('A', 'id name key')
def make_models(o):
if 'key' in o:
return A(o['id'], o['name'], B(id=o['key']['id'], des=o['key']['des']))
else:
return o
result = json.loads(json_str, object_hook=make_models)
print(type(result)) # outputs: <class '__main__.A'>
print(result.id) # outputs: 123
print(result.key.id) # outputs: 345
How do I serialise a Python Enum member to JSON, so that I can deserialise the resulting JSON back into a Python object?
For example, this code:
from enum import Enum
import json
class Status(Enum):
success = 0
json.dumps(Status.success)
results in the error:
TypeError: <Status.success: 0> is not JSON serializable
How can I avoid that?
I know this is old but I feel this will help people. I just went through this exact problem and discovered if you're using string enums, declaring your enums as a subclass of str works well for almost all situations:
import json
from enum import Enum
class LogLevel(str, Enum):
DEBUG = 'DEBUG'
INFO = 'INFO'
print(LogLevel.DEBUG)
print(json.dumps(LogLevel.DEBUG))
print(json.loads('"DEBUG"'))
print(LogLevel('DEBUG'))
Will output:
LogLevel.DEBUG
"DEBUG"
DEBUG
LogLevel.DEBUG
As you can see, loading the JSON outputs the string DEBUG but it is easily castable back into a LogLevel object. A good option if you don't want to create a custom JSONEncoder.
The correct answer depends on what you intend to do with the serialized version.
If you are going to unserialize back into Python, see Zero's answer.
If your serialized version is going to another language then you probably want to use an IntEnum instead, which is automatically serialized as the corresponding integer:
from enum import IntEnum
import json
class Status(IntEnum):
success = 0
failure = 1
json.dumps(Status.success)
and this returns:
'0'
If you want to encode an arbitrary enum.Enum member to JSON and then decode
it as the same enum member (rather than simply the enum member's value attribute), you can do so by writing a custom JSONEncoder class, and a decoding function to pass as the object_hook argument to json.load() or json.loads():
PUBLIC_ENUMS = {
'Status': Status,
# ...
}
class EnumEncoder(json.JSONEncoder):
def default(self, obj):
if type(obj) in PUBLIC_ENUMS.values():
return {"__enum__": str(obj)}
return json.JSONEncoder.default(self, obj)
def as_enum(d):
if "__enum__" in d:
name, member = d["__enum__"].split(".")
return getattr(PUBLIC_ENUMS[name], member)
else:
return d
The as_enum function relies on the JSON having been encoded using EnumEncoder, or something which behaves identically to it.
The restriction to members of PUBLIC_ENUMS is necessary to avoid a maliciously crafted text being used to, for example, trick calling code into saving private information (e.g. a secret key used by the application) to an unrelated database field, from where it could then be exposed (see https://chat.stackoverflow.com/transcript/message/35999686#35999686).
Example usage:
>>> data = {
... "action": "frobnicate",
... "status": Status.success
... }
>>> text = json.dumps(data, cls=EnumEncoder)
>>> text
'{"status": {"__enum__": "Status.success"}, "action": "frobnicate"}'
>>> json.loads(text, object_hook=as_enum)
{'status': <Status.success: 0>, 'action': 'frobnicate'}
In Python >= 3.7, can just use
json.dumps(enum_obj, default=str)
If you want to use the enum value, you can do
json.dumps(enum_obj, default=lambda x: x.value)
or if you want to use the enum name,
json.dumps(enum_obj, default=lambda x: x.name)
You just need to inherit from str or int class:
from enum import Enum, unique
#unique
class StatusEnum(int, Enum):
pending: int = 11
approved: int = 15
declined: int = 266
That's it, it will be serialised using any JSON encoder.
I liked Zero Piraeus' answer, but modified it slightly for working with the API for Amazon Web Services (AWS) known as Boto.
class EnumEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, Enum):
return obj.name
return json.JSONEncoder.default(self, obj)
I then added this method to my data model:
def ToJson(self) -> str:
return json.dumps(self.__dict__, cls=EnumEncoder, indent=1, sort_keys=True)
I hope this helps someone.
If you are using jsonpickle the easiest way should look as below.
from enum import Enum
import jsonpickle
#jsonpickle.handlers.register(Enum, base=True)
class EnumHandler(jsonpickle.handlers.BaseHandler):
def flatten(self, obj, data):
return obj.value # Convert to json friendly format
if __name__ == '__main__':
class Status(Enum):
success = 0
error = 1
class SimpleClass:
pass
simple_class = SimpleClass()
simple_class.status = Status.success
json = jsonpickle.encode(simple_class, unpicklable=False)
print(json)
After Json serialization you will have as expected {"status": 0} instead of
{"status": {"__objclass__": {"py/type": "__main__.Status"}, "_name_": "success", "_value_": 0}}
You can even combine the solutions mentioned above with the automatic value creation for Enums. I use this in combination with Pydantic and FastAPI to provide lower case names for a REST API:
from enum import Enum, auto
import json
class StrEnum(str, Enum):
pass
# this creates nice lowercase and JSON serializable names
# https://docs.python.org/3/library/enum.html#using-automatic-values
class AutoNameLower(StrEnum):
def _generate_next_value_(name, start, count, last_values):
return name.lower()
class AutoNameLowerStrEnum(AutoNameLower):
pass
class MyActualEnum(AutoNameLowerStrEnum):
THIS = auto()
THAT = auto()
FOO = auto()
BAR = auto()
print(MyActualEnum.THIS)
print(json.dumps(MyActualEnum.THIS))
print(list(MyActualEnum))
Console:
>>> MyActualEnum.THIS
>>> "this"
>>> [<MyActualEnum.THIS: 'this'>, <MyActualEnum.THAT: 'that'>, <MyActualEnum.FOO: 'foo'>, <MyActualEnum.BAR: 'bar'>]
This worked for me:
class Status(Enum):
success = 0
def __json__(self):
return self.value
Didn't have to change anything else. Obviously, you'll only get the value out of this and will need to do some other work if you want to convert the serialized value back into the enum later.
For a custom object I am able to encode into json using JSONEncoder.
class CustomEncoder(JSONEncoder):
def encode(self, custom):
prop_dict = {}
for prop in Custom.all_properties_names():
if custom.__getattribute__(prop) is not None:
if prop is 'created_timestamp':
prop_dict.update({prop: custom.__getattribute__(
prop).isoformat()})
else:
prop_dict.update({prop: custom.__getattribute__(prop)})
return prop_dict
To generate json, I am using json.dumps(custom, cls=CustomEncoder, indent=True)
Now I have a list of Custom class objects. How do convert the list to json?
custom_list = //get custom object list from service
How do I convert the whole list to json? Do I need to iterate and capture json of each custom object and append to a list with comma separated? I feel like there should be something straightforward I am missing here.
The custom encoder is called only when needed. If you have a custom thing that the JSON library thinks it can encode, like a string or dictionary, the custom encoder won't be called. The following example shows that encoding an object, or a list including an object, works with a single custom encoder:
import json
class Custom(object):
pass
class CustomEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, Custom):
return 'TASTY'
return CustomEncoder(self, o)
print json.dumps( Custom(), cls=CustomEncoder )
print json.dumps( [1, [2,'three'], Custom()], cls=CustomEncoder )
Output:
"TASTY"
[1, [2, "three"], "TASTY"]
In my way, I convert object to dict then using json.dumps list of dict:
def custom_to_dict(custom):
return {
'att1': custom.att1,
'att2': custom.att2,
...
}
#custom_list is your list of customs
out = json.dumps([custom_to_dict(custom) for custom in custom_list])
It might be helpful