Related
I have a Decimal('3.9') as part of an object, and wish to encode this to a JSON string which should look like {'x': 3.9}. I don't care about precision on the client side, so a float is fine.
Is there a good way to serialize this? JSONDecoder doesn't accept Decimal objects, and converting to a float beforehand yields {'x': 3.8999999999999999} which is wrong, and will be a big waste of bandwidth.
Simplejson 2.1 and higher has native support for Decimal type:
>>> json.dumps(Decimal('3.9'), use_decimal=True)
'3.9'
Note that use_decimal is True by default:
def dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True,
allow_nan=True, cls=None, indent=None, separators=None,
encoding='utf-8', default=None, use_decimal=True,
namedtuple_as_object=True, tuple_as_array=True,
bigint_as_string=False, sort_keys=False, item_sort_key=None,
for_json=False, ignore_nan=False, **kw):
So:
>>> json.dumps(Decimal('3.9'))
'3.9'
Hopefully, this feature will be included in standard library.
I would like to let everyone know that I tried Michał Marczyk's answer on my web server that was running Python 2.6.5 and it worked fine. However, I upgraded to Python 2.7 and it stopped working. I tried to think of some sort of way to encode Decimal objects and this is what I came up with:
import decimal
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
return str(o)
return super(DecimalEncoder, self).default(o)
Note that this will convert the decimal to its string representation (e.g.; "1.2300") to a. not lose significant digits and b. prevent rounding errors.
This should hopefully help anyone who is having problems with Python 2.7. I tested it and it seems to work fine. If anyone notices any bugs in my solution or comes up with a better way, please let me know.
Usage example:
json.dumps({'x': decimal.Decimal('5.5')}, cls=DecimalEncoder)
How about subclassing json.JSONEncoder?
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
# wanted a simple yield str(o) in the next line,
# but that would mean a yield on the line with super(...),
# which wouldn't work (see my comment below), so...
return (str(o) for o in [o])
return super(DecimalEncoder, self).default(o)
Then use it like so:
json.dumps({'x': decimal.Decimal('5.5')}, cls=DecimalEncoder)
The native Django option is missing so I'll add it for the next guy/gall that looks for it.
Starting on Django 1.7.x there is a built-in DjangoJSONEncoder that you can get it from django.core.serializers.json.
import json
from django.core.serializers.json import DjangoJSONEncoder
from django.forms.models import model_to_dict
model_instance = YourModel.object.first()
model_dict = model_to_dict(model_instance)
json.dumps(model_dict, cls=DjangoJSONEncoder)
Presto!
In my Flask app, Which uses python 2.7.11, flask alchemy(with 'db.decimal' types), and Flask Marshmallow ( for 'instant' serializer and deserializer), i had this error, every time i did a GET or POST. The serializer and deserializer, failed to convert Decimal types into any JSON identifiable format.
I did a "pip install simplejson", then
Just by adding
import simplejson as json
the serializer and deserializer starts to purr again. I did nothing else...
DEciamls are displayed as '234.00' float format.
I tried switching from simplejson to builtin json for GAE 2.7, and had issues with the decimal. If default returned str(o) there were quotes (because _iterencode calls _iterencode on the results of default), and float(o) would remove trailing 0.
If default returns an object of a class that inherits from float (or anything that calls repr without additional formatting) and has a custom __repr__ method, it seems to work like I want it to.
import json
from decimal import Decimal
class fakefloat(float):
def __init__(self, value):
self._value = value
def __repr__(self):
return str(self._value)
def defaultencode(o):
if isinstance(o, Decimal):
# Subclass float with custom repr?
return fakefloat(o)
raise TypeError(repr(o) + " is not JSON serializable")
json.dumps([10.20, "10.20", Decimal('10.20')], default=defaultencode)
'[10.2, "10.20", 10.20]'
For Django users:
Recently came across TypeError: Decimal('2337.00') is not JSON serializable
while JSON encoding i.e. json.dumps(data)
Solution:
# converts Decimal, Datetime, UUIDs to str for Encoding
from django.core.serializers.json import DjangoJSONEncoder
json.dumps(response.data, cls=DjangoJSONEncoder)
But, now the Decimal value will be a string, now we can explicitly set the decimal/float value parser when decoding data, using parse_float option in json.loads:
import decimal
data = json.loads(data, parse_float=decimal.Decimal) # default is float(num_str)
3.9 can not be exactly represented in IEEE floats, it will always come as 3.8999999999999999, e.g. try print repr(3.9), you can read more about it here:
http://en.wikipedia.org/wiki/Floating_point
http://docs.sun.com/source/806-3568/ncg_goldberg.html
So if you don't want float, only option you have to send it as string, and to allow automatic conversion of decimal objects to JSON, do something like this:
import decimal
from django.utils import simplejson
def json_encode_decimal(obj):
if isinstance(obj, decimal.Decimal):
return str(obj)
raise TypeError(repr(obj) + " is not JSON serializable")
d = decimal.Decimal('3.5')
print simplejson.dumps([d], default=json_encode_decimal)
My $.02!
I extend a bunch of the JSON encoder since I am serializing tons of data for my web server. Here's some nice code. Note that it's easily extendable to pretty much any data format you feel like and will reproduce 3.9 as "thing": 3.9
JSONEncoder_olddefault = json.JSONEncoder.default
def JSONEncoder_newdefault(self, o):
if isinstance(o, UUID): return str(o)
if isinstance(o, datetime): return str(o)
if isinstance(o, time.struct_time): return datetime.fromtimestamp(time.mktime(o))
if isinstance(o, decimal.Decimal): return str(o)
return JSONEncoder_olddefault(self, o)
json.JSONEncoder.default = JSONEncoder_newdefault
Makes my life so much easier...
For those who don't want to use a third-party library... An issue with Elias Zamaria's answer is that it converts to float, which can run into problems. For example:
>>> json.dumps({'x': Decimal('0.0000001')}, cls=DecimalEncoder)
'{"x": 1e-07}'
>>> json.dumps({'x': Decimal('100000000000.01734')}, cls=DecimalEncoder)
'{"x": 100000000000.01733}'
The JSONEncoder.encode() method lets you return the literal json content, unlike JSONEncoder.default(), which has you return a json compatible type (like float) that then gets encoded in the normal way. The problem with encode() is that it (normally) only works at the top level. But it's still usable, with a little extra work (python 3.x):
import json
from collections.abc import Mapping, Iterable
from decimal import Decimal
class DecimalEncoder(json.JSONEncoder):
def encode(self, obj):
if isinstance(obj, Mapping):
return '{' + ', '.join(f'{self.encode(k)}: {self.encode(v)}' for (k, v) in obj.items()) + '}'
if isinstance(obj, Iterable) and (not isinstance(obj, str)):
return '[' + ', '.join(map(self.encode, obj)) + ']'
if isinstance(obj, Decimal):
return f'{obj.normalize():f}' # using normalize() gets rid of trailing 0s, using ':f' prevents scientific notation
return super().encode(obj)
Which gives you:
>>> json.dumps({'x': Decimal('0.0000001')}, cls=DecimalEncoder)
'{"x": 0.0000001}'
>>> json.dumps({'x': Decimal('100000000000.01734')}, cls=DecimalEncoder)
'{"x": 100000000000.01734}'
From the JSON Standard Document, as linked in json.org:
JSON is agnostic about the semantics of numbers. In any programming language, there can be a variety of
number types of various capacities and complements, fixed or floating, binary or decimal. That can make
interchange between different programming languages difficult. JSON instead offers only the representation of
numbers that humans use: a sequence of digits. All programming languages know how to make sense of digit
sequences even if they disagree on internal representations. That is enough to allow interchange.
So it's actually accurate to represent Decimals as numbers (rather than strings) in JSON. Bellow lies a possible solution to the problem.
Define a custom JSON encoder:
import json
class CustomJsonEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, Decimal):
return float(obj)
return super(CustomJsonEncoder, self).default(obj)
Then use it when serializing your data:
json.dumps(data, cls=CustomJsonEncoder)
As noted from comments on the other answers, older versions of python might mess up the representation when converting to float, but that's not the case anymore.
To get the decimal back in Python:
Decimal(str(value))
This solution is hinted in Python 3.0 documentation on decimals:
To create a Decimal from a float, first convert it to a string.
This is what I have, extracted from our class
class CommonJSONEncoder(json.JSONEncoder):
"""
Common JSON Encoder
json.dumps(myString, cls=CommonJSONEncoder)
"""
def default(self, obj):
if isinstance(obj, decimal.Decimal):
return {'type{decimal}': str(obj)}
class CommonJSONDecoder(json.JSONDecoder):
"""
Common JSON Encoder
json.loads(myString, cls=CommonJSONEncoder)
"""
#classmethod
def object_hook(cls, obj):
for key in obj:
if isinstance(key, six.string_types):
if 'type{decimal}' == key:
try:
return decimal.Decimal(obj[key])
except:
pass
def __init__(self, **kwargs):
kwargs['object_hook'] = self.object_hook
super(CommonJSONDecoder, self).__init__(**kwargs)
Which passes unittest:
def test_encode_and_decode_decimal(self):
obj = Decimal('1.11')
result = json.dumps(obj, cls=CommonJSONEncoder)
self.assertTrue('type{decimal}' in result)
new_obj = json.loads(result, cls=CommonJSONDecoder)
self.assertEqual(new_obj, obj)
obj = {'test': Decimal('1.11')}
result = json.dumps(obj, cls=CommonJSONEncoder)
self.assertTrue('type{decimal}' in result)
new_obj = json.loads(result, cls=CommonJSONDecoder)
self.assertEqual(new_obj, obj)
obj = {'test': {'abc': Decimal('1.11')}}
result = json.dumps(obj, cls=CommonJSONEncoder)
self.assertTrue('type{decimal}' in result)
new_obj = json.loads(result, cls=CommonJSONDecoder)
self.assertEqual(new_obj, obj)
You can create a custom JSON encoder as per your requirement.
import json
from datetime import datetime, date
from time import time, struct_time, mktime
import decimal
class CustomJSONEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, datetime):
return str(o)
if isinstance(o, date):
return str(o)
if isinstance(o, decimal.Decimal):
return float(o)
if isinstance(o, struct_time):
return datetime.fromtimestamp(mktime(o))
# Any other serializer if needed
return super(CustomJSONEncoder, self).default(o)
The Decoder can be called like this,
import json
from decimal import Decimal
json.dumps({'x': Decimal('3.9')}, cls=CustomJSONEncoder)
and the output will be:
>>'{"x": 3.9}'
Based on stdOrgnlDave answer I have defined this wrapper that it can be called with optional kinds so the encoder will work only for certain kinds inside your projects. I believe the work should be done inside your code and not to use this "default" encoder since "it is better explicit than implicit", but I understand using this will save some of your time. :-)
import time
import json
import decimal
from uuid import UUID
from datetime import datetime
def JSONEncoder_newdefault(kind=['uuid', 'datetime', 'time', 'decimal']):
'''
JSON Encoder newdfeault is a wrapper capable of encoding several kinds
Use it anywhere on your code to make the full system to work with this defaults:
JSONEncoder_newdefault() # for everything
JSONEncoder_newdefault(['decimal']) # only for Decimal
'''
JSONEncoder_olddefault = json.JSONEncoder.default
def JSONEncoder_wrapped(self, o):
'''
json.JSONEncoder.default = JSONEncoder_newdefault
'''
if ('uuid' in kind) and isinstance(o, uuid.UUID):
return str(o)
if ('datetime' in kind) and isinstance(o, datetime):
return str(o)
if ('time' in kind) and isinstance(o, time.struct_time):
return datetime.fromtimestamp(time.mktime(o))
if ('decimal' in kind) and isinstance(o, decimal.Decimal):
return str(o)
return JSONEncoder_olddefault(self, o)
json.JSONEncoder.default = JSONEncoder_wrapped
# Example
if __name__ == '__main__':
JSONEncoder_newdefault()
If someone is still looking for the answer, it is most probably you have a 'NaN' in your data that you are trying to encode. Because NaN is considered as float by Python.
If you want to pass a dictionary containing decimals to the requests library (using the json keyword argument), you simply need to install simplejson:
$ pip3 install simplejson
$ python3
>>> import requests
>>> from decimal import Decimal
>>> # This won't error out:
>>> requests.post('https://www.google.com', json={'foo': Decimal('1.23')})
The reason of the problem is that requests uses simplejson only if it is present, and falls back to the built-in json if it is not installed.
For anybody that wants a quick solution here is how I removed Decimal from my queries in Django
total_development_cost_var = process_assumption_objects.values('total_development_cost').aggregate(sum_dev = Sum('total_development_cost', output_field=FloatField()))
total_development_cost_var = list(total_development_cost_var.values())
Step 1: use , output_field=FloatField() in you r query
Step 2: use list eg list(total_development_cost_var.values())
Hope it helps
This question is old, but there seems to be a better and much simpler solution in Python3 for most use-cases:
number = Decimal(0.55)
converted_number = float(number) # Returns: 0.55 (as type float)
You can just convert Decimal to float.
My 2 cents for easy solution, if you're sure Decimal is the only bad guy on your json dumps method:
print(json.loads(json.dumps({
'a': Decimal(1230),
'b': Decimal(11111111123.22),
}, default=lambda x: eval(str(x)))))
>>> {'a': 1230, 'b': 11111111123.22}
The "smart" thing here is using default to convert Decimal to int or float, automatically, taking advantage of eval function: default=lambda x: eval(str(x))
But always be careful using eval on your code as it can lead to security issues ;)
Decimal is not suitable to be converted through:
float due to precision problems
str due to openapi restrictions
We still need direct decimal to a number json serialisation.
Here is our extension of #tesdal 's fakefloat solution (closed in v3.5.2rc1).
It uses fakestr + monkeypatching to avoid quotation and "floatation" of decimals.
import json.encoder
from decimal import Decimal
def encode_fakestr(func):
def wrap(s):
if isinstance(s, fakestr):
return repr(s)
return func(s)
return wrap
json.encoder.encode_basestring = encode_fakestr(json.encoder.encode_basestring)
json.encoder.encode_basestring_ascii = encode_fakestr(json.encoder.encode_basestring_ascii)
class fakestr(str):
def __init__(self, value):
self._value = value
def __repr__(self):
return str(self._value)
class DecimalJsonEncoder(json.encoder.JSONEncoder):
def default(self, o):
if isinstance(o, Decimal):
return fakestr(o)
return super().default(o)
json.dumps([Decimal('1.1')], cls=DecimalJsonEncoder)
[1.1]
I don't understand why python developers force us using floats in places where it is not suitable.
I will share what worked for me with flask 2.1.0
When I was creating the dictionary which had to be used from jsonify I used rounding:
json_dict['price'] = round(self.price, ndigits=2) if self.price else 0
So this way I could return D.DD number or 0 without using some global configuration. And this is nice because some Decimals has to be bigger, like latitude and longitude coordinates.
return jsonify(json_dict)
I have a Decimal('3.9') as part of an object, and wish to encode this to a JSON string which should look like {'x': 3.9}. I don't care about precision on the client side, so a float is fine.
Is there a good way to serialize this? JSONDecoder doesn't accept Decimal objects, and converting to a float beforehand yields {'x': 3.8999999999999999} which is wrong, and will be a big waste of bandwidth.
Simplejson 2.1 and higher has native support for Decimal type:
>>> json.dumps(Decimal('3.9'), use_decimal=True)
'3.9'
Note that use_decimal is True by default:
def dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True,
allow_nan=True, cls=None, indent=None, separators=None,
encoding='utf-8', default=None, use_decimal=True,
namedtuple_as_object=True, tuple_as_array=True,
bigint_as_string=False, sort_keys=False, item_sort_key=None,
for_json=False, ignore_nan=False, **kw):
So:
>>> json.dumps(Decimal('3.9'))
'3.9'
Hopefully, this feature will be included in standard library.
I would like to let everyone know that I tried Michał Marczyk's answer on my web server that was running Python 2.6.5 and it worked fine. However, I upgraded to Python 2.7 and it stopped working. I tried to think of some sort of way to encode Decimal objects and this is what I came up with:
import decimal
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
return str(o)
return super(DecimalEncoder, self).default(o)
Note that this will convert the decimal to its string representation (e.g.; "1.2300") to a. not lose significant digits and b. prevent rounding errors.
This should hopefully help anyone who is having problems with Python 2.7. I tested it and it seems to work fine. If anyone notices any bugs in my solution or comes up with a better way, please let me know.
Usage example:
json.dumps({'x': decimal.Decimal('5.5')}, cls=DecimalEncoder)
How about subclassing json.JSONEncoder?
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
# wanted a simple yield str(o) in the next line,
# but that would mean a yield on the line with super(...),
# which wouldn't work (see my comment below), so...
return (str(o) for o in [o])
return super(DecimalEncoder, self).default(o)
Then use it like so:
json.dumps({'x': decimal.Decimal('5.5')}, cls=DecimalEncoder)
The native Django option is missing so I'll add it for the next guy/gall that looks for it.
Starting on Django 1.7.x there is a built-in DjangoJSONEncoder that you can get it from django.core.serializers.json.
import json
from django.core.serializers.json import DjangoJSONEncoder
from django.forms.models import model_to_dict
model_instance = YourModel.object.first()
model_dict = model_to_dict(model_instance)
json.dumps(model_dict, cls=DjangoJSONEncoder)
Presto!
In my Flask app, Which uses python 2.7.11, flask alchemy(with 'db.decimal' types), and Flask Marshmallow ( for 'instant' serializer and deserializer), i had this error, every time i did a GET or POST. The serializer and deserializer, failed to convert Decimal types into any JSON identifiable format.
I did a "pip install simplejson", then
Just by adding
import simplejson as json
the serializer and deserializer starts to purr again. I did nothing else...
DEciamls are displayed as '234.00' float format.
I tried switching from simplejson to builtin json for GAE 2.7, and had issues with the decimal. If default returned str(o) there were quotes (because _iterencode calls _iterencode on the results of default), and float(o) would remove trailing 0.
If default returns an object of a class that inherits from float (or anything that calls repr without additional formatting) and has a custom __repr__ method, it seems to work like I want it to.
import json
from decimal import Decimal
class fakefloat(float):
def __init__(self, value):
self._value = value
def __repr__(self):
return str(self._value)
def defaultencode(o):
if isinstance(o, Decimal):
# Subclass float with custom repr?
return fakefloat(o)
raise TypeError(repr(o) + " is not JSON serializable")
json.dumps([10.20, "10.20", Decimal('10.20')], default=defaultencode)
'[10.2, "10.20", 10.20]'
For Django users:
Recently came across TypeError: Decimal('2337.00') is not JSON serializable
while JSON encoding i.e. json.dumps(data)
Solution:
# converts Decimal, Datetime, UUIDs to str for Encoding
from django.core.serializers.json import DjangoJSONEncoder
json.dumps(response.data, cls=DjangoJSONEncoder)
But, now the Decimal value will be a string, now we can explicitly set the decimal/float value parser when decoding data, using parse_float option in json.loads:
import decimal
data = json.loads(data, parse_float=decimal.Decimal) # default is float(num_str)
3.9 can not be exactly represented in IEEE floats, it will always come as 3.8999999999999999, e.g. try print repr(3.9), you can read more about it here:
http://en.wikipedia.org/wiki/Floating_point
http://docs.sun.com/source/806-3568/ncg_goldberg.html
So if you don't want float, only option you have to send it as string, and to allow automatic conversion of decimal objects to JSON, do something like this:
import decimal
from django.utils import simplejson
def json_encode_decimal(obj):
if isinstance(obj, decimal.Decimal):
return str(obj)
raise TypeError(repr(obj) + " is not JSON serializable")
d = decimal.Decimal('3.5')
print simplejson.dumps([d], default=json_encode_decimal)
My $.02!
I extend a bunch of the JSON encoder since I am serializing tons of data for my web server. Here's some nice code. Note that it's easily extendable to pretty much any data format you feel like and will reproduce 3.9 as "thing": 3.9
JSONEncoder_olddefault = json.JSONEncoder.default
def JSONEncoder_newdefault(self, o):
if isinstance(o, UUID): return str(o)
if isinstance(o, datetime): return str(o)
if isinstance(o, time.struct_time): return datetime.fromtimestamp(time.mktime(o))
if isinstance(o, decimal.Decimal): return str(o)
return JSONEncoder_olddefault(self, o)
json.JSONEncoder.default = JSONEncoder_newdefault
Makes my life so much easier...
For those who don't want to use a third-party library... An issue with Elias Zamaria's answer is that it converts to float, which can run into problems. For example:
>>> json.dumps({'x': Decimal('0.0000001')}, cls=DecimalEncoder)
'{"x": 1e-07}'
>>> json.dumps({'x': Decimal('100000000000.01734')}, cls=DecimalEncoder)
'{"x": 100000000000.01733}'
The JSONEncoder.encode() method lets you return the literal json content, unlike JSONEncoder.default(), which has you return a json compatible type (like float) that then gets encoded in the normal way. The problem with encode() is that it (normally) only works at the top level. But it's still usable, with a little extra work (python 3.x):
import json
from collections.abc import Mapping, Iterable
from decimal import Decimal
class DecimalEncoder(json.JSONEncoder):
def encode(self, obj):
if isinstance(obj, Mapping):
return '{' + ', '.join(f'{self.encode(k)}: {self.encode(v)}' for (k, v) in obj.items()) + '}'
if isinstance(obj, Iterable) and (not isinstance(obj, str)):
return '[' + ', '.join(map(self.encode, obj)) + ']'
if isinstance(obj, Decimal):
return f'{obj.normalize():f}' # using normalize() gets rid of trailing 0s, using ':f' prevents scientific notation
return super().encode(obj)
Which gives you:
>>> json.dumps({'x': Decimal('0.0000001')}, cls=DecimalEncoder)
'{"x": 0.0000001}'
>>> json.dumps({'x': Decimal('100000000000.01734')}, cls=DecimalEncoder)
'{"x": 100000000000.01734}'
From the JSON Standard Document, as linked in json.org:
JSON is agnostic about the semantics of numbers. In any programming language, there can be a variety of
number types of various capacities and complements, fixed or floating, binary or decimal. That can make
interchange between different programming languages difficult. JSON instead offers only the representation of
numbers that humans use: a sequence of digits. All programming languages know how to make sense of digit
sequences even if they disagree on internal representations. That is enough to allow interchange.
So it's actually accurate to represent Decimals as numbers (rather than strings) in JSON. Bellow lies a possible solution to the problem.
Define a custom JSON encoder:
import json
class CustomJsonEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, Decimal):
return float(obj)
return super(CustomJsonEncoder, self).default(obj)
Then use it when serializing your data:
json.dumps(data, cls=CustomJsonEncoder)
As noted from comments on the other answers, older versions of python might mess up the representation when converting to float, but that's not the case anymore.
To get the decimal back in Python:
Decimal(str(value))
This solution is hinted in Python 3.0 documentation on decimals:
To create a Decimal from a float, first convert it to a string.
This is what I have, extracted from our class
class CommonJSONEncoder(json.JSONEncoder):
"""
Common JSON Encoder
json.dumps(myString, cls=CommonJSONEncoder)
"""
def default(self, obj):
if isinstance(obj, decimal.Decimal):
return {'type{decimal}': str(obj)}
class CommonJSONDecoder(json.JSONDecoder):
"""
Common JSON Encoder
json.loads(myString, cls=CommonJSONEncoder)
"""
#classmethod
def object_hook(cls, obj):
for key in obj:
if isinstance(key, six.string_types):
if 'type{decimal}' == key:
try:
return decimal.Decimal(obj[key])
except:
pass
def __init__(self, **kwargs):
kwargs['object_hook'] = self.object_hook
super(CommonJSONDecoder, self).__init__(**kwargs)
Which passes unittest:
def test_encode_and_decode_decimal(self):
obj = Decimal('1.11')
result = json.dumps(obj, cls=CommonJSONEncoder)
self.assertTrue('type{decimal}' in result)
new_obj = json.loads(result, cls=CommonJSONDecoder)
self.assertEqual(new_obj, obj)
obj = {'test': Decimal('1.11')}
result = json.dumps(obj, cls=CommonJSONEncoder)
self.assertTrue('type{decimal}' in result)
new_obj = json.loads(result, cls=CommonJSONDecoder)
self.assertEqual(new_obj, obj)
obj = {'test': {'abc': Decimal('1.11')}}
result = json.dumps(obj, cls=CommonJSONEncoder)
self.assertTrue('type{decimal}' in result)
new_obj = json.loads(result, cls=CommonJSONDecoder)
self.assertEqual(new_obj, obj)
You can create a custom JSON encoder as per your requirement.
import json
from datetime import datetime, date
from time import time, struct_time, mktime
import decimal
class CustomJSONEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, datetime):
return str(o)
if isinstance(o, date):
return str(o)
if isinstance(o, decimal.Decimal):
return float(o)
if isinstance(o, struct_time):
return datetime.fromtimestamp(mktime(o))
# Any other serializer if needed
return super(CustomJSONEncoder, self).default(o)
The Decoder can be called like this,
import json
from decimal import Decimal
json.dumps({'x': Decimal('3.9')}, cls=CustomJSONEncoder)
and the output will be:
>>'{"x": 3.9}'
Based on stdOrgnlDave answer I have defined this wrapper that it can be called with optional kinds so the encoder will work only for certain kinds inside your projects. I believe the work should be done inside your code and not to use this "default" encoder since "it is better explicit than implicit", but I understand using this will save some of your time. :-)
import time
import json
import decimal
from uuid import UUID
from datetime import datetime
def JSONEncoder_newdefault(kind=['uuid', 'datetime', 'time', 'decimal']):
'''
JSON Encoder newdfeault is a wrapper capable of encoding several kinds
Use it anywhere on your code to make the full system to work with this defaults:
JSONEncoder_newdefault() # for everything
JSONEncoder_newdefault(['decimal']) # only for Decimal
'''
JSONEncoder_olddefault = json.JSONEncoder.default
def JSONEncoder_wrapped(self, o):
'''
json.JSONEncoder.default = JSONEncoder_newdefault
'''
if ('uuid' in kind) and isinstance(o, uuid.UUID):
return str(o)
if ('datetime' in kind) and isinstance(o, datetime):
return str(o)
if ('time' in kind) and isinstance(o, time.struct_time):
return datetime.fromtimestamp(time.mktime(o))
if ('decimal' in kind) and isinstance(o, decimal.Decimal):
return str(o)
return JSONEncoder_olddefault(self, o)
json.JSONEncoder.default = JSONEncoder_wrapped
# Example
if __name__ == '__main__':
JSONEncoder_newdefault()
If someone is still looking for the answer, it is most probably you have a 'NaN' in your data that you are trying to encode. Because NaN is considered as float by Python.
If you want to pass a dictionary containing decimals to the requests library (using the json keyword argument), you simply need to install simplejson:
$ pip3 install simplejson
$ python3
>>> import requests
>>> from decimal import Decimal
>>> # This won't error out:
>>> requests.post('https://www.google.com', json={'foo': Decimal('1.23')})
The reason of the problem is that requests uses simplejson only if it is present, and falls back to the built-in json if it is not installed.
For anybody that wants a quick solution here is how I removed Decimal from my queries in Django
total_development_cost_var = process_assumption_objects.values('total_development_cost').aggregate(sum_dev = Sum('total_development_cost', output_field=FloatField()))
total_development_cost_var = list(total_development_cost_var.values())
Step 1: use , output_field=FloatField() in you r query
Step 2: use list eg list(total_development_cost_var.values())
Hope it helps
This question is old, but there seems to be a better and much simpler solution in Python3 for most use-cases:
number = Decimal(0.55)
converted_number = float(number) # Returns: 0.55 (as type float)
You can just convert Decimal to float.
My 2 cents for easy solution, if you're sure Decimal is the only bad guy on your json dumps method:
print(json.loads(json.dumps({
'a': Decimal(1230),
'b': Decimal(11111111123.22),
}, default=lambda x: eval(str(x)))))
>>> {'a': 1230, 'b': 11111111123.22}
The "smart" thing here is using default to convert Decimal to int or float, automatically, taking advantage of eval function: default=lambda x: eval(str(x))
But always be careful using eval on your code as it can lead to security issues ;)
Decimal is not suitable to be converted through:
float due to precision problems
str due to openapi restrictions
We still need direct decimal to a number json serialisation.
Here is our extension of #tesdal 's fakefloat solution (closed in v3.5.2rc1).
It uses fakestr + monkeypatching to avoid quotation and "floatation" of decimals.
import json.encoder
from decimal import Decimal
def encode_fakestr(func):
def wrap(s):
if isinstance(s, fakestr):
return repr(s)
return func(s)
return wrap
json.encoder.encode_basestring = encode_fakestr(json.encoder.encode_basestring)
json.encoder.encode_basestring_ascii = encode_fakestr(json.encoder.encode_basestring_ascii)
class fakestr(str):
def __init__(self, value):
self._value = value
def __repr__(self):
return str(self._value)
class DecimalJsonEncoder(json.encoder.JSONEncoder):
def default(self, o):
if isinstance(o, Decimal):
return fakestr(o)
return super().default(o)
json.dumps([Decimal('1.1')], cls=DecimalJsonEncoder)
[1.1]
I don't understand why python developers force us using floats in places where it is not suitable.
I will share what worked for me with flask 2.1.0
When I was creating the dictionary which had to be used from jsonify I used rounding:
json_dict['price'] = round(self.price, ndigits=2) if self.price else 0
So this way I could return D.DD number or 0 without using some global configuration. And this is nice because some Decimals has to be bigger, like latitude and longitude coordinates.
return jsonify(json_dict)
class MSG_TYPE(IntEnum):
REQUEST = 0
GRANT = 1
RELEASE = 2
FAIL = 3
INQUIRE = 4
YIELD = 5
def __json__(self):
return str(self)
class MessageEncoder(JSONEncoder):
def default(self, obj):
return obj.__json__()
class Message(object):
def __init__(self, msg_type, src, dest, data):
self.msg_type = msg_type
self.src = src
self.dest = dest
self.data = data
def __json__(self):
return dict (\
msg_type=self.msg_type, \
src=self.src, \
dest=self.dest, \
data=self.data,\
)
def ToJSON(self):
return json.dumps(self, cls=MessageEncoder)
msg = Message(msg_type=MSG_TYPE.FAIL, src=0, dest=1, data="hello world")
encoded_msg = msg.ToJSON()
decoded_msg = yaml.load(encoded_msg)
print type(decoded_msg['msg_type'])
When calling print type(decoded_msg['msg_type']), I get the result <type 'str'> instead of the original MSG_TYPTE type. I feel like I should also write a custom json decoder but kind of confused how to do that. Any ideas? Thanks.
When calling print type(decoded_msg['msg_type']), I get the result instead of the original MSG_TYPTE type.
Well, yeah, that's because you told MSG_TYPE to encode itself like this:
def __json__(self):
return str(self)
So, that's obviously going to decode back to a string. If you don't want that, come up with some unique way to encode the values, instead of just encoding their string representations.
The most common way to do this is to encode all of your custom types (including your enum types) using some specialized form of object—just like you've done for Message. For example, you might put a py-type field in the object which encodes the type of your object, and then the meanings of the other fields all depend on the type. Ideally you'll want to abstract out the commonalities instead of hardcoding the same thing 100 times, of course.
I feel like I should also write a custom json decoder but kind of confused how to do that.
Well, have you read the documentation? Where exactly are you confused? You're not going to get a complete tutorial by tacking on a followup to a StackOverflow question…
Assuming you've got a special object structure for all your types, you can use an object_hook to decode the values back to the originals. For example, as a quick hack:
class MessageEncoder(JSONEncoder):
def default(self, obj):
return {'py-type': type(obj).__name__, 'value': obj.__json__()}
class MessageDecoder(JSONDecoder):
def __init__(self, hook=None, *args, **kwargs):
if hook is None: hook = self.hook
return super().__init__(hook, *args, **kwargs)
def hook(self, obj):
if isinstance(obj, dict):
pytype = obj.get('py-type')
if pytype:
t = globals()[pytype]
return t.__unjson__(**obj['value'])
return obj
And now, in your Message class:
#classmethod
def __unjson__(cls, msg_type, src, dest, data):
return cls(msg_type, src, dest, data)
And you need a MSG_TYPE.__json__ that returns a dict, maybe just {'name': str(self)}, then an __unjson__ that does something like getattr(cls, name).
A real-life solution should probably either have the classes register themselves instead of looking them up by name, or should handle looking them up by qualified name instead of just going to globals(). And you may want to let things encode to something other than object—or, if not, to just cram py-type into the object instead of wrapping it in another one. And there may be other ways to make the JSON more compact and/or readable. And a little bit of error handling would be nice. And so on.
You may want to look at the implementation of jsonpickle—not because you want to do the exact same thing it does, but to see how it hooks up all the pieces.
Overriding the default method of the encoder won't matter in this case because your object never gets passed to the method. It's treated as an int.
If you run the encoder on its own:
msg_type = MSG_TYPE.RELEASE
MessageEncoder().encode(msg_type)
You'll get:
'MSG_TYPE.RELEASE'
If you can, use an Enum and you shouldn't have any issues. I also asked a similar question:
How do I serialize IntEnum from enum34 to json in python?
This question is based on a side-effect of that one.
My .py files are all have # -*- coding: utf-8 -*- encoding definer on the first line, like my api.py
As I mention on the related question, I use HttpResponse to return the api documentation. Since I defined encoding by:
HttpResponse(cy_content, content_type='text/plain; charset=utf-8')
Everything is ok, and when I call my API service, there are no encoding problems except the string formed from a dictionary by pprint
Since I am using Turkish characters in some values in my dict, pprint converts them to unichr equivalents, like:
API_STATUS = {
1: 'müşteri',
2: 'some other status message'
}
my_str = 'Here is the documentation part that contains Turkish chars like işüğçö'
my_str += pprint.pformat(API_STATUS, indent=4, width=1)
return HttpRespopnse(my_str, content_type='text/plain; charset=utf-8')
And my plain text output is like:
Here is the documentation part that contains Turkish chars like işüğçö
{
1: 'm\xc3\xbc\xc5\x9fteri',
2: 'some other status message'
}
I try to decode or encode pprint output to different encodings, with no success... What is the best practice to overcome this problem
pprint appears to use repr by default, you can work around this by overriding PrettyPrinter.format:
# coding=utf8
import pprint
class MyPrettyPrinter(pprint.PrettyPrinter):
def format(self, object, context, maxlevels, level):
if isinstance(object, unicode):
return (object.encode('utf8'), True, False)
return pprint.PrettyPrinter.format(self, object, context, maxlevels, level)
d = {'foo': u'işüğçö'}
pprint.pprint(d) # {'foo': u'i\u015f\xfc\u011f\xe7\xf6'}
MyPrettyPrinter().pprint(d) # {'foo': işüğçö}
You should use unicode strings instead of 8-bit ones:
API_STATUS = {
1: u'müşteri',
2: u'some other status message'
}
my_str = u'Here is the documentation part that contains Turkish chars like işüğçö'
my_str += pprint.pformat(API_STATUS, indent=4, width=1)
The pprint module is designed to print out all possible kind of nested structure in a readable way. To do that it will print the objects representation rather then convert it to a string, so you'll end up with the escape syntax wheather you use unicode strings or not. But if you're using unicode in your document, then you really should be using unicode literals!
Anyway, thg435 has given you a solution how to change this behaviour of pformat.
for p in db.collection.find({"test_set":"abc"}):
posts.append(p)
thejson = json.dumps({'results':posts})
return HttpResponse(thejson, mimetype="application/javascript")
In my Django/Python code, I can't return a JSON from a mongo query because of "ObjectID". The error says that "ObjectID" is not serializable.
What do I have to do?
A hacky way would be to loop through:
for p in posts:
p['_id'] = ""
The json module won't work due to things like the ObjectID.
Luckily PyMongo provides json_util which ...
... allow[s] for specialized encoding and
decoding of BSON documents into Mongo
Extended JSON's Strict mode. This lets
you encode / decode BSON documents to
JSON even when they use special BSON
types.
Here is a simple sample, using pymongo 2.2.1
import os
import sys
import json
import pymongo
from bson import BSON
from bson import json_util
if __name__ == '__main__':
try:
connection = pymongo.Connection('mongodb://localhost:27017')
database = connection['mongotest']
except:
print('Error: Unable to Connect')
connection = None
if connection is not None:
database["test"].insert({'name': 'foo'})
doc = database["test"].find_one({'name': 'foo'})
return json.dumps(doc, sort_keys=True, indent=4, default=json_util.default)
It's pretty easy to write a custom serializer which copes with the ObjectIds. Django already includes one which handles decimals and dates, so you can extend that:
from django.core.serializers.json import DjangoJSONEncoder
from bson import objectid
class MongoAwareEncoder(DjangoJSONEncoder):
"""JSON encoder class that adds support for Mongo objectids."""
def default(self, o):
if isinstance(o, objectid.ObjectId):
return str(o)
else:
return super(MongoAwareEncoder, self).default(o)
Now you can just tell json to use your custom serializer:
thejson = json.dumps({'results':posts}, cls=MongoAwareEncoder)
Something even simpler which works for me on Python 3.6 using
motor==1.1
pymongo==3.4.0
from bson.json_util import dumps, loads
for mongo_doc in await cursor.to_list(length=10):
# mongo_doc is a <class 'dict'> returned from the async mongo driver, in this acse motor / pymongo.
# result of executing a simple find() query.
json_string = dumps(mongo_doc)
# serialize the <class 'dict'> into a <class 'str'>
back_to_dict = loads(json_string)
# to unserialize, thus return the string back to a <class 'dict'> with the original 'ObjectID' type.