I'm having trouble encoding infinity in json.
json.dumps will convert this to "Infinity", but I would like it do convert it to null or another value of my choosing.
Unfortunately, setting default argument only seems to work if dumps does't already understand the object, otherwise the default handler appears to be bypassed.
Is there a way I can pre-encode the object, change the default way a type/class is encoded, or convert a certain type/class into a different object prior to normal encoding?
Look at the source here: http://hg.python.org/cpython/file/7ec9255d4189/Lib/json/encoder.py
If you subclass JSONEncoder, you can override just the iterencode(self, o, _one_shot=False) method, which has explicit special casing for Infinity (inside an inner function).
To make this reusable, you'll also want to alter the __init__ to take some new options, and store them in the class.
Alternatively, you could pick a json library from pypi which has the appropriate extensibility you are looking for: https://pypi.python.org/pypi?%3Aaction=search&term=json&submit=search
Here's an example:
import json
class FloatEncoder(json.JSONEncoder):
def __init__(self, nan_str = "null", **kwargs):
super(FloatEncoder,self).__init__(**kwargs)
self.nan_str = nan_str
# uses code from official python json.encoder module.
# Same licence applies.
def iterencode(self, o, _one_shot=False):
"""Encode the given object and yield each string
representation as available.
For example::
for chunk in JSONEncoder().iterencode(bigobject):
mysocket.write(chunk)
"""
if self.check_circular:
markers = {}
else:
markers = None
if self.ensure_ascii:
_encoder = json.encoder.encode_basestring_ascii
else:
_encoder = json.encoder.encode_basestring
if self.encoding != 'utf-8':
def _encoder(o, _orig_encoder=_encoder,
_encoding=self.encoding):
if isinstance(o, str):
o = o.decode(_encoding)
return _orig_encoder(o)
def floatstr(o, allow_nan=self.allow_nan,
_repr=json.encoder.FLOAT_REPR,
_inf=json.encoder.INFINITY,
_neginf=-json.encoder.INFINITY,
nan_str = self.nan_str):
# Check for specials. Note that this type of test is
# processor and/or platform-specific, so do tests which
# don't depend on the internals.
if o != o:
text = nan_str
elif o == _inf:
text = 'Infinity'
elif o == _neginf:
text = '-Infinity'
else:
return _repr(o)
if not allow_nan:
raise ValueError(
"Out of range float values are not JSON compliant: " +
repr(o))
return text
_iterencode = json.encoder._make_iterencode(
markers, self.default, _encoder, self.indent, floatstr,
self.key_separator, self.item_separator, self.sort_keys,
self.skipkeys, _one_shot)
return _iterencode(o, 0)
example_obj = {
'name': 'example',
'body': [
1.1,
{"3.3": 5, "1.1": float('Nan')},
[float('inf'), 2.2]
]}
print json.dumps(example_obj, cls=FloatEncoder)
ideone: http://ideone.com/dFWaNj
No, there is no simple way to achieve this. In fact, NaN and Infinity floating point values shouldn't be serialized with json at all, according to the standard.
Python uses an extension of the standard. You can make the python encoding standard-compliant passing the allow_nan=False parameter to dumps, but this will raise a ValueError for infinity/nans even if you provide a default function.
You have two ways of doing what you want:
Subclass JSONEncoder and change how these values are encoded. Note that you will have to take into account cases where a sequence can contain an infinity value etc. AFAIK there is no API to redefine how objects of a specific class are encoded.
Make a copy of the object to encode and replace any occurrence of infinity/nan with None or some other object that is encoded as you want.
A less robust, yet much simpler solution, is to modify the encoded data, for example replacing all Infinity substrings with null:
>>> import re
>>> infty_regex = re.compile(r'\bInfinity\b')
>>> def replace_infinities(encoded):
... regex = re.compile(r'\bInfinity\b')
... return regex.sub('null', encoded)
...
>>> import json
>>> replace_infinities(json.dumps([1, 2, 3, float('inf'), 4]))
'[1, 2, 3, null, 4]'
Obviously you should take into account the text Infinity inside strings etc., so even here a robust solution is not immediate, nor elegant.
Context
I ran into this issue and didn't want to bring an extra dependency into the project just to handle this case. Additionally, my project supports Python 2.6, 2.7, 3.3, and 3.4 and user's of simplejson. Unfortunately there are three different implementations of iterencode between these versions, so hard-coding a particular version was undesirable.
Hopefully this will help someone else with similar requirements!
Qualifiers
If the encoding time/processing-power surrounding your json.dumps call is small compared to other components of your project, you can un-encode/re-encode the JSON to get your desired result leveraging the parse_constant kwarg.
Benefits
It doesn't matter if the end-user has Python 2.x's json, Python 3.x's json or is using simplejson (e.g, import simplejson as json)
It only uses public json interfaces which are unlikely to change.
Caveats
This will take ~3X as long to encode things
This implementation doesn't handle object_pairs_hook because then it wouldn't work for python 2.6
Invalid separators will fail
Code
class StrictJSONEncoder(json.JSONEncoder):
def default(self, o):
"""Make sure we don't instantly fail"""
return o
def coerce_to_strict(self, const):
"""
This is used to ultimately *encode* into strict JSON, see `encode`
"""
# before python 2.7, 'true', 'false', 'null', were include here.
if const in ('Infinity', '-Infinity', 'NaN'):
return None
else:
return const
def encode(self, o):
"""
Load and then dump the result using parse_constant kwarg
Note that setting invalid separators will cause a failure at this step.
"""
# this will raise errors in a normal-expected way
encoded_o = super(StrictJSONEncoder, self).encode(o)
# now:
# 1. `loads` to switch Infinity, -Infinity, NaN to None
# 2. `dumps` again so you get 'null' instead of extended JSON
try:
new_o = json.loads(encoded_o, parse_constant=self.coerce_to_strict)
except ValueError:
# invalid separators will fail here. raise a helpful exception
raise ValueError(
"Encoding into strict JSON failed. Did you set the separators "
"valid JSON separators?"
)
else:
return json.dumps(new_o, sort_keys=self.sort_keys,
indent=self.indent,
separators=(self.item_separator,
self.key_separator))
You could do something along these lines:
import json
import math
target=[1.1,1,2.2,float('inf'),float('nan'),'a string',int(2)]
def ffloat(f):
if not isinstance(f,float):
return f
if math.isnan(f):
return 'custom NaN'
if math.isinf(f):
return 'custom inf'
return f
print 'regular json:',json.dumps(target)
print 'customized:',json.dumps(map(ffloat,target))
Prints:
regular json: [1.1, 1, 2.2, Infinity, NaN, "a string", 2]
customized: [1.1, 1, 2.2, "custom inf", "custom NaN", "a string", 2]
If you want to handle nested data structures, this is also not that hard:
import json
import math
from collections import Mapping, Sequence
def nested_json(o):
if isinstance(o, float):
if math.isnan(o):
return 'custom NaN'
if math.isinf(o):
return 'custom inf'
return o
elif isinstance(o, basestring):
return o
elif isinstance(o, Sequence):
return [nested_json(item) for item in o]
elif isinstance(o, Mapping):
return dict((key, nested_json(value)) for key, value in o.iteritems())
else:
return o
nested_tgt=[1.1,{1.1:float('inf'),3.3:5},(float('inf'),2.2),]
print 'regular json:',json.dumps(nested_tgt)
print 'nested json',json.dumps(nested_json(nested_tgt))
Prints:
regular json: [1.1, {"3.3": 5, "1.1": Infinity}, [Infinity, 2.2]]
nested json [1.1, {"3.3": 5, "1.1": "custom inf"}, ["custom inf", 2.2]]
Related
I have a situation with some code where eval() came up as a possible solution. Now I have never had to use eval() before but, I have come across plenty of information about the potential danger it can cause. That said, I'm very wary about using it.
My situation is that I have input being given by a user:
datamap = input('Provide some data here: ')
Where datamap needs to be a dictionary. I searched around and found that eval() could work this out. I thought that I might be able to check the type of the input before trying to use the data and that would be a viable security precaution.
datamap = eval(input('Provide some data here: ')
if not isinstance(datamap, dict):
return
I read through the docs and I am still unclear if this would be safe or not. Does eval evaluate the data as soon as its entered or after the datamap variable is called?
Is the ast module's .literal_eval() the only safe option?
datamap = eval(input('Provide some data here: ')) means that you actually evaluate the code before you deem it to be unsafe or not. It evaluates the code as soon as the function is called. See also the dangers of eval.
ast.literal_eval raises an exception if the input isn't a valid Python datatype, so the code won't be executed if it's not.
Use ast.literal_eval whenever you need eval. You shouldn't usually evaluate literal Python statements.
ast.literal_eval() only considers a small subset of Python's syntax to be valid:
The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.
Passing __import__('os').system('rm -rf /a-path-you-really-care-about') into ast.literal_eval() will raise an error, but eval() will happily delete your files.
Since it looks like you're only letting the user input a plain dictionary, use ast.literal_eval(). It safely does what you want and nothing more.
eval:
This is very powerful, but is also very dangerous if you accept strings to evaluate from untrusted input. Suppose the string being evaluated is "os.system('rm -rf /')" ? It will really start deleting all the files on your computer.
ast.literal_eval:
Safely evaluate an expression node or a string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None, bytes and sets.
Syntax:
eval(expression, globals=None, locals=None)
import ast
ast.literal_eval(node_or_string)
Example:
# python 2.x - doesn't accept operators in string format
import ast
ast.literal_eval('[1, 2, 3]') # output: [1, 2, 3]
ast.literal_eval('1+1') # output: ValueError: malformed string
# python 3.0 -3.6
import ast
ast.literal_eval("1+1") # output : 2
ast.literal_eval("{'a': 2, 'b': 3, 3:'xyz'}") # output : {'a': 2, 'b': 3, 3:'xyz'}
# type dictionary
ast.literal_eval("",{}) # output : Syntax Error required only one parameter
ast.literal_eval("__import__('os').system('rm -rf /')") # output : error
eval("__import__('os').system('rm -rf /')")
# output : start deleting all the files on your computer.
# restricting using global and local variables
eval("__import__('os').system('rm -rf /')",{'__builtins__':{}},{})
# output : Error due to blocked imports by passing '__builtins__':{} in global
# But still eval is not safe. we can access and break the code as given below
s = """
(lambda fc=(
lambda n: [
c for c in
().__class__.__bases__[0].__subclasses__()
if c.__name__ == n
][0]
):
fc("function")(
fc("code")(
0,0,0,0,"KABOOM",(),(),(),"","",0,""
),{}
)()
)()
"""
eval(s, {'__builtins__':{}})
In the above code ().__class__.__bases__[0] nothing but object itself.
Now we instantiated all the subclasses, here our main enter code hereobjective is to find one class named n from it.
We need to code object and function object from instantiated subclasses. This is an alternative way from CPython to access subclasses of object and attach the system.
From python 3.7 ast.literal_eval() is now stricter. Addition and subtraction of arbitrary numbers are no longer allowed. link
Python's eager in its evaluation, so eval(input(...)) (Python 3) will evaluate the user's input as soon as it hits the eval, regardless of what you do with the data afterwards. Therefore, this is not safe, especially when you eval user input.
Use ast.literal_eval.
As an example, entering this at the prompt could be very bad for you:
__import__('os').system('rm -rf /a-path-you-really-care-about')
In recent Python3 ast.literal_eval() no longer parses simple strings, instead you are supposed to use the ast.parse() method to create an AST then interpret it.
This is a complete example of using ast.parse() correctly in Python 3.6+ to evaluate simple arithmetic expressions safely.
import ast, operator, math
import logging
logger = logging.getLogger(__file__)
def safe_eval(s):
def checkmath(x, *args):
if x not in [x for x in dir(math) if not "__" in x]:
raise SyntaxError(f"Unknown func {x}()")
fun = getattr(math, x)
return fun(*args)
binOps = {
ast.Add: operator.add,
ast.Sub: operator.sub,
ast.Mult: operator.mul,
ast.Div: operator.truediv,
ast.Mod: operator.mod,
ast.Pow: operator.pow,
ast.Call: checkmath,
ast.BinOp: ast.BinOp,
}
unOps = {
ast.USub: operator.neg,
ast.UAdd: operator.pos,
ast.UnaryOp: ast.UnaryOp,
}
ops = tuple(binOps) + tuple(unOps)
tree = ast.parse(s, mode='eval')
def _eval(node):
if isinstance(node, ast.Expression):
logger.debug("Expr")
return _eval(node.body)
elif isinstance(node, ast.Str):
logger.debug("Str")
return node.s
elif isinstance(node, ast.Num):
logger.debug("Num")
return node.value
elif isinstance(node, ast.Constant):
logger.info("Const")
return node.value
elif isinstance(node, ast.BinOp):
logger.debug("BinOp")
if isinstance(node.left, ops):
left = _eval(node.left)
else:
left = node.left.value
if isinstance(node.right, ops):
right = _eval(node.right)
else:
right = node.right.value
return binOps[type(node.op)](left, right)
elif isinstance(node, ast.UnaryOp):
logger.debug("UpOp")
if isinstance(node.operand, ops):
operand = _eval(node.operand)
else:
operand = node.operand.value
return unOps[type(node.op)](operand)
elif isinstance(node, ast.Call):
args = [_eval(x) for x in node.args]
r = checkmath(node.func.id, *args)
return r
else:
raise SyntaxError(f"Bad syntax, {type(node)}")
return _eval(tree)
if __name__ == "__main__":
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
logger.addHandler(ch)
assert safe_eval("1+1") == 2
assert safe_eval("1+-5") == -4
assert safe_eval("-1") == -1
assert safe_eval("-+1") == -1
assert safe_eval("(100*10)+6") == 1006
assert safe_eval("100*(10+6)") == 1600
assert safe_eval("2**4") == 2**4
assert safe_eval("sqrt(16)+1") == math.sqrt(16) + 1
assert safe_eval("1.2345 * 10") == 1.2345 * 10
print("Tests pass")
If all you need is a user provided dictionary, a possible better solution is json.loads. The main limitation is that JSON dicts ("objects") require string keys. Also you can only provide literal data, but that is also the case for ast.literal_eval.
I want to change certain values in a json file (nested dicts and arrays). I thought a handy way to do that would be to take advantage of the JSONDecoder.
However, it's not working as I'd expect it to. I've done this exact same approach for getting JSONEncoder to convert np.arrays to lists so it wouldn't break the encoder.
After not getting it to do what I wanted, I thought maybe to try the Decoder instead. Same issue, it never calls default for handling strings it seems. Maybe default is never called when handling a string, just when handling other types of objects?
# key, val are arguments passed in, e.g. ("bar", "2.0rc1")
# Replace the value "2.0rc1" everywhere the "bar" key is found
class StringReplaceDecoder(json.JSONDecoder):
def default(self, obj):
if isinstance(obj, str):
print("Handling obj str: {}".format(obj))
if obj == key:
return val
return json.JSONEncoder.default(self, obj)
json_dump = json.dumps(dict)
json_load = json.loads(json_dump, cls=StringReplaceDecoder)
# Example input
{a:{foo:"", bar:"1.3"}, b:{d:{foo:""}, z:{bar:"1.5"}}}
# Example desired output:
{a:{foo:"", bar:"2.0rc1"}, b:{d:{foo:""}, z:{bar:"2.0rc1"}}}
After finding a solution that worked, I found a far superior one liner that didn't come up in previous google searches.
The correct answer for my problem is from nested_lookup import nested_update
For what it's worth, I also found the object_hook did exactly what I wanted as well:
def val_hook(obj):
return_d = {}
if isinstance(obj, dict):
for k in obj:
if in_key == k:
return_d[k] = in_val
else:
return_d[k] = obj[k]
return return_d
else:
return obj
json_dump = json.dumps(in_dict)
json_load = json.loads(json_dump, object_hook=val_hook)
References
https://gist.github.com/douglasmiranda/5127251
https://github.com/russellballestrini/nested-lookup
https://pypi.org/project/nested-lookup/
Suppose I have a naive class definition:
import yaml
class A:
def __init__(self):
self.abc = 1
self.hidden = 100
self.xyz = 2
def __repr__(self):
return yaml.dump(self)
A()
printing
!!python/object:__main__.A
abc: 1
hidden: 100
xyz: 2
Is there a clean way to remove a line containing hidden: 100 from yaml dump's printed output? The key name hidden is known in advance, but its numeric value may change.
Desired output:
!!python/object:__main__.A
abc: 1
xyz: 2
FYI: This dump is for display only and will not be loaded.
I suppose one can suppress key/value pair with key=hidden with use of yaml.representative. Another way is find hidden: [number] with RegEx in a string output.
I looked at the documentation for pyyaml and did not find a way to achieve your objective. A work-around would be to delete the attribte hidden, call yaml.dump, then add it back in:
def __repr__(self):
hidden = self.hidden
del self.hidden
return yaml.dump(self)
self.hidden = hidden
Taking a step back, why do you want to use yaml for __repr__? Can you just roll your own instead of relying on yaml?
json is mature solution and (at the moment of writing) have much better docs than pyyaml;
I'd use it instead while pyyaml's docs are hard to fully understand. As a bonus, YAML is (almost) superset of JSON, so you'll be able to read your data as YAML without converting it.
However, to easily use all goodies of YAML you will probably have to convert the data to YAML
json module is unable to serialize custom objects by default, but it can be easily extended:
import json
def default(o):
if isinstance(o, A):
result = vars(o).copy()
del result['hidden']
result['__class__'] = o.__class__.__name__
return result
else:
return o
json.dumps(A(), default=default) # => '{"__class__": "A", "xyz": 2, "abc": 1}'
If you don't want to write default=default everywhere you dumps, you can create custom serializer:
dumper = json.JSONEncoder(default=default)
dumper.encode(A()) # => '{"__class__": "A", "xyz": 2, "abc": 1}'
Or, to be able to easily extend it even further via subclassing:
class Dumper(json.JSONEncoder):
__slots__ = ()
def default(self, o):
if isinstance(o, A):
result = vars(o).copy()
del result['hidden']
result['__class__'] = o.__class__.__name__
return result
else:
return super().default(o)
dumper = Dumper()
dumper.encode(A()) # => '{"__class__": "A", "xyz": 2, "abc": 1}'
Note that fields in JSON are unordered.
Also, if you want to use this, I'd advise you not to serialize dict with key __class__, because it might be hard to distinguish it from serialized object.
See it working online
I have a situation with some code where eval() came up as a possible solution. Now I have never had to use eval() before but, I have come across plenty of information about the potential danger it can cause. That said, I'm very wary about using it.
My situation is that I have input being given by a user:
datamap = input('Provide some data here: ')
Where datamap needs to be a dictionary. I searched around and found that eval() could work this out. I thought that I might be able to check the type of the input before trying to use the data and that would be a viable security precaution.
datamap = eval(input('Provide some data here: ')
if not isinstance(datamap, dict):
return
I read through the docs and I am still unclear if this would be safe or not. Does eval evaluate the data as soon as its entered or after the datamap variable is called?
Is the ast module's .literal_eval() the only safe option?
datamap = eval(input('Provide some data here: ')) means that you actually evaluate the code before you deem it to be unsafe or not. It evaluates the code as soon as the function is called. See also the dangers of eval.
ast.literal_eval raises an exception if the input isn't a valid Python datatype, so the code won't be executed if it's not.
Use ast.literal_eval whenever you need eval. You shouldn't usually evaluate literal Python statements.
ast.literal_eval() only considers a small subset of Python's syntax to be valid:
The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.
Passing __import__('os').system('rm -rf /a-path-you-really-care-about') into ast.literal_eval() will raise an error, but eval() will happily delete your files.
Since it looks like you're only letting the user input a plain dictionary, use ast.literal_eval(). It safely does what you want and nothing more.
eval:
This is very powerful, but is also very dangerous if you accept strings to evaluate from untrusted input. Suppose the string being evaluated is "os.system('rm -rf /')" ? It will really start deleting all the files on your computer.
ast.literal_eval:
Safely evaluate an expression node or a string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None, bytes and sets.
Syntax:
eval(expression, globals=None, locals=None)
import ast
ast.literal_eval(node_or_string)
Example:
# python 2.x - doesn't accept operators in string format
import ast
ast.literal_eval('[1, 2, 3]') # output: [1, 2, 3]
ast.literal_eval('1+1') # output: ValueError: malformed string
# python 3.0 -3.6
import ast
ast.literal_eval("1+1") # output : 2
ast.literal_eval("{'a': 2, 'b': 3, 3:'xyz'}") # output : {'a': 2, 'b': 3, 3:'xyz'}
# type dictionary
ast.literal_eval("",{}) # output : Syntax Error required only one parameter
ast.literal_eval("__import__('os').system('rm -rf /')") # output : error
eval("__import__('os').system('rm -rf /')")
# output : start deleting all the files on your computer.
# restricting using global and local variables
eval("__import__('os').system('rm -rf /')",{'__builtins__':{}},{})
# output : Error due to blocked imports by passing '__builtins__':{} in global
# But still eval is not safe. we can access and break the code as given below
s = """
(lambda fc=(
lambda n: [
c for c in
().__class__.__bases__[0].__subclasses__()
if c.__name__ == n
][0]
):
fc("function")(
fc("code")(
0,0,0,0,"KABOOM",(),(),(),"","",0,""
),{}
)()
)()
"""
eval(s, {'__builtins__':{}})
In the above code ().__class__.__bases__[0] nothing but object itself.
Now we instantiated all the subclasses, here our main enter code hereobjective is to find one class named n from it.
We need to code object and function object from instantiated subclasses. This is an alternative way from CPython to access subclasses of object and attach the system.
From python 3.7 ast.literal_eval() is now stricter. Addition and subtraction of arbitrary numbers are no longer allowed. link
Python's eager in its evaluation, so eval(input(...)) (Python 3) will evaluate the user's input as soon as it hits the eval, regardless of what you do with the data afterwards. Therefore, this is not safe, especially when you eval user input.
Use ast.literal_eval.
As an example, entering this at the prompt could be very bad for you:
__import__('os').system('rm -rf /a-path-you-really-care-about')
In recent Python3 ast.literal_eval() no longer parses simple strings, instead you are supposed to use the ast.parse() method to create an AST then interpret it.
This is a complete example of using ast.parse() correctly in Python 3.6+ to evaluate simple arithmetic expressions safely.
import ast, operator, math
import logging
logger = logging.getLogger(__file__)
def safe_eval(s):
def checkmath(x, *args):
if x not in [x for x in dir(math) if not "__" in x]:
raise SyntaxError(f"Unknown func {x}()")
fun = getattr(math, x)
return fun(*args)
binOps = {
ast.Add: operator.add,
ast.Sub: operator.sub,
ast.Mult: operator.mul,
ast.Div: operator.truediv,
ast.Mod: operator.mod,
ast.Pow: operator.pow,
ast.Call: checkmath,
ast.BinOp: ast.BinOp,
}
unOps = {
ast.USub: operator.neg,
ast.UAdd: operator.pos,
ast.UnaryOp: ast.UnaryOp,
}
ops = tuple(binOps) + tuple(unOps)
tree = ast.parse(s, mode='eval')
def _eval(node):
if isinstance(node, ast.Expression):
logger.debug("Expr")
return _eval(node.body)
elif isinstance(node, ast.Str):
logger.debug("Str")
return node.s
elif isinstance(node, ast.Num):
logger.debug("Num")
return node.value
elif isinstance(node, ast.Constant):
logger.info("Const")
return node.value
elif isinstance(node, ast.BinOp):
logger.debug("BinOp")
if isinstance(node.left, ops):
left = _eval(node.left)
else:
left = node.left.value
if isinstance(node.right, ops):
right = _eval(node.right)
else:
right = node.right.value
return binOps[type(node.op)](left, right)
elif isinstance(node, ast.UnaryOp):
logger.debug("UpOp")
if isinstance(node.operand, ops):
operand = _eval(node.operand)
else:
operand = node.operand.value
return unOps[type(node.op)](operand)
elif isinstance(node, ast.Call):
args = [_eval(x) for x in node.args]
r = checkmath(node.func.id, *args)
return r
else:
raise SyntaxError(f"Bad syntax, {type(node)}")
return _eval(tree)
if __name__ == "__main__":
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
logger.addHandler(ch)
assert safe_eval("1+1") == 2
assert safe_eval("1+-5") == -4
assert safe_eval("-1") == -1
assert safe_eval("-+1") == -1
assert safe_eval("(100*10)+6") == 1006
assert safe_eval("100*(10+6)") == 1600
assert safe_eval("2**4") == 2**4
assert safe_eval("sqrt(16)+1") == math.sqrt(16) + 1
assert safe_eval("1.2345 * 10") == 1.2345 * 10
print("Tests pass")
If all you need is a user provided dictionary, a possible better solution is json.loads. The main limitation is that JSON dicts ("objects") require string keys. Also you can only provide literal data, but that is also the case for ast.literal_eval.
I'm running Python 2.7 and I'm trying to create a custom FloatEncoder subclass of JSONEncoder. I've followed many examples such as this but none seem to work. Here is my FloatEncoder class:
class FloatEncoder(JSONEncoder):
def _iterencode(self, obj, markers=None):
if isinstance(obj, float):
return (str(obj) for obj in [obj])
return super(FloatEncoder, self)._iterencode(obj, markers)
And here is where I call json.dumps:
with patch("utils.fileio.FloatEncoder") as float_patch:
for val,res in ((.00123456,'0.0012'),(.00009,'0.0001'),(0.99999,'1.0000'),({'hello':1.00001,'world':[True,1.00009]},'{"world": [true, 1.0001], "hello": 1.0000}')):
untrusted = dumps(val, cls=FloatEncoder)
self.assertTrue(float_patch._iterencode.called)
self.assertEqual(untrusted, res)
The first assertion fails, meaning that _iterencode is not being executed. After reading the JSON documentation,I tried overriding the default() method but that also was not being called.
You seem to be trying to round float values down to 4 decimal points while generating JSON (based on test examples).
JSONEncoder shipping with Python 2.7 does not have have _iterencode method, so that's why it's not getting called. Also a quick glance at json/encoder.py suggests that this class is written in such a way that makes it difficult to change the float encoding behavior. Perhaps, it would be better to separate concerns, and round the floats before doing JSON serialization.
EDIT: Alex Martelli also supplies a monkey-patch solution in a related answer. The problem with that approach is that you're introducing a global modification to json library behavior that may unwittingly affect some other piece of code in your application that was written with assumption that floats were encoded without rounding.
Try this:
from collections import Mapping, Sequence
from unittest import TestCase, main
from json import dumps
def round_floats(o):
if isinstance(o, float):
return round(o, 4)
elif isinstance(o, basestring):
return o
elif isinstance(o, Sequence):
return [round_floats(item) for item in o]
elif isinstance(o, Mapping):
return dict((key, round_floats(value)) for key, value in o.iteritems())
else:
return o
class TestFoo(TestCase):
def test_it(self):
for val, res in ((.00123456, '0.0012'),
(.00009, '0.0001'),
(0.99999, '1.0'),
({'hello': 1.00001, 'world': [True, 1.00009]},
'{"world": [true, 1.0001], "hello": 1.0}')):
untrusted = dumps(round_floats(val))
self.assertEqual(untrusted, res)
if __name__ == '__main__':
main()
Don't define _iterencode, define default, as shown in the third answer on that page.