I'm very new to python and I wish I could do . notation to access values of a dict.
Lets say I have test like this:
>>> test = dict()
>>> test['name'] = 'value'
>>> print(test['name'])
value
But I wish I could do test.name to get value. Infact I did it by overriding the __getattr__ method in my class like this:
class JuspayObject:
def __init__(self,response):
self.__dict__['_response'] = response
def __getattr__(self,key):
try:
return self._response[key]
except KeyError,err:
sys.stderr.write('Sorry no key matches')
and this works! when I do:
test.name // I get value.
But the problem is when I just print test alone I get the error as:
'Sorry no key matches'
Why is this happening?
This functionality already exists in the standard libraries, so I recommend you just use their class.
>>> from types import SimpleNamespace
>>> d = {'key1': 'value1', 'key2': 'value2'}
>>> n = SimpleNamespace(**d)
>>> print(n)
namespace(key1='value1', key2='value2')
>>> n.key2
'value2'
Adding, modifying and removing values is achieved with regular attribute access, i.e. you can use statements like n.key = val and del n.key.
To go back to a dict again:
>>> vars(n)
{'key1': 'value1', 'key2': 'value2'}
The keys in your dict should be string identifiers for attribute access to work properly.
Simple namespace was added in Python 3.3. For older versions of the language, argparse.Namespace has similar behaviour.
I assume that you are comfortable in Javascript and want to borrow that kind of syntax... I can tell you by personal experience that this is not a great idea.
It sure does look less verbose and neat; but in the long run it is just obscure. Dicts are dicts, and trying to make them behave like objects with attributes will probably lead to (bad) surprises.
If you need to manipulate the fields of an object as if they were a dictionary, you can always resort to use the internal __dict__ attribute when you need it, and then it is explicitly clear what you are doing. Or use getattr(obj, 'key') to have into account the inheritance structure and class attributes too.
But by reading your example it seems that you are trying something different... As the dot operator will already look in the __dict__ attribute without any extra code.
In addition to this answer, one can add support for nested dicts as well:
from types import SimpleNamespace
class NestedNamespace(SimpleNamespace):
def __init__(self, dictionary, **kwargs):
super().__init__(**kwargs)
for key, value in dictionary.items():
if isinstance(value, dict):
self.__setattr__(key, NestedNamespace(value))
else:
self.__setattr__(key, value)
nested_namespace = NestedNamespace({
'parent': {
'child': {
'grandchild': 'value'
}
},
'normal_key': 'normal value',
})
print(nested_namespace.parent.child.grandchild) # value
print(nested_namespace.normal_key) # normal value
Note that this does not support dot notation for dicts that are somewhere inside e.g. lists.
Could you use a named tuple?
from collections import namedtuple
Test = namedtuple('Test', 'name foo bar')
my_test = Test('value', 'foo_val', 'bar_val')
print(my_test)
print(my_test.name)
__getattr__ is used as a fallback when all other attribute lookup rules have failed. When you try to "print" your object, Python look for a __repr__ method, and since you don't implement it in your class it ends up calling __getattr__ (yes, in Python methods are attributes too). You shouldn't assume which key getattr will be called with, and, most important, __getattr__ must raise an AttributeError if it cannot resolve key.
As a side note: don't use self.__dict__ for ordinary attribute access, just use the plain attribute notation:
class JuspayObject:
def __init__(self,response):
# don't use self.__dict__ here
self._response = response
def __getattr__(self,key):
try:
return self._response[key]
except KeyError,err:
raise AttributeError(key)
Now if your class has no other responsability (and your Python version is >= 2.6 and you don't need to support older versions), you may just use a namedtuple : http://docs.python.org/2/library/collections.html#collections.namedtuple
You can use the built-in method argparse.Namespace():
import argparse
args = argparse.Namespace()
args.name = 'value'
print(args.name)
# 'value'
You can also get the original dict via vars(args).
class convert_to_dot_notation(dict):
"""
Access dictionary attributes via dot notation
"""
__getattr__ = dict.get
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
test = {"name": "value"}
data = convert_to_dot_notation(test)
print(data.name)
You have to be careful when using __getattr__, because it's used for a lot of builtin Python functionality.
Try something like this...
class JuspayObject:
def __init__(self,response):
self.__dict__['_response'] = response
def __getattr__(self, key):
# First, try to return from _response
try:
return self.__dict__['_response'][key]
except KeyError:
pass
# If that fails, return default behavior so we don't break Python
try:
return self.__dict__[key]
except KeyError:
raise AttributeError, key
>>> j = JuspayObject({'foo': 'bar'})
>>> j.foo
'bar'
>>> j
<__main__.JuspayObject instance at 0x7fbdd55965f0>
Here is a simple, handy dot notation helper example that is working with nested items:
def dict_get(data:dict, path:str, default = None):
pathList = re.split(r'\.', path, flags=re.IGNORECASE)
result = data
for key in pathList:
try:
key = int(key) if key.isnumeric() else key
result = result[key]
except:
result = default
break
return result
Usage example:
my_dict = {"test1": "str1", "nested_dict": {"test2": "str2"}, "nested_list": ["str3", {"test4": "str4"}]}
print(dict_get(my_dict, "test1"))
# str1
print(dict_get(my_dict, "nested_dict.test2"))
# str2
print(dict_get(my_dict, "nested_list.1.test4"))
# str4
With a small addition to this answer you can support lists as well:
class NestedNamespace(SimpleNamespace):
def __init__(self, dictionary, **kwargs):
super().__init__(**kwargs)
for key, value in dictionary.items():
if isinstance(value, dict):
self.__setattr__(key, NestedNamespace(value))
elif isinstance(value, list):
self.__setattr__(key, map(NestedNamespace, value))
else:
self.__setattr__(key, value)
2022 answer: I've created the dotwiz package -- this is a fast, tiny library that seems to perform really well in most cases.
>>> from dotwiz import DotWiz
>>> test = DotWiz(hello='world')
>>> test.works = True
>>> test
✫(hello='world', works=True)
>>> test.hello
'world'
>>> assert test.works
This feature is baked into OmegaConf:
from omegaconf import OmegaConf
your_dict = {"k" : "v", "list" : [1, {"a": "1", "b": "2", 3: "c"}]}
adot_dict = OmegaConf.create(your_dict)
print(adot_dict.k)
print(adot_dict.list)
Installation is:
pip install omegaconf
This lib comes in handy for configurations, which it is actually made for:
from omegaconf import OmegaConf
cfg = OmegaConf.load('config.yml')
print(cfg.data_path)
I use the dotted_dict package:
>>> from dotted_dict import DottedDict
>>> test = DottedDict()
>>> test.name = 'value'
>>> print(test.name)
value
Advantages over SimpleNamespace
(See #win's answer.) DottedDict is an actual dict:
>>> isinstance(test, dict)
True
This allows, for example, checking for membership:
>>> 'name' in test
True
whereas for SimpleNamespace you need something much less readable like hasattr(test, 'name').
Don't use DotMap
I found this out the hard way. If you reference a non-member it adds it rather than throwing an error. This can lead to hard to find bugs in code:
>>> from dotmap import DotMap
>>> dm = DotMap()
>>> 'a' in dm
False
>>> x = dm.a
>>> 'a' in dm
True
#!/usr/bin/env python3
import json
from sklearn.utils import Bunch
from collections.abc import MutableMapping
def dotted(inpt: MutableMapping,
*args,
**kwargs
) -> Bunch:
"""
Enables recursive dot notation for ``dict``.
"""
return json.loads(json.dumps(inpt),
object_hook=lambda x:
Bunch(**{**Bunch(), **x}))
You can make hacks adding dot notation to Dicts mostly work, but there are always namespace problems. As in, what does this do?
x = DotDict()
x["values"] = 1989
print(x. values)
I use pydash, which is a Python port of JS's lodash, to do these things a different way when the nesting gets too ugly.
Add a __repr__() method to the class so that you can customize the text to be shown on
print text
Learn more here: https://web.archive.org/web/20121022015531/http://diveintopython.net/object_oriented_framework/special_class_methods2.html
Related
What is the recommended way of serializing a namedtuple to json with the field names retained?
Serializing a namedtuple to json results in only the values being serialized and the field names being lost in translation. I would like the fields also to be retained when json-ized and hence did the following:
class foobar(namedtuple('f', 'foo, bar')):
__slots__ = ()
def __iter__(self):
yield self._asdict()
The above serializes to json as I expect and behaves as namedtuple in other places I use (attribute access etc.,) except with a non-tuple like results while iterating it (which fine for my use case).
What is the "correct way" of converting to json with the field names retained?
If it's just one namedtuple you're looking to serialize, using its _asdict() method will work (with Python >= 2.7)
>>> from collections import namedtuple
>>> import json
>>> FB = namedtuple("FB", ("foo", "bar"))
>>> fb = FB(123, 456)
>>> json.dumps(fb._asdict())
'{"foo": 123, "bar": 456}'
This is pretty tricky, since namedtuple() is a factory which returns a new type derived from tuple. One approach would be to have your class also inherit from UserDict.DictMixin, but tuple.__getitem__ is already defined and expects an integer denoting the position of the element, not the name of its attribute:
>>> f = foobar('a', 1)
>>> f[0]
'a'
At its heart the namedtuple is an odd fit for JSON, since it is really a custom-built type whose key names are fixed as part of the type definition, unlike a dictionary where key names are stored inside the instance. This prevents you from "round-tripping" a namedtuple, e.g. you cannot decode a dictionary back into a namedtuple without some other a piece of information, like an app-specific type marker in the dict {'a': 1, '#_type': 'foobar'}, which is a bit hacky.
This is not ideal, but if you only need to encode namedtuples into dictionaries, another approach is to extend or modify your JSON encoder to special-case these types. Here is an example of subclassing the Python json.JSONEncoder. This tackles the problem of ensuring that nested namedtuples are properly converted to dictionaries:
from collections import namedtuple
from json import JSONEncoder
class MyEncoder(JSONEncoder):
def _iterencode(self, obj, markers=None):
if isinstance(obj, tuple) and hasattr(obj, '_asdict'):
gen = self._iterencode_dict(obj._asdict(), markers)
else:
gen = JSONEncoder._iterencode(self, obj, markers)
for chunk in gen:
yield chunk
class foobar(namedtuple('f', 'foo, bar')):
pass
enc = MyEncoder()
for obj in (foobar('a', 1), ('a', 1), {'outer': foobar('x', 'y')}):
print enc.encode(obj)
{"foo": "a", "bar": 1}
["a", 1]
{"outer": {"foo": "x", "bar": "y"}}
It looks like you used to be able to subclass simplejson.JSONEncoder to make this work, but with the latest simplejson code, that is no longer the case: you have to actually modify the project code. I see no reason why simplejson should not support namedtuples, so I forked the project, added namedtuple support, and I'm currently waiting for my branch to be pulled back into the main project. If you need the fixes now, just pull from my fork.
EDIT: Looks like the latest versions of simplejson now natively support this with the namedtuple_as_object option, which defaults to True.
I wrote a library for doing this: https://github.com/ltworf/typedload
It can go from and to named-tuple and back.
It supports quite complicated nested structures, with lists, sets, enums, unions, default values. It should cover most common cases.
edit: The library also supports dataclass and attr classes.
It's impossible to serialize namedtuples correctly with the native python json library. It will always see tuples as lists, and it is impossible to override the default serializer to change this behaviour. It's worse if objects are nested.
Better to use a more robust library like orjson:
import orjson
from typing import NamedTuple
class Rectangle(NamedTuple):
width: int
height: int
def default(obj):
if hasattr(obj, '_asdict'):
return obj._asdict()
rectangle = Rectangle(width=10, height=20)
print(orjson.dumps(rectangle, default=default))
=>
{
"width":10,
"height":20
}
There is a more convenient solution is to use the decorator (it uses the protected field _fields).
Python 2.7+:
import json
from collections import namedtuple, OrderedDict
def json_serializable(cls):
def as_dict(self):
yield OrderedDict(
(name, value) for name, value in zip(
self._fields,
iter(super(cls, self).__iter__())))
cls.__iter__ = as_dict
return cls
#Usage:
C = json_serializable(namedtuple('C', 'a b c'))
print json.dumps(C('abc', True, 3.14))
# or
#json_serializable
class D(namedtuple('D', 'a b c')):
pass
print json.dumps(D('abc', True, 3.14))
Python 3.6.6+:
import json
from typing import TupleName
def json_serializable(cls):
def as_dict(self):
yield {name: value for name, value in zip(
self._fields,
iter(super(cls, self).__iter__()))}
cls.__iter__ = as_dict
return cls
# Usage:
#json_serializable
class C(NamedTuple):
a: str
b: bool
c: float
print(json.dumps(C('abc', True, 3.14))
It recursively converts the namedTuple data to json.
print(m1)
## Message(id=2, agent=Agent(id=1, first_name='asd', last_name='asd', mail='2#mai.com'), customer=Customer(id=1, first_name='asd', last_name='asd', mail='2#mai.com', phone_number=123123), type='image', content='text', media_url='h.com', la=123123, ls=4512313)
def reqursive_to_json(obj):
_json = {}
if isinstance(obj, tuple):
datas = obj._asdict()
for data in datas:
if isinstance(datas[data], tuple):
_json[data] = (reqursive_to_json(datas[data]))
else:
print(datas[data])
_json[data] = (datas[data])
return _json
data = reqursive_to_json(m1)
print(data)
{'agent': {'first_name': 'asd',
'last_name': 'asd',
'mail': '2#mai.com',
'id': 1},
'content': 'text',
'customer': {'first_name': 'asd',
'last_name': 'asd',
'mail': '2#mai.com',
'phone_number': 123123,
'id': 1},
'id': 2,
'la': 123123,
'ls': 4512313,
'media_url': 'h.com',
'type': 'image'}
The jsonplus library provides a serializer for NamedTuple instances. Use its compatibility mode to output simple objects if needed, but prefer the default as it is helpful for decoding back.
This is an old question. However:
A suggestion for all those with the same question, think carefully about using any of the private or internal features of the NamedTuple because they have before and will change again over time.
For example, if your NamedTuple is a flat value object and you're only interested in serializing it and not in cases where it is nested into another object, you could avoid the troubles that would come up with __dict__ being removed or _as_dict() changing and just do something like (and yes this is Python 3 because this answer is for the present):
from typing import NamedTuple
class ApiListRequest(NamedTuple):
group: str="default"
filter: str="*"
def to_dict(self):
return {
'group': self.group,
'filter': self.filter,
}
def to_json(self):
return json.dumps(self.to_dict())
I tried to use the default callable kwarg to dumps in order to do the to_dict() call if available, but that didn't get called as the NamedTuple is convertible to a list.
Here is my take on the problem. It serializes the NamedTuple, takes care of folded NamedTuples and Lists inside of them
def recursive_to_dict(obj: Any) -> dict:
_dict = {}
if isinstance(obj, tuple):
node = obj._asdict()
for item in node:
if isinstance(node[item], list): # Process as a list
_dict[item] = [recursive_to_dict(x) for x in (node[item])]
elif getattr(node[item], "_asdict", False): # Process as a NamedTuple
_dict[item] = recursive_to_dict(node[item])
else: # Process as a regular element
_dict[item] = (node[item])
return _dict
simplejson.dump() instead of json.dump does the job. It may be slower though.
I want to create a dictionary that would store its keys as string variables,
but would still be able to retrieve its values using enums.
The reason for that is that my code is sharing this dictionary with other programs and processes, by saving it into a database and by sending it to other applications using REST. In order to make it easier to manage the dictionary itself, I would like the keys to be of type string. However, I would appreciate it if I would be able to get values by enums.
I don't want to access values simply by using strings since that would be using "magic strings".
The option of constants variables is just ugly, and it seems like enum is the nicest solution to this problem. That being said, I don't want it to use the .value each time because it makes the code longer and uglier.
Is it possible to add support for both strings and enums?
You would need to override the __setitem__ and __getitem__ methods in order to support both string and enum types.
Your goal is to get the method's parameter and cast it into a string if it's an enum. That way would be able to retrieve and store string values using the enum variable.
Your code supposes to look like that:
from collections import UserDict
from enum import Enum
class EnumDict(UserDict):
def __setitem__(self, key, value):
string_key = key.value if isinstance(key, Enum) else key
super().__setitem__(string_key, value)
def __getitem__(self, item):
string_item = item.value if isinstance(item, Enum) else item
return super().__getitem__(string_item)
Pay attention that this would work only if you actually inherit from UserDict.
Please read this excellent blog post "The problem with inheriting from dict and list in Python" in order to understand why I'm using UserDict instead of a regular dict. Otherwise, you would need to override __init__, update, and get methods too.
Pay attention that if you're using python2 (which you shouldn't since it is no longer supported) the above code won't work because UserDict is used as a backward compatibility thing for the time in which you couldn't inherit directly from dict. It does not have the same set of features and capabilities as in python3.
Anyway, if you're using python2 or just don't want to inherit from UserDict,
this is what your code should look like:
from enum import Enum
class EnumDict(dict):
def __init__(self, *args, **kwargs):
self.update(*args, **kwargs)
def update(self, *args, **kwargs):
try:
for key, value in args[0].items():
self.__setitem__(key, value)
except Exception:
super().update(*args, *kwargs)
def __setitem__(self, key, value):
string_key = key.value if isinstance(key, Enum) else key
super().__setitem__(string_key, value)
def __getitem__(self, item):
string_item = item.value if isinstance(item, Enum) else item
return super().__getitem__(string_item)
def get(self, key, default=None):
string_key = key.value if isinstance(key, Enum) else key
return super().get(string_key, default)
To sum things up, the whole idea here is to cast enum into its string value.
When using dict and not UserDict, one should also take care of the
initialization, update and regular get processes - making sure you won't get
stuck with enums instead of regular strings.
Here is an example of how this dictionary works:
In [1]: class Test(Enum):
...: A = "abc"
...: B = "xyz"
...: C = "qwerty"
In [2]: my_dict = EnumDict({"ok": "fine", Test.B: "bla-bla"})
In [3]: my_dict
Out[3]: {'ok': 'fine', 'xyz': 'bla-bla'}
In [4]: my_dict[Test.A] = "hello"
In [5]: my_dict["testing"] = "works"
In [6]: my_dict
Out[6]: {'ok': 'fine', 'xyz': 'bla-bla', 'abc': 'hello', 'testing': 'works'}
In [7]: my_dict.update({"isOkay": "yes", Test.B: "wow"})
In [8]: my_dict
Out[8]: {'ok': 'fine', 'xyz': 'wow', 'abc': 'hello', 'testing': 'works', 'isOkay': 'yes'}
In [9]: my_dict.get(Test.C, "does not exist")
Out[9]: 'does not exist'
In [10]: my_dict.get(Test.A)
Out[10]: 'hello'
In [11]: my_dict[Test.A]
Out[11]: 'hello'
In [12]: my_dict["xyz"]
Out[12]: 'wow'
Combine your Enum with str, then each member will be a string as well as an Enum and you can use a normal dict:
class Test(str, Enum):
A = "abc"
B = "xyz"
C = "qwerty"
my_dict = {"ok": "fine", Test.B: "bla-bla"}
The only difference is the display of the dict when an Enum member is the key:
>>> my_dict
{'ok': 'fine', <Test.B: 'xyz'>: 'bla-bla'}
But it still works just fine:
>>> my_dict['xyz']
'bla-bla'
>>> my_dict[Test.B]
'bla-bla'
This is a bit of a silly thing, but I want to know if there is concise way in Python to define class variables that contain string representations of their own names. For example, one can define:
class foo(object):
bar = 'bar'
baz = 'baz'
baf = 'baf'
Probably a more concise way to write it in terms of lines consumed is:
class foo(object):
bar, baz, baf = 'bar', 'baz', 'baf'
Even there, though, I still have to type each identifier twice, once on each side of the assignment, and the opportunity for typos is rife.
What I want is something like what sympy provides in its var method:
sympy.var('a,b,c')
The above injects into the namespace the variables a, b, and c, defined as the corresponding sympy symbolic variables.
Is there something comparable that would do this for plain strings?
class foo(object):
[nifty thing]('bar', 'baz', 'baf')
EDIT: To note, I want to be able to access these as separate identifiers in code that uses foo:
>>> f = foo(); print(f.bar)
bar
ADDENDUM: Given the interest in the question, I thought I'd provide more context on why I want to do this. I have two use-cases at present: (1) typecodes for a set of custom exceptions (each Exception subclass has a distinct typecode set); and (2) lightweight enum. My desired feature set is:
Only having to type the typecode / enum name (or value) once in the source definition. class foo(object): bar = 'bar' works fine but means I have to type it out twice in-source, which gets annoying for longer names and exposes a typo risk.
Valid typecodes / enum values exposed for IDE autocomplete.
Values stored internally as comprehensible strings:
For the Exception subclasses, I want to be able to define myError.__str__ as just something like return self.typecode + ": " + self.message + " (" + self.source + ")", without having to do a whole lot of dict-fu to back-reference an int value of self.typecode to a comprehensible and meaningful string.
For the enums, I want to just be able to obtain widget as output from e = myEnum.widget; print(e), again without a lot of dict-fu.
I recognize this will increase overhead. My application is not speed-sensitive (GUI-based tool for driving a separate program), so I don't think this will matter at all.
Straightforward membership testing, by also including (say) a frozenset containing all of the typecodes / enum string values as myError.typecodes/myEnum.E classes. This addresses potential problems from accidental (or intentional.. but why?!) use of an invalid typecode / enum string via simple sanity checks like if not enumVal in myEnum.E: raise(ValueError('Invalid enum value: ' + str(enumVal))).
Ability to import individual enum / exception subclasses via, say, from errmodule import squirrelerror, to avoid cluttering the namespace of the usage environment with non-relevant exception subclasses. I believe this prohibits any solutions requiring post-twiddling on the module level like what Sinux proposed.
For the enum use case, I would rather avoid introducing an additional package dependency since I don't (think I) care about any extra functionality available in the official enum class. In any event, it still wouldn't resolve #1.
I've already figured out implementation I'm satisfied with for all of the above but #1. My interest in a solution to #1 (without breaking the others) is partly a desire to typo-proof entry of the typecode / enum values into source, and partly plain ol' laziness. (Says the guy who just typed up a gigantic SO question on the topic.)
I recommend using collections.namedtuple:
Example:
>>> from collections import namedtuple as nifty_thing
>>> Data = nifty_thing("Data", ["foo", "bar", "baz"])
>>> data = Data(foo=1, bar=2, baz=3)
>>> data.foo
1
>>> data.bar
2
>>> data.baz
3
Side Note: If you are using/on Python 3.x I'd recommend Enum as per #user2357112's comment. This is the standardized approach going forward for Python 3+
Update: Okay so if I understand the OP's exact requirement(s) here I think the only way to do this (and presumably sympy does this too) is to inject the names/variables into the globals() or locals() namespaces. Example:
#!/usr/bin/env python
def nifty_thing(*names):
d = globals()
for name in names:
d[name] = None
nifty_thing("foo", "bar", "baz")
print foo, bar, bar
Output:
$ python foo.py
None None None
NB: I don't really recommend this! :)
Update #2: The other example you showed in your question is implemented like this:
#!/usr/bin/env python
import sys
def nifty_thing(*names):
frame = sys._getframe(1)
locals = frame.f_locals
for name in names:
locals[name] = None
class foo(object):
nifty_thing("foo", "bar", "baz")
f = foo()
print f.foo, f.bar, f.bar
Output:
$ python foo.py
None None None
NB: This is inspired by zope.interface.implements().
current_list = ['bar', 'baz', 'baf']
class foo(object):
"""to be added"""
for i in current_list:
setattr(foo, i, i)
then run this:
>>>f = foo()
>>>print(f.bar)
bar
>>>print(f.baz)
baz
This doesn't work exactly like what you asked for, but it seems like it should do the job:
class AutoNamespace(object):
def __init__(self, names):
try:
# Support space-separated name strings
names = names.split()
except AttributeError:
pass
for name in names:
setattr(self, name, name)
Demo:
>>> x = AutoNamespace('a b c')
>>> x.a
'a'
If you want to do what SymPy does with var, you can, but I would strongly recommend against it. That said, here's a function based on the source code of sympy.var:
def var(names):
from inspect import currentframe
frame = currentframe().f_back
try:
names = names.split()
except AttributeError:
pass
for name in names:
frame.f_globals[name] = name
Demo:
>>> var('foo bar baz')
>>> bar
'bar'
It'll always create global variables, even if you call it from inside a function or class. inspect is used to get at the caller's globals, whereas globals() would get var's own globals.
How about you define the variable as emtpy string and then get their name:
class foo(object):
def __getitem__(self, item):
return item
foo = foo()
print foo['test']
Here's an extension of bman's idea. This has its advantages and disadvantages, but at least it does work with some autocompleters.
class FooMeta(type):
def __getattr__(self, attr):
return attr
def __dir__(self):
return ['bar', 'baz', 'baf']
class foo:
__metaclass__ = FooMeta
This allows access like foo.xxx → 'xxx' for all xxx, but also guides autocomplete through __dir__.
Figured out what I was looking for:
>>> class tester:
... E = frozenset(['this', 'that', 'the', 'other'])
... for s in E:
... exec(str(s) + "='" + str(s) + "'") # <--- THIS
...
>>> tester()
<__main__.tester instance at 0x03018BE8>
>>> t = tester()
>>> t.this
'this'
>>> t.that in tester.E
True
Only have to define the element strings once, and I'm pretty sure it will work for all of my requirements listed in the question. In actual implementation, I plan to encapsulate the str(s) + "='" + str(s) + "'" in a helper function, so that I can just call exec(helper(s)) in the for loop. (I'm pretty sure that the exec has to be placed in the body of the class, not in the helper function, or else the new variables would be injected into the (transitory) scope of the helper function, not that of the class.)
EDIT: Upon detailed testing, this DOES NOT WORK -- the use of exec prevents the introspection of the IDE from knowing of the existence of the created variables.
I think you can achieve a rather beautiful solution using metaclasses, but I'm not fluent enough in using those to present that as an answer, but I do have an option which seems to work rather nicely:
def new_enum(name, *class_members):
"""Builds a class <name> with <class_members> having the name as value."""
return type(name, (object, ), { val : val for val in class_members })
Foo = new_enum('Foo', 'bar', 'baz', 'baf')
This should recreate the class you've given as example, and if you want you can change the inheritance by changing the second parameter of the call to class type(name, bases, dict).
What is the recommended way of serializing a namedtuple to json with the field names retained?
Serializing a namedtuple to json results in only the values being serialized and the field names being lost in translation. I would like the fields also to be retained when json-ized and hence did the following:
class foobar(namedtuple('f', 'foo, bar')):
__slots__ = ()
def __iter__(self):
yield self._asdict()
The above serializes to json as I expect and behaves as namedtuple in other places I use (attribute access etc.,) except with a non-tuple like results while iterating it (which fine for my use case).
What is the "correct way" of converting to json with the field names retained?
If it's just one namedtuple you're looking to serialize, using its _asdict() method will work (with Python >= 2.7)
>>> from collections import namedtuple
>>> import json
>>> FB = namedtuple("FB", ("foo", "bar"))
>>> fb = FB(123, 456)
>>> json.dumps(fb._asdict())
'{"foo": 123, "bar": 456}'
This is pretty tricky, since namedtuple() is a factory which returns a new type derived from tuple. One approach would be to have your class also inherit from UserDict.DictMixin, but tuple.__getitem__ is already defined and expects an integer denoting the position of the element, not the name of its attribute:
>>> f = foobar('a', 1)
>>> f[0]
'a'
At its heart the namedtuple is an odd fit for JSON, since it is really a custom-built type whose key names are fixed as part of the type definition, unlike a dictionary where key names are stored inside the instance. This prevents you from "round-tripping" a namedtuple, e.g. you cannot decode a dictionary back into a namedtuple without some other a piece of information, like an app-specific type marker in the dict {'a': 1, '#_type': 'foobar'}, which is a bit hacky.
This is not ideal, but if you only need to encode namedtuples into dictionaries, another approach is to extend or modify your JSON encoder to special-case these types. Here is an example of subclassing the Python json.JSONEncoder. This tackles the problem of ensuring that nested namedtuples are properly converted to dictionaries:
from collections import namedtuple
from json import JSONEncoder
class MyEncoder(JSONEncoder):
def _iterencode(self, obj, markers=None):
if isinstance(obj, tuple) and hasattr(obj, '_asdict'):
gen = self._iterencode_dict(obj._asdict(), markers)
else:
gen = JSONEncoder._iterencode(self, obj, markers)
for chunk in gen:
yield chunk
class foobar(namedtuple('f', 'foo, bar')):
pass
enc = MyEncoder()
for obj in (foobar('a', 1), ('a', 1), {'outer': foobar('x', 'y')}):
print enc.encode(obj)
{"foo": "a", "bar": 1}
["a", 1]
{"outer": {"foo": "x", "bar": "y"}}
It looks like you used to be able to subclass simplejson.JSONEncoder to make this work, but with the latest simplejson code, that is no longer the case: you have to actually modify the project code. I see no reason why simplejson should not support namedtuples, so I forked the project, added namedtuple support, and I'm currently waiting for my branch to be pulled back into the main project. If you need the fixes now, just pull from my fork.
EDIT: Looks like the latest versions of simplejson now natively support this with the namedtuple_as_object option, which defaults to True.
I wrote a library for doing this: https://github.com/ltworf/typedload
It can go from and to named-tuple and back.
It supports quite complicated nested structures, with lists, sets, enums, unions, default values. It should cover most common cases.
edit: The library also supports dataclass and attr classes.
It's impossible to serialize namedtuples correctly with the native python json library. It will always see tuples as lists, and it is impossible to override the default serializer to change this behaviour. It's worse if objects are nested.
Better to use a more robust library like orjson:
import orjson
from typing import NamedTuple
class Rectangle(NamedTuple):
width: int
height: int
def default(obj):
if hasattr(obj, '_asdict'):
return obj._asdict()
rectangle = Rectangle(width=10, height=20)
print(orjson.dumps(rectangle, default=default))
=>
{
"width":10,
"height":20
}
There is a more convenient solution is to use the decorator (it uses the protected field _fields).
Python 2.7+:
import json
from collections import namedtuple, OrderedDict
def json_serializable(cls):
def as_dict(self):
yield OrderedDict(
(name, value) for name, value in zip(
self._fields,
iter(super(cls, self).__iter__())))
cls.__iter__ = as_dict
return cls
#Usage:
C = json_serializable(namedtuple('C', 'a b c'))
print json.dumps(C('abc', True, 3.14))
# or
#json_serializable
class D(namedtuple('D', 'a b c')):
pass
print json.dumps(D('abc', True, 3.14))
Python 3.6.6+:
import json
from typing import TupleName
def json_serializable(cls):
def as_dict(self):
yield {name: value for name, value in zip(
self._fields,
iter(super(cls, self).__iter__()))}
cls.__iter__ = as_dict
return cls
# Usage:
#json_serializable
class C(NamedTuple):
a: str
b: bool
c: float
print(json.dumps(C('abc', True, 3.14))
It recursively converts the namedTuple data to json.
print(m1)
## Message(id=2, agent=Agent(id=1, first_name='asd', last_name='asd', mail='2#mai.com'), customer=Customer(id=1, first_name='asd', last_name='asd', mail='2#mai.com', phone_number=123123), type='image', content='text', media_url='h.com', la=123123, ls=4512313)
def reqursive_to_json(obj):
_json = {}
if isinstance(obj, tuple):
datas = obj._asdict()
for data in datas:
if isinstance(datas[data], tuple):
_json[data] = (reqursive_to_json(datas[data]))
else:
print(datas[data])
_json[data] = (datas[data])
return _json
data = reqursive_to_json(m1)
print(data)
{'agent': {'first_name': 'asd',
'last_name': 'asd',
'mail': '2#mai.com',
'id': 1},
'content': 'text',
'customer': {'first_name': 'asd',
'last_name': 'asd',
'mail': '2#mai.com',
'phone_number': 123123,
'id': 1},
'id': 2,
'la': 123123,
'ls': 4512313,
'media_url': 'h.com',
'type': 'image'}
The jsonplus library provides a serializer for NamedTuple instances. Use its compatibility mode to output simple objects if needed, but prefer the default as it is helpful for decoding back.
This is an old question. However:
A suggestion for all those with the same question, think carefully about using any of the private or internal features of the NamedTuple because they have before and will change again over time.
For example, if your NamedTuple is a flat value object and you're only interested in serializing it and not in cases where it is nested into another object, you could avoid the troubles that would come up with __dict__ being removed or _as_dict() changing and just do something like (and yes this is Python 3 because this answer is for the present):
from typing import NamedTuple
class ApiListRequest(NamedTuple):
group: str="default"
filter: str="*"
def to_dict(self):
return {
'group': self.group,
'filter': self.filter,
}
def to_json(self):
return json.dumps(self.to_dict())
I tried to use the default callable kwarg to dumps in order to do the to_dict() call if available, but that didn't get called as the NamedTuple is convertible to a list.
Here is my take on the problem. It serializes the NamedTuple, takes care of folded NamedTuples and Lists inside of them
def recursive_to_dict(obj: Any) -> dict:
_dict = {}
if isinstance(obj, tuple):
node = obj._asdict()
for item in node:
if isinstance(node[item], list): # Process as a list
_dict[item] = [recursive_to_dict(x) for x in (node[item])]
elif getattr(node[item], "_asdict", False): # Process as a NamedTuple
_dict[item] = recursive_to_dict(node[item])
else: # Process as a regular element
_dict[item] = (node[item])
return _dict
simplejson.dump() instead of json.dump does the job. It may be slower though.
JavaScript has object literals, e.g.
var p = {
name: "John Smith",
age: 23
}
and .NET has anonymous types, e.g.
var p = new { Name = "John Smith", Age = 23}; // C#
Something similar can be emulated in Python by (ab)using named arguments:
class literal(object):
def __init__(self, **kwargs):
for (k,v) in kwargs.iteritems():
self.__setattr__(k, v)
def __repr__(self):
return 'literal(%s)' % ', '.join('%s = %r' % i for i in sorted(self.__dict__.iteritems()))
def __str__(self):
return repr(self)
Usage:
p = literal(name = "John Smith", age = 23)
print p # prints: literal(age = 23, name = 'John Smith')
print p.name # prints: John Smith
But is this kind of code considered to be Pythonic?
Why not just use a dictionary?
p = {'name': 'John Smith', 'age': 23}
print p
print p['name']
print p['age']
Have you considered using a named tuple?
Using your dict notation
>>> from collections import namedtuple
>>> L = namedtuple('literal', 'name age')(**{'name': 'John Smith', 'age': 23})
or keyword arguments
>>> L = namedtuple('literal', 'name age')(name='John Smith', age=23)
>>> L
literal(name='John Smith', age=23)
>>> L.name
'John Smith'
>>> L.age
23
It is possible to wrap this behaviour into a function easily enough
def literal(**kw):
return namedtuple('literal', kw)(**kw)
the lambda equivalent would be
literal = lambda **kw: namedtuple('literal', kw)(**kw)
but personally I think it's silly giving names to "anonymous" functions
From ActiveState:
class Bunch:
def __init__(self, **kwds):
self.__dict__.update(kwds)
# that's it! Now, you can create a Bunch
# whenever you want to group a few variables:
point = Bunch(datum=y, squared=y*y, coord=x)
# and of course you can read/write the named
# attributes you just created, add others, del
# some of them, etc, etc:
if point.squared > threshold:
point.isok = 1
I don't see anything wrong with creating "anonymous" classes/instances. It's often very convienient to create one with simple function call in one line of code. I personally use something like this:
def make_class( *args, **attributes ):
"""With fixed inability of using 'name' and 'bases' attributes ;)"""
if len(args) == 2:
name, bases = args
elif len(args) == 1:
name, bases = args[0], (object, )
elif not args:
name, bases = "AnonymousClass", (object, )
return type( name, bases, attributes )
obj = make_class( something = "some value" )()
print obj.something
For creating dummy objects it works just fine. Namedtuple is ok, but is immutable, which can be inconvenient at times. And dictionary is... well, a dictionary, but there are situations when you have to pass something with __getattr__ defined, instead of __getitem__.
I don't know whether it's pythonic or not, but it sometimes speeds things up and for me it's good enough reason to use it (sometimes).
I'd say that the solution you implemented looks pretty Pythonic; that being said, types.SimpleNamespace (documented here) already wraps this functionality:
from types import SimpleNamespace
p = SimpleNamespace(name = "John Smith", age = 23)
print(p)
From the Python IAQ:
As of Python 2.3 you can use the syntax
dict(a=1, b=2, c=3, dee=4)
which is good enough as far as I'm concerned. Before Python 2.3 I used the one-line function
def Dict(**dict): return dict
I think object literals make sense in JavaScript for two reasons:
In JavaScript, objects are only way to create a “thing” with string-index properties. In Python, as noted in another answer, the dictionary type does that.
JavaScript‘s object system is prototype-based. There’s no such thing as a class in JavaScript (although it‘s coming in a future version) — objects have prototype objects instead of classes. Thus it’s natural to create an object “from nothing”, via a literal, because all objects only require the built-in root object as a prototype. In Python, every object has a class — you’re sort of expected to use objects for things where you’d have multiple instances, rather than just for one-offs.
Thus no, object literals aren’t Pythonic, but they are JavaScripthonic.
A simple dictionary should be enough for most cases.
If you are looking for a similar API to the one you indicated for the literal case, you can still use dictionaries and simply override the special __getattr__ function:
class CustomDict(dict):
def __getattr__(self, name):
return self[name]
p = CustomDict(user='James', location='Earth')
print p.user
print p.location
Note: Keep in mind though that contrary to namedtuples, fields are not validated and you are in charge of making sure your arguments are sane. Arguments such as p['def'] = 'something' are tolerated inside a dictionary but you will not be able to access them via p.def.