I am creating a class for retrieving details about a computer such as host_name, kernel_version, bios_version, and so on. Some details are more expensive to collect than others so I have a get_* function to retrieve them, but keep the results cached in the object if they are needed again. I am considering implementing them to look like a dictionary object so the kernel version can be retrieved as so:
system = System()
kver = system['kernel_version']
This will call the instance method get_kernel_version(self) internally to retrieve the data. If the kernel version is retrieved a second time from the above instantiated object, it will returned the cached result from the original call to get_kernel_version(self). Note that all these key/value pairs are read-only, there are a fixed number of them based on the available get_* methods, and no new keys can be added later so it doesn't feel like a regular dictionary. There also shouldn't be a need to call something like the values() function which would simply cause all the get_* functions to be needlessly hit. Also, the syntax is a little more verbose than I'd like. Using system.kernel_version instead seems more natural for this use case.
I'm considering whether a better approach is to use dynamic attributes on a class instance. However, I need a natural way to retrieve a list of all attributes, but not the internal methods supporting them. I would probably use the __dir__ special method to return a list similar the keys() list of the dictionary. I would want to see kernel_version and host_name in the list, but not __class__ or get_kernel_version. This seems to go against recommended practice for the definition of __dir__ so I'm not sure if this is the right approach to use.
I could return a proxy class instance whose sole job calls back to a concrete class with the get_* functions when it doesn't have the appropriate attribute already defined.
Here is an example of a version I'm experimenting with implementing the dictionary approach:
class System(dict):
def __getitem__(self, key):
try:
return getattr(self, 'get_'+key)()
except AttributeError as ex:
raise KeyError(ex.message)
def __setitem__(self, key, value):
raise Exception('Read-only')
def __delitem__(self, key, value):
raise Exception('Read-only')
def keys(self):
return [ x[4:] for x in dir(self) if x.startswith('get_') ]
def get_host_name(self):
return 'localhost'
def get_kernel_version(self):
return '4.7.0'
system = System()
print repr(system.keys())
for key in system.keys():
print '{0}: {1}'.format(key, system[key])
try:
system['bios']
except Exception as ex:
print str(ex)
try:
system['kernel_version'] = '5.0'
except Exception as ex:
print str(ex)
Which produced the following output:
['host_name', 'kernel_version']
host_name: localhost
kernel_version: 4.7.0
"'System' object has no attribute 'get_bios'"
Read-only
The code above does not yet implement the caching of values yet, but that is easy to add. However, it's feeling more like I should be doing this as attributes. I am just not sure if when doing so I should abuse __dir__ to emulate the same functionality above I get with keys().
Should I stick with emulating a read-only dictionary or present a class instance with dynamic attributes?
I think sticking with the read-only dictionary subclass approach you're using is fine. However your implementation could be improved somewhat by creating a generic read-only dictionary superclass from which to derive your specific subclass, and using a metaclass to create the value returned by the keys() method. Doing both is illustrated below.
This means you don't have to "abuse" dir() (there's no such thing as a __dir__ attribute) any longer. You can also use reuse the generic MetaReadonlyDict and ReadonlyDict classes to create other similar types.
class MetaReadonlyDict(type):
def __new__(mcls, classname, bases, classdict):
classobj = type.__new__(mcls, classname, bases, classdict)
prefix = classdict['prefix']
_keys = set(name[len(prefix):] for name in classdict
if name.startswith(prefix))
setattr(classobj, 'keys', lambda self: list(_keys)) # define keys()
return classobj
class ReadonlyDict(dict):
def __getitem__(self, key):
try:
return getattr(self, self.prefix + key)()
except AttributeError as ex:
raise Exception(
"{} object has no {!r} key".format(self.__class__.__name__, key))
def __setitem__(self, key, value):
verb = "redefined" if key in self else "defined"
raise Exception(
"{} object is read-only: {!r} "
"key can not be {}".format(self.__class__.__name__, key, verb))
def __delitem__(self, key):
raise Exception(
"{} object is read-only: {!r} "
"key can not be deleted".format(self.__class__.__name__, key))
def __contains__(self, key):
return key in self.keys()
class System(ReadonlyDict):
__metaclass__ = MetaReadonlyDict
prefix = '_get_'
def _get_host_name(self):
return 'localhost'
def _get_kernel_version(self):
return '4.7.0'
system = System()
print('system.keys(): {!r}'.format(system.keys()))
print('values associated with system.keys():')
for key in system.keys():
print(' {!r}: {!r}'.format(key, system[key]))
try:
system['bios']
except Exception as ex:
print(str(ex))
try:
system['kernel_version'] = '5.0'
except Exception as ex:
print(str(ex))
try:
del system['host_name']
except Exception as ex:
print(str(ex))
Output:
system.keys(): ['kernel_version', 'host_name']
values associated with system.keys():
'kernel_version': '4.7.0'
'host_name': 'localhost'
System object has no 'bios' key
System object is read-only: 'kernel_version' key can not be redefined
System object is read-only: 'host_name' key can not be deleted
Related
I have a cache system which use Redis. To interact with it, I use two functions
def get_from_cache(key: str, default=None) -> Any:
"""
Return data from the redis Cache
If the data doesn't exist, return default
"""
data = redis_instance.get(key)
if data is None:
return default
return pickle.loads(data)
def save_to_cache(key: str, value: Any, **kwargs):
"""
Save data into the Redis Cache
If the data is None, delete the redis key
"""
if value is None:
redis_instance.delete(key[0])
else:
pickled_data = pickle.dumps(value)
redis_instance.set(key[0], pickled_data, **kwargs)
Because each cache key can only store specific binarized data, I would like to ensure that when I'm calling save_to_cache with a bad Data Type, MyPy will raise an error.
Since all my key values are stored into a specific file to ensure all my program use the same key for the same things, I was thinking about giving key argument a specific type, something like this:
_T = TypeVar('_T')
def get_from_cache(key: CacheEndpoint[_T], default: Optional[_T]=None) -> _T:
...
def save_to_cache(key: CacheEndpoint[_T], value: Optional[_T], **kwargs):
...
But I still need to use key as a string at some point. So I don't really know ho I could declare such a CacheEndpoint custom type ?
(Another solution would be to make a type check during execution, but it seems quite complicated to do and really inefficient).
I am reading mmdetecton project on github, and I'm so confused with code(I screened out all other irrelevant factors.):
class A:
def __init__(self, a):
self.a = a
self._dict_test = {"b": a}
def __getattr__(self, item):
print("You call __getattr__ !")
return getattr(self._dict_test, item)
test = A(2)
judge = test.get("b", False)
print("a is", test.a)
print("judge is ", judge)
print(test.__dict__)
I didn't declare the get() function in the class, I checked the documentation where it says:
Attribute references are translated to lookups in this dictionary, e.g., m.x is equivalent to m.dict["x"].
So,
（1）I wonder how should my code be interpreted? is it test.__dict__.get(), or test.__dict__['get()']
Has this ever happened to anyone?
（2）why getattr is invoked???
I check the doc where it says
getattr Called when the default attribute access fails with an AttributeError
but isn't get() the dict's function ? why get() fails with an AttributeError?
I am searching for a long time on net. But no use, and thanks in advance!
If you remove the __getattr__ method, you will see an exception:
judge = test.get("b", False)
AttributeError: 'A' object has no attribute 'get'
because there is no get defined in the A class.
With the __getattr__ method, test.get evaluates to getattr(self._dict_test, item) inside that method which is getattr(test._dict_test, "get") which is test._dict_test.get which is the usual dict.get method for test._dict_test, not test.__dict__.
Hey so right now I'm developing backend api using Google ProtoRPC and Endpoints. I'm using the endpoints-proto-datastore library.
So strange things happen here, here is the EndpointsModel class
class AssetData(EndpointsModel):
type = msgprop.EnumProperty(AssetType, indexed=True)
def auth_id_set(self, value):
if ApplicationID.get_by_id(value) is None:
raise endpoints.UnauthorizedException('no auth_id')
self._auth_id = value
#EndpointsAliasProperty(required=True, setter=auth_id_set, property_type=messages.IntegerField)
def auth_id(self):
return self._auth_id
def app_id_set(self, value):
if ApplicationID.query(ApplicationID.app_id == value).get() is None:
raise endpoints.UnauthorizedException('wrong app_id')
self._app_id = value
if self.check_auth_app_id_pair(self.auth_id, value):
self._app_id = value
else:
raise endpoints.BadRequestException('auth_id and app_id mismatch')
#EndpointsAliasProperty(required=True, setter=app_id_set)
def app_id(self):
return self._app_id
#staticmethod
def check_auth_app_id_pair(authen_id, applic_id):
dat = ApplicationID.get_by_id(authen_id)
if dat.app_id != applic_id:
return False
else:
return True
and this is the API class
#endpoints.api(...)
class AssetDatabaseAPI(remote.Service):
#AssetData.query_method(query_fields=('limit', 'order', 'pageToken', 'type', 'auth_id', 'app_id'),
path='assets', http_method='GET', name='assets.getAssetMultiple')
def assets_get_multiple(self, query):
return query
When I deploy this, everytime I tried to access assets.getMultipleAssets it just gives me this error
raised BadRequestError(Key path element must not be incomplete: [ApplicationID: ]). Strangely enough this only happen to method using #Model.query_method, I have other methods using the same system but using #Model.method and it just runs ok.
If I tried it in development server, sometimes it just gives me RuntimeError: BadRequestError('missing key id/name',) then if I just re-save the .py file and retry it, it will work (sometimes not and another re-save can also make the error happens again).
Can anyone tell me my mistake?
Thanks
I think your problem is how you call this method - it's a static method, so you have to access it through class, not the instance (self):
if AssetData.check_auth_app_id_pair(self.auth_id, value):
self._app_id = value
else:
raise endpoints.BadRequestException('auth_id and app_id mismatch')
class MSG_TYPE(IntEnum):
REQUEST = 0
GRANT = 1
RELEASE = 2
FAIL = 3
INQUIRE = 4
YIELD = 5
def __json__(self):
return str(self)
class MessageEncoder(JSONEncoder):
def default(self, obj):
return obj.__json__()
class Message(object):
def __init__(self, msg_type, src, dest, data):
self.msg_type = msg_type
self.src = src
self.dest = dest
self.data = data
def __json__(self):
return dict (\
msg_type=self.msg_type, \
src=self.src, \
dest=self.dest, \
data=self.data,\
)
def ToJSON(self):
return json.dumps(self, cls=MessageEncoder)
msg = Message(msg_type=MSG_TYPE.FAIL, src=0, dest=1, data="hello world")
encoded_msg = msg.ToJSON()
decoded_msg = yaml.load(encoded_msg)
print type(decoded_msg['msg_type'])
When calling print type(decoded_msg['msg_type']), I get the result <type 'str'> instead of the original MSG_TYPTE type. I feel like I should also write a custom json decoder but kind of confused how to do that. Any ideas? Thanks.
When calling print type(decoded_msg['msg_type']), I get the result instead of the original MSG_TYPTE type.
Well, yeah, that's because you told MSG_TYPE to encode itself like this:
def __json__(self):
return str(self)
So, that's obviously going to decode back to a string. If you don't want that, come up with some unique way to encode the values, instead of just encoding their string representations.
The most common way to do this is to encode all of your custom types (including your enum types) using some specialized form of object—just like you've done for Message. For example, you might put a py-type field in the object which encodes the type of your object, and then the meanings of the other fields all depend on the type. Ideally you'll want to abstract out the commonalities instead of hardcoding the same thing 100 times, of course.
I feel like I should also write a custom json decoder but kind of confused how to do that.
Well, have you read the documentation? Where exactly are you confused? You're not going to get a complete tutorial by tacking on a followup to a StackOverflow question…
Assuming you've got a special object structure for all your types, you can use an object_hook to decode the values back to the originals. For example, as a quick hack:
class MessageEncoder(JSONEncoder):
def default(self, obj):
return {'py-type': type(obj).__name__, 'value': obj.__json__()}
class MessageDecoder(JSONDecoder):
def __init__(self, hook=None, *args, **kwargs):
if hook is None: hook = self.hook
return super().__init__(hook, *args, **kwargs)
def hook(self, obj):
if isinstance(obj, dict):
pytype = obj.get('py-type')
if pytype:
t = globals()[pytype]
return t.__unjson__(**obj['value'])
return obj
And now, in your Message class:
#classmethod
def __unjson__(cls, msg_type, src, dest, data):
return cls(msg_type, src, dest, data)
And you need a MSG_TYPE.__json__ that returns a dict, maybe just {'name': str(self)}, then an __unjson__ that does something like getattr(cls, name).
A real-life solution should probably either have the classes register themselves instead of looking them up by name, or should handle looking them up by qualified name instead of just going to globals(). And you may want to let things encode to something other than object—or, if not, to just cram py-type into the object instead of wrapping it in another one. And there may be other ways to make the JSON more compact and/or readable. And a little bit of error handling would be nice. And so on.
You may want to look at the implementation of jsonpickle—not because you want to do the exact same thing it does, but to see how it hooks up all the pieces.
Overriding the default method of the encoder won't matter in this case because your object never gets passed to the method. It's treated as an int.
If you run the encoder on its own:
msg_type = MSG_TYPE.RELEASE
MessageEncoder().encode(msg_type)
You'll get:
'MSG_TYPE.RELEASE'
If you can, use an Enum and you shouldn't have any issues. I also asked a similar question:
How do I serialize IntEnum from enum34 to json in python?
I have browsed the web and pydoc to find my answer without success.
My issue is the following:
I want to define a class with properties, as I would do habitually.
class Container(object):
def __init__(self, content):
assert isinstance(content, dict), "The container can only contain a dictionary"
self._content = content
#property
def name():
try:
return self._content["its_name"]
except KeyError:
raise AttributeError
Now, to access the content's field "its_name", I can use container.name, with a slight modification between the field's name and the attribute's.
I would like to have a default behavior when no specific getter property is set.
I mean, if I call container.description, I want my class to try returning self._content["description"], and throw an AttributeError if there is no such key.
While still calling the specific property for cases like container.name.
Thanks in advance for your help.
This is what the __getattr__ special method is for:
def __getattr__(self, attrname):
# Only called if the other ways of accessing the attribute fail.
try:
return self._content[attrname]
except KeyError:
raise AttributeError
Note that if for some reason you try to retrieve an unknown attribute when the _content attribute doesn't exist, the line
return self._content[attrname]
will recursively invoke __getattr__ in an attempt to get the _content attribute, and that call will call __getattr__, and so on until stack overflow.