I am a newbie to Python. I got some Python sample code from a software vendor who extended their software API with boost.python so we can call them in Python. I am confused with some of the segments, such as:
settings = zoo.AddAnimalSettings(carni_bird_list)
settings.Name = 'birds'
settings.Type = settings.Type.enum.Bird
settings.water_min = 1, units.Litre
settings.food_min = 10, units.Gram
All the variable names are replaced to be these funny things anyway, just for explanation of the general idea.
So here the problem is in the 3rd line. How can we set the variable settings.Type with its sub property settings.Type.enum.Bird, where enum.Bird I suppose is some kind of enum of different kind of animals, which is a sub-property of settings.Type?
I tried doing some test to add one line following the above 5 lines to see if enum.Bird is still there:
settings.Type = settings.Type.enum.Bird
and it works ok.
So for this instance settings, it's sub property Type is not overwritten by its sub property of enum.Bird, it still knows enum.Bird is its sub-property.
Can you advise if I need to implement this line in Python, how can I do that?
I suppose it would be a quite interesting knowledge for people learning Python, so I raised this question here for discussing. I am trying to think in a C++ way, but I didn't figure it out.
I don't really see what's the issue. Consider an Enum defined in python:
import enum
class Type(enum.Enum):
Bird = 0
Cat = 1
The Type.Bird and Type.Cat are instances of the Type class:
>>> Type.Bird
<Type.Bird: 0>
>>> Type.Cat
<Type.Cat: 1>
As such they have access to their own class, which is Type:
>>> Type.Bird.__class__
<enum 'Type'>
Now you can just add a property to the Type class and obtain that behaviour:
class Type(enum.Enum):
Bird = 0
Cat = 1
#property
def enum(self):
return self.__class__
and now you have:
>>> Type.Bird
<Type.Bird: 0>
>>> Type.Bird.enum
<enum 'Type'>
>>> Type.Bird.enum.Bird
<Type.Bird: 0>
>>> Type.Bird.enum.Cat
<Type.Cat: 1>
Note that while the above allows you to write Bird.enum doesn't allow you to access as in Type.enum because this would return the property object.
To obtain the exact behaviour you see in that code you could:
Set the settings.Type attribute to be an instance of Type (possibly an Invalid one) and be done:
def AddAnimalSettings(*args)
settings = MyClass(*args)
settings.Type = Type.Bird
return settings
Replace the use of property with a custom made descriptor that will handle the access via the class too. In this case read the documentation about property which also provides its python code equivalent. The case you have to change is __get__ when obj is None:
class MyProperty(object):
# omissis
def __get__(self, obj, objtype=None):
if obj is None:
return objtype # <-- changed this line
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
Use this as:
class Type(enum.Enum):
Bird = 0
Cat = 1
#MyProperty
def enum(self):
return self.__class__
And now you have:
>>> Type.enum
<enum 'Type'>
so that Type.enum.Bird works.
Related
Problem description
I've been trying to add MyPy support for my implementation of a slightly modified Enum class, where it is possible to define a certain containers for enum values.
My enum and descriptor classes definitions (enums.py file):
from __future__ import annotations
import inspect
from enum import Enum, EnumMeta
from typing import Iterable, Any, Generic, TypeVar
T = TypeVar("T", bound=Iterable[Any])
K = TypeVar("K", bound=dict[Any, Any])
class EnumMapper(Generic[T]):
"""Simple descriptor class that enables definitions of iterable within Enum classes."""
def __init__(self, container: T):
self._container = container
def __get__(self, instance: Any, instance_cls: Any) -> T:
return self._container
class StrictEnumMapper(EnumMapper[K]):
"""
Alias for `EnumMapper` descriptor class that requires all enum values
to be present in definition of mapping before Enum class can be
instantiated.
Only dict-based mappings are supported!
"""
def __init__(self, container: K):
if not issubclass(type(container), dict):
raise ValueError(
f"{StrictEnumMapper.__name__} supports only dict-based containers, {type(container)} provided"
)
super().__init__(container)
class NamespacedEnumMeta(EnumMeta):
"""
Metaclass checking Enum-based class for `StrictEnumMapper` fields
and ensuring that all enum values are provided within such fields.
E.g.
>>> class Items(metaclass=NamespacedEnumMeta):
>>> spam = "spam"
>>> eggs = "eggs"
>>> foo = "foo"
>>>
>>> food_preferences = StrictEnumMapper({
>>> spam: "I like spam!",
>>> eggs: "I really don't like eggs...",
>>> })
will not instantiate and the `RuntimeError` informing about missing mapping
for `Items.foo` will be raised.
This class takes burden of remembering to add new enum values to mapping
off programmers' shoulders by doing it automatically during runtime.
The app will simply not start and inform about a mistake.
"""
def __new__(
mcs,
cls,
bases: tuple[type, ...],
namespace: dict[str, Any], # namespace is actually of _EnumDict type
):
enum_values = [namespace[member] for member in namespace._member_names] # type: ignore
strict_enum_mappers_violated = [
field_name
for (field_name, field) in namespace.items()
if (
inspect.ismethoddescriptor(field)
and issubclass(type(field), StrictEnumMapper)
and not all(enum in field._container for enum in enum_values)
)
]
if strict_enum_mappers_violated:
raise RuntimeError(
f"The following {cls} fields do not contain all possible "
+ f"enum values: {strict_enum_mappers_violated}"
)
return EnumMeta.__new__(mcs, cls, bases, namespace)
class NamespacedEnum(Enum, metaclass=NamespacedEnumMeta):
"""Extension of the basic Enum class, allowing for EnumMapper and StrictEnumMapper usages."""
Example usage of the classes (main.py file):
from enums import NamespacedEnum, StrictEnumMapper
class Food(NamespacedEnum):
spam = "spam"
eggs = "eggs"
foo = "foo"
reactions = StrictEnumMapper(
{
spam: "I like it",
eggs: "I don't like it...",
foo: "I like?",
}
)
if __name__ == "__main__":
print(Food.reactions[Food.eggs.value])
try:
class ActionEnum(NamespacedEnum):
action1 = "action1"
action2 = "action2"
func_for_action = StrictEnumMapper({
action1: lambda: print("Doing action 1"),
})
except RuntimeError as e:
print(f"Properly detected missing enum value in `func_for_action`. Error message: {e}"
Running above should result in:
$ python main.py
I don't like it...
Properly detected missing enum value in `func_for_action`. Error message: The following ActionEnum fields do not contain all possible enum values: ['func_for_action']
Running MyPy (version 0.910) returns following errors:
$ mypy --python-version 3.9 enums.py main.py 1 ✘ NamespacedEnums 11:53:05
main.py:19: error: Value of type "Food" is not indexable
Found 1 error in 1 file (checked 2 source files)
Is there some kind of MyPy magic/annotations hack that would allow me to explicitly inform MyPy that the descriptors are not converted to enum values at runtime? I don't want to use # type: ignore on each line where the "mappers" are used.
EDIT (on 6th Oct 2021):
Just to rephrase my issue: I don't want to "teach" MyPy that it should detect any missing enum values defined in StrictEnumMapper - its metaclass already does it for me. I want MyPy to properly detect type of the "reaction" field in the Food class example above - it should determine that this field is not Food enum but actually a dict.
The rationale of my approach
(this part is optional, read if you want to understand my idea behind this approach and maybe suggest a better solution that I may not be aware of)
In my projects I often define enums that encapsulate set of possible values, e.g. Django ChoiceField (I'm aware of the TextChoices class but for the sake of this issue lets assume that it wasn't the tool I wanted to use), etc.
I've noticed that often there's a need to define some corresponding values/operations for each of the defined enum value. Usually such actions are defined using separate function, that has branching paths for enum values, e.g.
def perform_action(action_identifier: ActionEnum) -> None:
if action_identifier == ActionEnum.action1:
run_action1()
elif action_identifier == ActionEnum.action2:
run_action2()
[...]
else:
raise ValueError("Unrecognized enum value")
The problems arise when we update the enum and add new possible values. Programmers have to go through all the code and update places where the enum is used. I have prepared my own solution for this called NamespacedEnum - a simple class inheriting from Enum class, with slightly extended metaclass. This metaclass, along with my descriptors, allows defining containers within class definition. What's more, the metaclass takes care of ensuring that all enum values exist in the mapper. To make it more clear, the above example of perform_action() function would be rewritten using my class like that:
class MyAction(NamespacedEnum):
action1 = "action1"
action2 = "action2"
[...]
action_for_value = StrictEnumMapper({
action1: run_action1,
action2: run_action2,
[...]
})
def perform_action(action_identifier: ActionEnum) -> None:
ActionEnum.action_for_value[action_identifier.value]()
I find it a lot easier to use this approach in projects that rely heavily on enums. I also think that this approach is better than defining separate containers for enum values outside of enum class definition because of the namespace clutter.
If i understand you correctly, you want to statically check that a dictionary contains a key-value pair for every variant of a given enum.
This approach is probably overkill but you can write a custom mypy plugin to support this:
First, the runtime check:
# file: test.py
from enum import Enum
from typing import TypeVar, Dict, Type
E = TypeVar('E', bound='Enum')
T = TypeVar('T')
# check that all enum variants have an entry in the dict
def enum_extras(enum: Type[E], **kwargs: T) -> Dict[E, T]:
missing = ', '.join(e.name for e in enum if e.name not in kwargs)
assert not missing, f"Missing enum mappings: {missing}"
extras = {
enum[key]: value for key, value in kwargs.items()
}
return extras
# a dummy enum
class MyEnum(Enum):
x = 1
y = 2
# oups, misspelled the key 'y'
names = enum_extras(MyEnum, x="ex", zzz="why")
This raises AssertionError: Missing enum mappings: y at runtime but mypy does not find the error.
Here is the plugin:
# file: plugin.py
from typing import Optional, Callable, Type
from mypy.plugin import Plugin, FunctionContext
from mypy.types import Type as MypyType
# this is needed for mypy to discover the plugin
def plugin(_: str) -> Type[Plugin]:
return MyPlugin
# we define a plugin
class MyPlugin(Plugin):
# this callback looks at every function call
def get_function_hook(self, fullname: str) -> Optional[Callable[[FunctionContext], MypyType]]:
if fullname == 'test.enum_extras':
# return a custom checker for our special function
return enum_extras_plugin
return None
# this is called for every function call of `enum_extras`
def enum_extras_plugin(ctx: FunctionContext) -> MypyType:
# these are the dictionary keys
_, dict_keys = ctx.arg_names
# the first argument of the function call (a function that returns an enum class)
first_argument = ctx.arg_types[0][0]
try:
# the names of the enum variants
enum_keys = first_argument.ret_type.type.names.keys()
except AttributeError:
return ctx.default_return_type
# check that every variant has a key in the dict
missing = enum_keys - set(dict_keys)
if missing:
missing_keys = ', '.join(missing)
# signal an error if a variant is missing
ctx.api.fail(f"Missing value for enum variants: {missing_keys}", ctx.context)
return ctx.default_return_type
To use the plugin we need a mypy.ini:
[mypy]
plugins = plugin.py
And now mypy will find missing keys:
test.py:35: error: Missing value for enum variants: y
Found 1 error in 1 file (checked 1 source file)
Note: I was using python=3.6 and mypy=0.910
Warning: when you pass something unexpected to enum_extras mypy will crash. Some more error handling could help.
I would like to store a bunch of variables under a Python namespace without creating a separate module. I notice that the result of ArgumentParser's parse_args() is a argparse.Namespace object. You can access the arguments through dot-syntax.
from argparse import ArgumentParser
parser = ArgumentParser()
# some arg definitions here...
args = parser.parse_args() # returns a `argparse.Namespace` object
How can I create the equivalent of an argparse.Namespace? I know I can do something similar with a dict but I would like to use dot-syntax. Is there any built-in class that just lets you assign arbitrary attributes?
Starting with python3.3 you can use types.SimpleNamespace.
However an alternative is simply:
class Namespace(object):
pass
namespaceA = Namespace()
namespaceA.x = 1
The full code for SimpleNamespace isn't much longer.
Note that you cannot simply use an object instance:
>>> o = object()
>>> o.x = 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'x'
This is because instances of object do not have a __dict__ attribute:
>>> vars(object())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: vars() argument must have __dict__ attribute
Which means you cannot set the attributes of an instance of object.
Any object subclass that does not have the __slots__ attribute set does have the __dict__ which is used (by default) to store/retrieve attributes:
>>> class Namespace(object):
... pass
...
>>> a = Namespace()
>>> a.x = 1 # same as as.__dict__['a'] = 1
>>> a.__dict__
{'x': 1}
For further information about attribute setting/lookup you should learn about descriptors.
A class can be used as a namespace, where the variables are class members:
class Namespace1:
foo = 'a'
bar = 5
To prevent callers from trying to instantiate, you can use a baseclass like:
class objectless(object):
def __new__(cls, *args, **kwargs):
raise RuntimeError('%s should not be instantiated' % cls)
And use it like:
class Namespace1(objectless):
...
It sounds like you want a python class. See the docs.
Depending on what you want exactly, you can define a bunch of variables as attributes of a class (either a variable of an instance or of the class itself) and access them that way.
If you want "the equivalent of an argparse.Namespace", use argparse.Namespace:
from argparse import Namespace
ns = Namespace(a=1)
print ns.a
If I'm understanding correctly, you want to dynamically add attributes to it. For example, a class parses command-line flags you access them directly like args.verbose, right? If so, you may be thinking of setattr() that lets you add arbitrary attributes.
class Foo(object):
pass
foo = Foo()
setattr(foo, 'ack', 'bar')
print(foo.ack) # prints 'bar'
class UpperAttrMetaclass(type):
var = "test"
def __new__(upperattr_metaclass, future_class_name,
future_class_parents, future_class_attr):
print("hello world")
uppercase_attr = {}
for name, val in future_class_attr.items():
if not name.startswith('__'):
uppercase_attr[name.upper()] = val
else:
uppercase_attr[name] = val
# reuse the type.__new__ method
# this is basic OOP, nothing magic in there
return type.__new__(upperattr_metaclass, future_class_name,
future_class_parents, uppercase_attr)
class Hello(object):
__metaclass__ = UpperAttrMetaclass
bar = "test"
obj = Hello()
print(obj.BAR) # obj has no attribute named BAR
Traceback (most recent call last):
File "E:\python\test.py", line 32, in
print(obj.BAR)
AttributeError: 'Hello' object has no attribute 'BAR'
Why metaclass UpperAttrMetaclass does not work?
In Python3 the way to specify a metaclass has changed from Python2 in an incompatible way.
Since Python 3.0, the way to specify a metaclass is to use the metaclass name as if it were a Named parameter on the class statement itself.
Thus, in the above example, you shuld declare your Hello class as:
class Hello(metaclass=UpperAttrMetaclass):
bar = "test"
Check the documentation at: https://docs.python.org/3.0/whatsnew/3.0.html#changed-syntax
Besides that, as you've noted, putting a __metaclass__ attribute in a c alss body is not an error, but it does nothing at all, but declaring an attribute with that name.
After a couple releases of Python3.x versions, this is the only syntactic change that is incompatible with Python 2 and can't be work-around in a straightforward way so that the code is both Python 2.x and Python 3.x compatible at the same time.
If you need the same code base to run on Python 2 and Python 3, the package named six brings the call with_metaclass which builds a dynamic class base with a syntax that is compatible with both versions.
(https://pythonhosted.org/six/#syntax-compatibility)
Is it possible to check if some class exists? I have class names in a json config file.
I know that I can simply try to create an object by class name string but that is actually a bad idea because the class constructor can do some unexpected stuff while at this point in time I just want to check if my config is valid and all mentioned classes are available.
Is there any way to do it?
EDIT: Also I do understand that u can get all the methods from some module, in my case I am not sure and don't actually care from what module comes the method. It can be from any import statement and I probably don't know where exactly from.
Using eval() leaves the door open for arbitrary code execution, for security's sake it should be avoided.
Especially if you ask for a solution for such a problem here.
Then we can assume that you do not know these risks sufficiently.
import sys
def str_to_class(str):
return reduce(getattr, str.split("."), sys.modules[__name__])
try:
cls = str_to_class(<json-fragment-here>)
except AttributeError:
cls = None
if cls:
obj = cls(...)
else:
# fight against this
This avoids using eval and is approved by several SO users.
Solution is similar to Convert string to Python class object?.
You can parse the source to get all the class names:
from ast import ClassDef, parse
import importlib
import inspect
mod = "test"
mod = importlib.import_module(mod)
p = parse(inspect.getsource(mod))
names = [kls.name for kls in p.body if isinstance(kls, ClassDef)]
Input:
class Foo(object):
pass
class Bar(object):
pass
Output:
['Foo', 'Bar']
Just compare the class names from the config to the names returned.
{set of names in config}.difference(names)
If you want to include imported names you can parse the module it was imported from but depending on how it was imported you can still find cases that won't work:
from ast import ClassDef, parse, ImportFrom
import importlib
import inspect
mod = "test"
mod = importlib.import_module(mod)
p = parse(inspect.getsource(mod))
names = []
for node in p.body:
if isinstance(node, ClassDef):
names.append(node.name)
elif isinstance(node, ImportFrom):
names.extend(imp.name for imp in node.names)
print(names)
Input:
from test2 import Foobar, Barbar, foo
class Foo(object):
pass
class Bar(object):
pass
test2:
foo = 123
class Foobar(object):
pass
class Barbar(object):
pass
Output:
['Foobar', 'Barbar', 'Foo', 'Bar']
I tried the built-in type function, which worked for me, but there is maybe a more pythonic way to test for the existence of a class:
import types
def class_exist(className):
result = False
try:
result = (eval("type("+className+")") == types.ClassType)
except NameError:
pass
return result
# this is a test class, it's only purpose is pure existence:
class X:
pass
print class_exist('X')
print class_exist('Y')
The output is
True
False
Of course, this is a basic solution which should be used only with well-known input: the eval function can be a great a back door opener. There is a more reliable (but also compact) solution by wenzul.
Does Python have extension methods like C#? Is it possible to call a method like:
MyRandomMethod()
on existing types like int?
myInt.MyRandomMethod()
You can add whatever methods you like on class objects defined in Python code (AKA monkey patching):
>>> class A(object):
>>> pass
>>> def stuff(self):
>>> print self
>>> A.test = stuff
>>> A().test()
This does not work on builtin types, because their __dict__ is not writable (it's a dictproxy).
So no, there is no "real" extension method mechanism in Python.
It can be done with Forbidden Fruit (https://pypi.python.org/pypi/forbiddenfruit)
Install forbiddenfruit:
pip install forbiddenfruit
Then you can extend built-in types:
>>> from forbiddenfruit import curse
>>> def percent(self, delta):
... return self * (1 + delta / 100)
>>> curse(float, 'percent', percent)
>>> 1.0.percent(5)
1.05
Forbidden Fruit is fundamentally dependent on the C API, it works only on cpython implementations and won’t work on other python implementations, such as Jython, pypy, etc.
not sure if that what you're asking but you can extend existing types and then call whatever you like on the new thing:
class int(int):
def random_method(self):
return 4 # guaranteed to be random
v = int(5) # you'll have to instantiate all you variables like this
v.random_method()
class int(int):
def xkcd(self):
import antigravity
print(42)
>>>v.xkcd()
Traceback (most recent call last):
File "<pyshell#81>", line 1, in <module>
v.xkcd()
AttributeError: 'int' object has no attribute 'xkcd'
c = int(1)
>>> c.random_method()
4
>>> c.xkcd()
42
hope that clarifies your question
The following context manager adds the method like Forbidden Fruit would without the limitations of it. Besides that it has the additional benefit of removing the extension method afterwards:
class extension_method:
def __init__(self, obj, method):
method_name = method.__name__
setattr(obj, method_name, method)
self.obj = obj
self.method_name = method_name
def __enter__(self):
return self.obj
def __exit__(self, type, value, traceback):
# remove this if you want to keep the extension method after context exit
delattr(self.obj, self.method_name)
Usage is as follows:
class C:
pass
def get_class_name(self):
return self.__class__.__name__
with extension_method(C, get_class_name):
assert hasattr(C, 'get_class_name') # the method is added to C
c = C()
print(c.get_class_name()) # prints 'C'
assert not hasattr(C, 'get_class_name') # the method is gone from C
I've had great luck with the method described here:
http://mail.python.org/pipermail/python-dev/2008-January/076194.html
I have no idea if it works on builtins though.
Another option is to override the meta-class. This allows you to, among other things, specify functions that should exist in all classes.
This article starts to discuss it:
http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html