I want to be able to get the length of the _fields member of a namedtuple from another module. However, it is flagged as protected.
The workaround I have is as follows:
MyTuple = namedtuple(
'MyTuple',
'a b'
)
"""MyTuple description
Attributes:
a (float): A descrip
b (float): B descrip
"""
NUM_MY_TUPLE_FIELDS = len(MyTuple._fields)
Then I import NUM_MY_TUPLE_FIELDS from the external module.
I was trying to find a way to make the functionality part of the class, such as to extend the namedtuple with a __len__ method. Is there a more pythonic way to get the number of fields in a namedtuple from an external module?
Updated to show the autodoc comments. The protected warning is seen in PyCharm. Originally, in the external module I simply imported MyTuple, then used:
x = len(MyTuple._fields)
I tried the following suggestion and thought it was going to work, but I get the following: TypeError: object of type 'type' has no len().
class MyTuple(typing.MyTuple):
a: float
b: float
"""MyTuple doc
Attributes:
a (float): A doc
b (float): B doc
"""
def __len__(self) -> int:
return len(self._fields)
fmt_str = f"<L {len(MyTuple)}f" # for struct.pack usage
print(fmt_str)
you can use inheritance:
class MyTuple(namedtuple('MyTuple', 'a b c d e f')):
"""MyTuple description
Attributes:
a (float): A description
...
"""
#property
def fields(self):
# _fields is a class level attribute and available via
# MyTuple._fields from external modules
return self._fields
def __len__(self):
# your implementation if you need it
return len(self._fields)
or use typing.NamedTuple if you are using python 3.5+
class MyTuple(typing.NamedTuple):
a: int
# other fields
One way is to use inspect.signature and just count how many parameters the __new__ method requires:
import inspect
n_fields = len(inspect.signature(NTClass).parameters)
This works because typing.NamedTuple disallows overriding the __new__ method, and that is unlikely to change due to the way it is implemented:
>>> import inspect
>>> from typing import NamedTuple
>>> class NTClass(NamedTuple):
... x: int
... y: float
...
>>> len(inspect.signature(NTClass).parameters)
2
It also works for the old collections.namedtuple:
>>> from collections import namedtuple
>>> NTClass = namedtuple("NTClass", "x y")
>>> len(inspect.signature(NTClass).parameters)
2
Related
Suppose I've got a map like function:
def generate(data, per_element):
for element in data:
per_element(element)
How can I add type-hints so that if I call generate(some_data, some_function) where some_data: List[SomeClass], I get a warning if SomeClass is missing a field used by some_function?
As an example - with the following code:
def some_function(x):
print(x.value)
some_data: List[int] = [1, 2, 3]
generate(some_data, some_function)
I would like to get a warning that int does not have the attribute value.
Use a type variable to make generate generic in the type of object that data contains and that per_element expects as an argument.
from typing import TypeVar, List, Callable
T = TypeVar('T')
def generate(data: List[T], per_element: Callable[[T], Any]):
for element in data:
per_element(element)
class Foo:
def __init__(self):
self.value = 3
def foo(x: Foo):
print(x.value)
def bar(x: int):
pass
generate([Foo(), Foo()], foo) # OK
# Argument 2 to "generate" has incompatible type "Callable[[Foo], Any]"; expected "Callable[[int], Any]"
generate([1,2,3], foo)
Whatever T is, it has to be the same type for both the list and the function, to ensure that per_element can, in fact, be called on every value in data. The error produced by the second call to generate isn't exactly what you asked for, but it essentially catches the same problem: the list establishes what type T is bound to, and the function doesn't accept the correct type.
If you specifically want to require that T be a type whose instances have a value attribute, it's a bit trickier. It's similar to the use case for Protocol, but that only supports methods (or class attributes in general?), not instance attributes, as far as I know. Perhaps someone else can provide a better answer.
Seems like you're searching for:
def generate(data: List[AClass], per_element):
for element in data:
per_element(element)
So that AClass implements the method you need.
Your class needs the value attribute:
class SomeClass:
value: Any # I used any but use whatever type hint is appropriate
Then using typing.Callable in your function as well as the builtin types. starting with python 3.7 and finally fully implemented in python 3.9 you can use the builtins themselves as well as in python 3.9 you can use parameter specifications
from typing import ParamSpec, TypeVar, Callable
P = ParamSpec("P")
R = TypeVar("R")
def generate(data: list[SomeClass], per_element: Callable[P, R]) -> None:
for element in data:
per_element(element)
Then in some_function using the class type hint and None return variable:
def some_function(x: SomeClass) -> None:
print(x.value)
Unfortunately I have to load a dictionary containing an invalid name (which I can't change):
dict = {..., "invalid-name": 0, ...}
I would like to cast this dictionary into a dataclass object, but I can't define an attribute with this name.
from dataclasses import dataclass
#dataclass
class Dict:
...
invalid-name: int # can't do this
...
The only solution I could find is to change the dictionary key into a valid one right before casting it into a dataclass object:
dict["valid_name"] = dict.pop("invalid-name")
But I would like to avoid using string literals...
Is there any better solution to this?
One solution would be using dict-to-dataclass. As mentioned in its documents it has two options:
1.passing dictionary keys
It's probably quite common that your dataclass fields have the same names as the dictionary keys they map to but in case they don't, you can pass the dictionary key as the first argument (or the dict_key keyword argument) to field_from_dict:
#dataclass
class MyDataclass(DataclassFromDict):
name_in_dataclass: str = field_from_dict("nameInDictionary")
origin_dict = {
"nameInDictionary": "field value"
}
dataclass_instance = MyDataclass.from_dict(origin_dict)
>>> dataclass_instance.name_in_dataclass
"field value"
Custom converters
If you need to convert a dictionary value that isn't covered by the defaults, you can pass in a converter function using field_from_dict's converter parameter:
def yes_no_to_bool(yes_no: str) -> bool:
return yes_no == "yes"
#dataclass
class MyDataclass(DataclassFromDict):
is_yes: bool = field_from_dict(converter=yes_no_to_bool)
dataclass_instance = MyDataclass.from_dict({"is_yes": "yes"})
>>> dataclass_instance.is_yes
True
The following code allow to filter the nonexistent keys :
import dataclasses
#dataclasses.dataclass
class ClassDict:
valid-name0: str
valid-name1: int
...
dict = {..., "invalid-name": 0, ...}
dict = {k:v for k,v in dict.items() if k in tuple(e.name for e in dataclasses.fields(ClassDict).keys())}
However, I'm sure there should be a better way to do it since this is a bit hacky.
I would define a from_dict class method anyway, which would be a natural place to make the change.
#dataclass
class MyDict:
...
valid_name: int
...
#classmethod
def from_dict(cls, d):
d['valid_name'] = d.pop('invalid-name')
return cls(**d)
md = MyDict.from_dict({'invalid-name': 3, ...})
Whether you should modify d in place or do something to avoid unnecessary copies is another matter.
Another option could be to use the dataclass-wizard library, which is likewise a de/serialization library built on top of dataclasses. It should similarly support custom key mappings, as needed in this case.
I've also timed it with the builtin timeit module, and found it to be (on average) about 5x faster than a solution with dict_to_dataclass. I've added the code I used for comparison below.
from dataclasses import dataclass
from timeit import timeit
from typing_extensions import Annotated # Note: in Python 3.9+, can import this from `typing` instead
from dataclass_wizard import JSONWizard, json_key
from dict_to_dataclass import DataclassFromDict, field_from_dict
#dataclass
class ClassDictWiz(JSONWizard):
valid_name: Annotated[int, json_key('invalid-name')]
#dataclass
class ClassDict(DataclassFromDict):
valid_name: int = field_from_dict('invalid-name')
my_dict = {"invalid-name": 0}
n = 100_000
print('dict-to-dataclass: ', round(timeit('ClassDict.from_dict(my_dict)', globals=globals(), number=n), 3))
print('dataclass-wizard: ', round(timeit('ClassDictWiz.from_dict(my_dict)', globals=globals(), number=n), 3))
i1, i2 = ClassDict.from_dict(my_dict), ClassDictWiz.from_dict(my_dict)
# assert we get the same result with both approaches
assert i1.__dict__ == i2.__dict__
Results, on my Mac OS X laptop:
dict-to-dataclass: 0.594
dataclass-wizard: 0.098
This is a bit of a silly thing, but I want to know if there is concise way in Python to define class variables that contain string representations of their own names. For example, one can define:
class foo(object):
bar = 'bar'
baz = 'baz'
baf = 'baf'
Probably a more concise way to write it in terms of lines consumed is:
class foo(object):
bar, baz, baf = 'bar', 'baz', 'baf'
Even there, though, I still have to type each identifier twice, once on each side of the assignment, and the opportunity for typos is rife.
What I want is something like what sympy provides in its var method:
sympy.var('a,b,c')
The above injects into the namespace the variables a, b, and c, defined as the corresponding sympy symbolic variables.
Is there something comparable that would do this for plain strings?
class foo(object):
[nifty thing]('bar', 'baz', 'baf')
EDIT: To note, I want to be able to access these as separate identifiers in code that uses foo:
>>> f = foo(); print(f.bar)
bar
ADDENDUM: Given the interest in the question, I thought I'd provide more context on why I want to do this. I have two use-cases at present: (1) typecodes for a set of custom exceptions (each Exception subclass has a distinct typecode set); and (2) lightweight enum. My desired feature set is:
Only having to type the typecode / enum name (or value) once in the source definition. class foo(object): bar = 'bar' works fine but means I have to type it out twice in-source, which gets annoying for longer names and exposes a typo risk.
Valid typecodes / enum values exposed for IDE autocomplete.
Values stored internally as comprehensible strings:
For the Exception subclasses, I want to be able to define myError.__str__ as just something like return self.typecode + ": " + self.message + " (" + self.source + ")", without having to do a whole lot of dict-fu to back-reference an int value of self.typecode to a comprehensible and meaningful string.
For the enums, I want to just be able to obtain widget as output from e = myEnum.widget; print(e), again without a lot of dict-fu.
I recognize this will increase overhead. My application is not speed-sensitive (GUI-based tool for driving a separate program), so I don't think this will matter at all.
Straightforward membership testing, by also including (say) a frozenset containing all of the typecodes / enum string values as myError.typecodes/myEnum.E classes. This addresses potential problems from accidental (or intentional.. but why?!) use of an invalid typecode / enum string via simple sanity checks like if not enumVal in myEnum.E: raise(ValueError('Invalid enum value: ' + str(enumVal))).
Ability to import individual enum / exception subclasses via, say, from errmodule import squirrelerror, to avoid cluttering the namespace of the usage environment with non-relevant exception subclasses. I believe this prohibits any solutions requiring post-twiddling on the module level like what Sinux proposed.
For the enum use case, I would rather avoid introducing an additional package dependency since I don't (think I) care about any extra functionality available in the official enum class. In any event, it still wouldn't resolve #1.
I've already figured out implementation I'm satisfied with for all of the above but #1. My interest in a solution to #1 (without breaking the others) is partly a desire to typo-proof entry of the typecode / enum values into source, and partly plain ol' laziness. (Says the guy who just typed up a gigantic SO question on the topic.)
I recommend using collections.namedtuple:
Example:
>>> from collections import namedtuple as nifty_thing
>>> Data = nifty_thing("Data", ["foo", "bar", "baz"])
>>> data = Data(foo=1, bar=2, baz=3)
>>> data.foo
1
>>> data.bar
2
>>> data.baz
3
Side Note: If you are using/on Python 3.x I'd recommend Enum as per #user2357112's comment. This is the standardized approach going forward for Python 3+
Update: Okay so if I understand the OP's exact requirement(s) here I think the only way to do this (and presumably sympy does this too) is to inject the names/variables into the globals() or locals() namespaces. Example:
#!/usr/bin/env python
def nifty_thing(*names):
d = globals()
for name in names:
d[name] = None
nifty_thing("foo", "bar", "baz")
print foo, bar, bar
Output:
$ python foo.py
None None None
NB: I don't really recommend this! :)
Update #2: The other example you showed in your question is implemented like this:
#!/usr/bin/env python
import sys
def nifty_thing(*names):
frame = sys._getframe(1)
locals = frame.f_locals
for name in names:
locals[name] = None
class foo(object):
nifty_thing("foo", "bar", "baz")
f = foo()
print f.foo, f.bar, f.bar
Output:
$ python foo.py
None None None
NB: This is inspired by zope.interface.implements().
current_list = ['bar', 'baz', 'baf']
class foo(object):
"""to be added"""
for i in current_list:
setattr(foo, i, i)
then run this:
>>>f = foo()
>>>print(f.bar)
bar
>>>print(f.baz)
baz
This doesn't work exactly like what you asked for, but it seems like it should do the job:
class AutoNamespace(object):
def __init__(self, names):
try:
# Support space-separated name strings
names = names.split()
except AttributeError:
pass
for name in names:
setattr(self, name, name)
Demo:
>>> x = AutoNamespace('a b c')
>>> x.a
'a'
If you want to do what SymPy does with var, you can, but I would strongly recommend against it. That said, here's a function based on the source code of sympy.var:
def var(names):
from inspect import currentframe
frame = currentframe().f_back
try:
names = names.split()
except AttributeError:
pass
for name in names:
frame.f_globals[name] = name
Demo:
>>> var('foo bar baz')
>>> bar
'bar'
It'll always create global variables, even if you call it from inside a function or class. inspect is used to get at the caller's globals, whereas globals() would get var's own globals.
How about you define the variable as emtpy string and then get their name:
class foo(object):
def __getitem__(self, item):
return item
foo = foo()
print foo['test']
Here's an extension of bman's idea. This has its advantages and disadvantages, but at least it does work with some autocompleters.
class FooMeta(type):
def __getattr__(self, attr):
return attr
def __dir__(self):
return ['bar', 'baz', 'baf']
class foo:
__metaclass__ = FooMeta
This allows access like foo.xxx → 'xxx' for all xxx, but also guides autocomplete through __dir__.
Figured out what I was looking for:
>>> class tester:
... E = frozenset(['this', 'that', 'the', 'other'])
... for s in E:
... exec(str(s) + "='" + str(s) + "'") # <--- THIS
...
>>> tester()
<__main__.tester instance at 0x03018BE8>
>>> t = tester()
>>> t.this
'this'
>>> t.that in tester.E
True
Only have to define the element strings once, and I'm pretty sure it will work for all of my requirements listed in the question. In actual implementation, I plan to encapsulate the str(s) + "='" + str(s) + "'" in a helper function, so that I can just call exec(helper(s)) in the for loop. (I'm pretty sure that the exec has to be placed in the body of the class, not in the helper function, or else the new variables would be injected into the (transitory) scope of the helper function, not that of the class.)
EDIT: Upon detailed testing, this DOES NOT WORK -- the use of exec prevents the introspection of the IDE from knowing of the existence of the created variables.
I think you can achieve a rather beautiful solution using metaclasses, but I'm not fluent enough in using those to present that as an answer, but I do have an option which seems to work rather nicely:
def new_enum(name, *class_members):
"""Builds a class <name> with <class_members> having the name as value."""
return type(name, (object, ), { val : val for val in class_members })
Foo = new_enum('Foo', 'bar', 'baz', 'baf')
This should recreate the class you've given as example, and if you want you can change the inheritance by changing the second parameter of the call to class type(name, bases, dict).
I have a class:
class A:
s = 'some string'
b = <SOME OTHER INSTANCE>
now I want this class to have the functionality of a string whenever it can. That is:
a = A()
print a.b
will print b's value. But I want functions that expect a string (for example replace) to work. For example:
'aaaa'.replace('a', a)
to actually do:
'aaa'.replace('a', a.s)
I tried overidding __get__ but this isn't correct.
I see that you can do this by subclassing str, but is there a way without it?
If you want your class to have the functionality of a string, just extend the built in string class.
>>> class A(str):
... b = 'some other value'
...
>>> a = A('x')
>>> a
'x'
>>> a.b
'some other value'
>>> 'aaa'.replace('a',a)
'xxx'
I found an answer in Subclassing Python tuple with multiple __init__ arguments .
I used Dave's solution and extended str, and then added a new function:
def __new__(self,a,b):
s=a
return str.__new__(A,s)
Override __str__ or __unicode__ to set the string representation of an object (Python documentation).
In Javascript it would be:
var newObject = { 'propertyName' : 'propertyValue' };
newObject.propertyName; // returns "propertyValue"
But the same syntax in Python would create a dictionary, and that's not what I want
new_object = {'propertyName': 'propertyValue'}
new_object.propertyName # raises an AttributeError
obj = type('obj', (object,), {'propertyName' : 'propertyValue'})
there are two kinds of type function uses.
Python 3.3 added the SimpleNamespace class for that exact purpose:
>>> from types import SimpleNamespace
>>> obj = SimpleNamespace(propertyName='propertyValue')
>>> obj
namespace(propertyName='propertyValue')
>>> obj.propertyName
'propertyValue'
In addition to the appropriate constructor to build the object, SimpleNamespace defines __repr__ and __eq__ (documented in 3.4) to behave as expected.
Peter's answer
obj = lambda: None
obj.propertyName = 'propertyValue'
I don't know if there's a built-in way to do it, but you can always define a class like this:
class InlineClass(object):
def __init__(self, dict):
self.__dict__ = dict
obj = InlineClass({'propertyName' : 'propertyValue'})
I like Smashery's idea, but Python seems content to let you modify classes on your own:
>>> class Inline(object):
... pass
...
>>> obj = Inline()
>>> obj.test = 1
>>> obj.test
1
>>>
Works just fine in Python 2.5 for me. Note that you do have to do this to a class derived from object - it won't work if you change the line to obj = object.
It is easy in Python to declare a class with an __init__() function that can set up the instance for you, with optional arguments. If you don't specify the arguments you get a blank instance, and if you specify some or all of the arguments you initialize the instance.
I explained it here (my highest-rated answer to date) so I won't retype the explanation. But, if you have questions, ask and I'll answer.
If you just want a generic object whose class doesn't really matter, you can do this:
class Generic(object):
pass
x = Generic()
x.foo = 1
x.bar = 2
x.baz = 3
An obvious extension would be to add an __str__() function that prints something useful.
This trick is nice sometimes when you want a more-convenient dictionary. I find it easier to type x.foo than x["foo"].
SilentGhost had a good answer, but his code actually creates a new object of metaclass type, in other words it creates a class. And classes are objects in Python!
obj = type('obj', (object,), {'propertyName' : 'propertyValue'})
type(obj)
gives
<class 'type'>
To create a new object of a custom or build-in class with dict attributes (aka properties) in one line I'd suggest to just call it:
new_object = type('Foo', (object,), {'name': 'new object'})()
and now
type(new_object)
is
<class '__main__.Foo'>
which means it's an object of class Foo
I hope it helps those who are new to Python.
Another viable option is to use namedtuple:
from collections import namedtuple
message = namedtuple('Message', ['propertyName'], verbose=True)
messages = [
message('propertyValueOne'),
message('propertyValueTwo')
]
class test:
def __setattr__(self,key,value):
return value
myObj = test()
myObj.mykey = 'abc' # set your property and value