Couldn't find a way to phrase the title better, feel free to correct.
I'm pretty new to Python, currently experimenting with the language.. I've noticed that all built-ins types cannot be extended with other members.. I'd like for example to add an each method to the list type, but that would be impossible. I realize that it's designed that way for efficiency reasons, and that most of the built-ins types are implemented in C.
Well, one why I found to override this behavior is be defining a new class, which extends list but otherwise does nothing. Then I can assign the variable list to that new class, and each time I would like to instantiate a new list, I'd use the list constructor, like it would have been used to create the original list type.
class MyList(list):
def each(self, func):
for item in self:
func(item)
list = MyList
my_list = list((1,2,3,4))
my_list.each(lambda x: print(x))
Output:
1
2
3
4
The idea can be generalize of course by defining a method that gets a built it type and returns a class that extends that type. Further more, the original list variable can be saved in another variable to keep an access to it.
Only problem I'm facing right now, is that when you instantiate a list by its literal form, (i.e. [1,2,3,4]), it will still use the original list constructor (or does it?). Is there a way to override this behavior? If the answer is no, do you know of some other way of enabling the user to extend the built-ins types? (just like javascript allows extending built-ins prototypes).
I find this limitation of built-ins (being unable to add members to them) one of Python's drawbacks, making it inconsistent with other user-defined types... Overall I really love the language, and I really don't understand why this limitation is REALLY necessary.
This is a conscientious choice from Python.
Firstly, with regards to patching inbuilt type, this is primarily a design decision and only secondarily an optimization. I have learnt from much lurking on the Python Mailing List that monkey patching on builtin types, although enjoyable for small scripts, serves no good purpose in anything larger.
Libraries, for one, make certain assumptions about types. If it were encouraged to extend default types, many libraries would end up fighting each other. It would also discourage making new types – a deque is a deque, an ordered set is an ordered set, a dictionary is a dictionary and that should be that.
Literal syntax is a particularly important point. If you cannot guarantee that [1, 2, 3] is a list, what can you guarantee? If people could change those behaviours it would have such global impacts as to destroy the stability of a lot of code. There is a reason goto and global variables are discouraged.
There is one particular hack that I am fond of, though. When you see r"hello", this seems to be an extended literal form.
So why not r[1, 2, 3]?
class ListPrefixer:
def __init__(self, typ):
self.typ = typ
def __getitem__(self, args):
return self.typ(args)
class MyList(list):
def each(self, func):
return MyList(func(x) for x in self)
e = ListPrefixer(MyList)
e[1, 2, 3, 4].each(lambda x: x**2)
#>>> [1, 4, 9, 16]
Finally, if you really want to do deep AST hacks, check out MacroPy.
Related
Couldn't find a way to phrase the title better, feel free to correct.
I'm pretty new to Python, currently experimenting with the language.. I've noticed that all built-ins types cannot be extended with other members.. I'd like for example to add an each method to the list type, but that would be impossible. I realize that it's designed that way for efficiency reasons, and that most of the built-ins types are implemented in C.
Well, one why I found to override this behavior is be defining a new class, which extends list but otherwise does nothing. Then I can assign the variable list to that new class, and each time I would like to instantiate a new list, I'd use the list constructor, like it would have been used to create the original list type.
class MyList(list):
def each(self, func):
for item in self:
func(item)
list = MyList
my_list = list((1,2,3,4))
my_list.each(lambda x: print(x))
Output:
1
2
3
4
The idea can be generalize of course by defining a method that gets a built it type and returns a class that extends that type. Further more, the original list variable can be saved in another variable to keep an access to it.
Only problem I'm facing right now, is that when you instantiate a list by its literal form, (i.e. [1,2,3,4]), it will still use the original list constructor (or does it?). Is there a way to override this behavior? If the answer is no, do you know of some other way of enabling the user to extend the built-ins types? (just like javascript allows extending built-ins prototypes).
I find this limitation of built-ins (being unable to add members to them) one of Python's drawbacks, making it inconsistent with other user-defined types... Overall I really love the language, and I really don't understand why this limitation is REALLY necessary.
This is a conscientious choice from Python.
Firstly, with regards to patching inbuilt type, this is primarily a design decision and only secondarily an optimization. I have learnt from much lurking on the Python Mailing List that monkey patching on builtin types, although enjoyable for small scripts, serves no good purpose in anything larger.
Libraries, for one, make certain assumptions about types. If it were encouraged to extend default types, many libraries would end up fighting each other. It would also discourage making new types – a deque is a deque, an ordered set is an ordered set, a dictionary is a dictionary and that should be that.
Literal syntax is a particularly important point. If you cannot guarantee that [1, 2, 3] is a list, what can you guarantee? If people could change those behaviours it would have such global impacts as to destroy the stability of a lot of code. There is a reason goto and global variables are discouraged.
There is one particular hack that I am fond of, though. When you see r"hello", this seems to be an extended literal form.
So why not r[1, 2, 3]?
class ListPrefixer:
def __init__(self, typ):
self.typ = typ
def __getitem__(self, args):
return self.typ(args)
class MyList(list):
def each(self, func):
return MyList(func(x) for x in self)
e = ListPrefixer(MyList)
e[1, 2, 3, 4].each(lambda x: x**2)
#>>> [1, 4, 9, 16]
Finally, if you really want to do deep AST hacks, check out MacroPy.
I'm designing a library with a fairly powerful class - we can think of it as a list.
I'm expecting users to want to create their own lists, prepopulated with certain content. For the sake of example, things like "add n identical items to the list", or "initialise with the numbers from a->b". Note that these things don't really change how the list works, they just set it up in some complex way.
The users will need to use these from different bits of code, be able to reuse them in the future.
I can see two possible ways for the users to do this:
Create a function which builds the object for them, e.g.:
def make_item_list(n,x):
ls = [x]
ls *= n
return ls
Or, alternatively, subclass the list to add an initialiser:
class RepeatList(List):
def __init__(self,n,x):
super().__init__()
self.listitems = [x]*n
Clearly this is a toy example but which of these is more sensible and 'pythonic'? If there are other patterns in Python to do this, what are they? If the choice of pattern depends on something else I haven't mentioned, what is it and how would it influence decision making? I'm happy to provide more info if you need it. Thank you!
Most style guides frown upon excessive subclassing. There is a reason why pylint's default configuration will complain about classes without at least a few methods. Python has enough more light-weight methods to achieve the same. As you've said it yourself, a factory function does the same job.
It's also somewhat awkward. Do you really want your users to create classes called something like ListFromFile and ListFromString instead of simple functions make_list_from_file and make_list_from_string?
Beyond simple style, there are functional reasons to consider:
A subclass would allow you to check the type information. Is this relevant? Probably not, but there are of course cases where you subclass for the simple purpose of creating a new type, e.g. exception classes
A subclass has higher runtime overhead. Especially method lookup will be slower.
Subclassing can lead to confusing behavior or may need additional care in programming. For example, let's say you want to implement __getitem__(self, slice) and like Python's list, return a copy of the given slice (like list[0:10]). Will the copy be an instance of your base class or your subclass? If it is an instance of the subclass, how do you make sure the init method accepts the correct parameters? If it will be an instance of the base class, any other overwritten behavior will be gone.
I suggest the following:
Create a simple and robust init method and tell people not to overwrite it, or only overwrite it with the same arguments
Use classmethods as additional constructors for certain types of arguments, e.g. from file, from python list, whatever
Advocate simple free-standing functions for simple cases
Suggest subclassing only when adding new behavior
Advocate adding new classmethods when adding new constructors for these complex cases
A last thought: Know your users. If yours are like mine, they are more familiar with structured programming using free functions and modules, not object oriented programming with inheritance. Of course, if your users are more familiar with OOP, maybe because you used to be a Java shop, the opposite may be true.
Addendum: As a case study where subclassing went wrong, look at numpy and its ndarray subclasses. Especially matrix and memmap. Try them sometime and see how far you come before something breaks because the semantics are not the same. If you pass a matrix to a function, you probably want a matrix as your return value. But if you pass a memmap, you cannot do this because you can't simply create a new memmap. Now you also have to think hard at every function whether you want to sanitize your input with np.asarray or np.asanyarray and know the different semantics that you may or may not invoke.
If even the numpy folks messed this up, how likely are you and your users getting it right?
I tend to favor on using the class as you can extend its functionality like adding methods that could be helpful to the users.
class List:
pass
class GenerateList(List):
def __init__(self, n, x, repeat=True):
super().__init__()
if repeat:
self.listitems = [x] * n
else:
self.listitems = list(range(n, x+1))
def get_even(self, flip=False):
ret = [v for v in self.listitems if v%2 == 0]
return ret if not flip else ret[::-1]
def get_odd(self, flip=False):
ret = [v for v in self.listitems if v%2 != 0]
return ret if not flip else ret[::-1]
# Request a repeated list.
m = GenerateList(3, 2, repeat=True)
repeatlist = m.listitems
# Request a list in a range.
a = GenerateList(1, 10, repeat=False)
mylist = a.listitems
mylistodd = a.get_odd()
mylisteven = a.get_even(flip=True)
print(repeatlist)
print(mylist)
print(mylistodd)
print(mylisteven)
Output:
[2, 2, 2]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 3, 5, 7, 9]
[10, 8, 6, 4, 2]
Couldn't find a way to phrase the title better, feel free to correct.
I'm pretty new to Python, currently experimenting with the language.. I've noticed that all built-ins types cannot be extended with other members.. I'd like for example to add an each method to the list type, but that would be impossible. I realize that it's designed that way for efficiency reasons, and that most of the built-ins types are implemented in C.
Well, one why I found to override this behavior is be defining a new class, which extends list but otherwise does nothing. Then I can assign the variable list to that new class, and each time I would like to instantiate a new list, I'd use the list constructor, like it would have been used to create the original list type.
class MyList(list):
def each(self, func):
for item in self:
func(item)
list = MyList
my_list = list((1,2,3,4))
my_list.each(lambda x: print(x))
Output:
1
2
3
4
The idea can be generalize of course by defining a method that gets a built it type and returns a class that extends that type. Further more, the original list variable can be saved in another variable to keep an access to it.
Only problem I'm facing right now, is that when you instantiate a list by its literal form, (i.e. [1,2,3,4]), it will still use the original list constructor (or does it?). Is there a way to override this behavior? If the answer is no, do you know of some other way of enabling the user to extend the built-ins types? (just like javascript allows extending built-ins prototypes).
I find this limitation of built-ins (being unable to add members to them) one of Python's drawbacks, making it inconsistent with other user-defined types... Overall I really love the language, and I really don't understand why this limitation is REALLY necessary.
This is a conscientious choice from Python.
Firstly, with regards to patching inbuilt type, this is primarily a design decision and only secondarily an optimization. I have learnt from much lurking on the Python Mailing List that monkey patching on builtin types, although enjoyable for small scripts, serves no good purpose in anything larger.
Libraries, for one, make certain assumptions about types. If it were encouraged to extend default types, many libraries would end up fighting each other. It would also discourage making new types – a deque is a deque, an ordered set is an ordered set, a dictionary is a dictionary and that should be that.
Literal syntax is a particularly important point. If you cannot guarantee that [1, 2, 3] is a list, what can you guarantee? If people could change those behaviours it would have such global impacts as to destroy the stability of a lot of code. There is a reason goto and global variables are discouraged.
There is one particular hack that I am fond of, though. When you see r"hello", this seems to be an extended literal form.
So why not r[1, 2, 3]?
class ListPrefixer:
def __init__(self, typ):
self.typ = typ
def __getitem__(self, args):
return self.typ(args)
class MyList(list):
def each(self, func):
return MyList(func(x) for x in self)
e = ListPrefixer(MyList)
e[1, 2, 3, 4].each(lambda x: x**2)
#>>> [1, 4, 9, 16]
Finally, if you really want to do deep AST hacks, check out MacroPy.
Couldn't find a way to phrase the title better, feel free to correct.
I'm pretty new to Python, currently experimenting with the language.. I've noticed that all built-ins types cannot be extended with other members.. I'd like for example to add an each method to the list type, but that would be impossible. I realize that it's designed that way for efficiency reasons, and that most of the built-ins types are implemented in C.
Well, one why I found to override this behavior is be defining a new class, which extends list but otherwise does nothing. Then I can assign the variable list to that new class, and each time I would like to instantiate a new list, I'd use the list constructor, like it would have been used to create the original list type.
class MyList(list):
def each(self, func):
for item in self:
func(item)
list = MyList
my_list = list((1,2,3,4))
my_list.each(lambda x: print(x))
Output:
1
2
3
4
The idea can be generalize of course by defining a method that gets a built it type and returns a class that extends that type. Further more, the original list variable can be saved in another variable to keep an access to it.
Only problem I'm facing right now, is that when you instantiate a list by its literal form, (i.e. [1,2,3,4]), it will still use the original list constructor (or does it?). Is there a way to override this behavior? If the answer is no, do you know of some other way of enabling the user to extend the built-ins types? (just like javascript allows extending built-ins prototypes).
I find this limitation of built-ins (being unable to add members to them) one of Python's drawbacks, making it inconsistent with other user-defined types... Overall I really love the language, and I really don't understand why this limitation is REALLY necessary.
This is a conscientious choice from Python.
Firstly, with regards to patching inbuilt type, this is primarily a design decision and only secondarily an optimization. I have learnt from much lurking on the Python Mailing List that monkey patching on builtin types, although enjoyable for small scripts, serves no good purpose in anything larger.
Libraries, for one, make certain assumptions about types. If it were encouraged to extend default types, many libraries would end up fighting each other. It would also discourage making new types – a deque is a deque, an ordered set is an ordered set, a dictionary is a dictionary and that should be that.
Literal syntax is a particularly important point. If you cannot guarantee that [1, 2, 3] is a list, what can you guarantee? If people could change those behaviours it would have such global impacts as to destroy the stability of a lot of code. There is a reason goto and global variables are discouraged.
There is one particular hack that I am fond of, though. When you see r"hello", this seems to be an extended literal form.
So why not r[1, 2, 3]?
class ListPrefixer:
def __init__(self, typ):
self.typ = typ
def __getitem__(self, args):
return self.typ(args)
class MyList(list):
def each(self, func):
return MyList(func(x) for x in self)
e = ListPrefixer(MyList)
e[1, 2, 3, 4].each(lambda x: x**2)
#>>> [1, 4, 9, 16]
Finally, if you really want to do deep AST hacks, check out MacroPy.
I need a container that can collect a number of objects and provides some reporting functionality on the container's elements. Essentially, I'd like to be able to do:
magiclistobject = MagicList()
magiclistobject.report() ### generates all my needed info about the list content
So I thought of subclassing the normal list and adding a report() method. That way, I get to use all the built-in list functionality.
class SubClassedList(list):
def __init__(self):
list.__init__(self)
def report(self): # forgive the silly example
if 999 in self:
print "999 Alert!"
Instead, I could also create my own class that has a magiclist attribute but I would then have to create new methods for appending, extending, etc., if I want to get to the list using:
magiclistobject.append() # instead of magiclistobject.list.append()
I would need something like this (which seems redundant):
class MagicList():
def __init__(self):
self.list = []
def append(self,element):
self.list.append(element)
def extend(self,element):
self.list.extend(element)
# more list functionality as needed...
def report(self):
if 999 in self.list:
print "999 Alert!"
I thought that subclassing the list would be a no-brainer. But this post here makes it sounds like a no-no. Why?
One reason why extending list might be bad is since it ties together your 'MagicReport' object too closely to the list. For example, a Python list supports the following methods:
append
count
extend
index
insert
pop
remove
reverse
sort
It also contains a whole host of other operations (adding, comparisons using < and >, slicing, etc).
Are all of those operations things that your 'MagicReport' object actually wants to support? For example, the following is legal Python:
b = [1, 2]
b *= 3
print b # [1, 2, 1, 2, 1, 2]
This is a pretty contrived example, but if you inherit from 'list', your 'MagicReport' object will do exactly the same thing if somebody inadvertently does something like this.
As another example, what if you try slicing your MagicReport object?
m = MagicReport()
# Add stuff to m
slice = m[2:3]
print type(slice)
You'd probably expect the slice to be another MagicReport object, but it's actually a list. You'd need to override __getslice__ in order to avoid surprising behavior, which is a bit of a pain.
It also makes it harder for you to change the implementation of your MagicReport object. If you end up needing to do more sophisticated analysis, it often helps to be able to change the underlying data structure into something more suited for the problem.
If you subclass list, you could get around this problem by just providing new append, extend, etc methods so that you don't change the interface, but you won't have any clear way of determining which of the list methods are actually being used unless you read through the entire codebase. However, if you use composition and just have a list as a field and create methods for the operations you support, you know exactly what needs to be changed.
I actually ran into a scenario very similar to your at work recently. I had an object which contained a collection of 'things' which I first internally represented as a list. As the requirements of the project changed, I ended up changing the object to internally use a dict, a custom collections object, then finally an OrderedDict in rapid succession. At least in my experience, composition makes it much easier to change how something is implemented as opposed to inheritance.
That being said, I think extending list might be ok in scenarios where your 'MagicReport' object is legitimately a list in all but name. If you do want to use MagicReport as a list in every single way, and don't plan on changing its implementation, then it just might be more convenient to subclass list and just be done with it.
Though in that case, it might be better to just use a list and write a 'report' function -- I can't imagine you needing to report the contents of the list more than once, and creating a custom object with a custom method just for that purpose might be overkill (though this obviously depends on what exactly you're trying to do)
As a general rule, whenever you ask yourself "should I inherit or have a member of that type", choose not to inherit. This rule of thumb is known as "favour composition over inheritance".
The reason why this is so is: composition is appropriate where you want to use features of another class; inheritance is appropriate if other code needs to use the features of the other class with the class you are creating.