can the primitive "and" be overridden?
for instance If I try to do something like this
class foo():
def __and__(self,other):
return "bar"
this gets outputted
>> foo() and 4
4
>> foo().__and__(4)
'bar'
my intuition is that the built in and cannot be overridden and shouldn't be overridden to not cause confusion.
I guess the and function shouldn't be changed because it doesn't need to be, since it behaves like
def and(self,other):
if( not bool(self) ):
return self
else:
return other
__and__ overrides the bitwise operator &, not the logic operator.
To give your class boolean logic handling, define a __nonzero__ method: http://docs.python.org/reference/datamodel.html#object.__nonzero__
The and operator cannot be overridden, since it doesn't always evaluate both of its operands. There is no way to provide a function that can do what it does, arguments to functions are always fully evaluated before the function is invoked. The same goes for or.
Boolean and and or can't be overridden in Python. It is true that they have special behavior (short-circuiting, such that both operands are not always evaluated) and this has something to do with why they are not overrideable. However, this is not (IMHO) the whole story.
In fact, Python could provide a facility that supports short-circuiting while still allowing the and and or operators to be overridden; it simply does not do so. This may be a deliberate choice to keep things simple, or it may be because there are other things the Python developers think would be more useful to developers and are working on them instead.
However, it is a choice on the part of Python's designers, not a law of nature.
Related
According to the object.__eq__() documentation, the default (that is, in the object class) implementation for == is as follows:
True if x is y else NotImplemented
Still following the documentation for NotImplemented, I inferred that NotImplemented implies that the Python runtime will try the comparison the other way around. That is try y.__eq__(x) if x.__eq__(y) returns NotImplemented (in the case of the == operator).
Now, the following code prints False and True in python 3.9:
class A:
pass
print(A() == A())
print(bool(NotImplemented))
So my question is the following: where does the documentation mention the special behavior of NotImplemented in the context of __eq__ ?
PS : I found an answer in CPython source code but I guess that this must/should be somewhere in the documentation.
According to the object.__eq__() documentation, the default (that is, in the object class) implementation for == is as follows
No; that is the default implementation of __eq__. ==, being an operator, cannot be implemented in classes.
Python's implementation of operators is cooperative. There is hard-coded logic that uses the dunder methods to figure out what should happen, and possibly falls back on a default. This logic is outside of any class.
You can see another example with the built-in len: a class can return whatever it likes from its __len__ method, and you can in principle call it directly and get a value of any type. However, this does not properly implement the protocol, and len will complain when it doesn't get a positive integer back. There is not any class which contains that type-checking and value-checking logic. It is external.
Still following the documentation for NotImplemented, I inferred that NotImplemented implies that the Python runtime will try the comparison the other way around. That is try y.__eq__(x) if x.__eq__(y) returns NotImplemented (in the case of the == operator).
NotImplemented is just an object. It is not syntax. It does not have any special behavior, and in Python, simply returning a value does not trigger special behavior besides that the value is returned.
The external code for binary operators will try to look for the matching __op__, and try to look for the matching __rop__ if __op__ didn't work. At this point, NotImplemented is not an acceptable answer (it is a sentinel that exists specifically for this purpose, because None is an acceptable answer). In general, if the answer so far is still NotImplemented, then the external code will raise NotImplementedError.
As a special case, objects that don't provide their own comparison (i.e., the default from object is used for __eq__ or __ne__) will compare as "not equal" unless they are identical. The C implementation repeats the identity check (in case a class explicitly defines __eq__ or __ne__ to return NotImplemented directly, I guess). This is because it is considered sensible to give this result, and obnoxious to make == fail all the time when there is a sensible default.
However, the two objects are still not orderable without explicit logic, since there isn't a reasonable default. (You could compare the pointer values, but they're arbitrary and don't have anything to do with the Python logic that got you to that point; so ordering things that way isn't realistically useful for writing Python code.) So, for example, x < y will raise a TypeError if the comparison logic isn't provided. (It does this even if x is y; you could reasonably say that <= and >= should be true in this case, and < and > should be false, but it makes things too complicated and is not very useful.)
[Observation: print(bool(NotImplemented)) prints True]
Well, yes; NotImplemented is an object, so it's truthy by default; and it doesn't represent a numeric value, and isn't a container, so there's no reason for it to be falsy.
However, that also doesn't tell us anything useful. We don't care about the truthiness of NotImplemented here, and it isn't used that way in the Python implementation. It is just a sentinel value.
where does the documentation mention the special behavior of NotImplemented in the context of __eq__ ?
Nowhere, because it isn't a behavior of NotImplemented, as explained above.
Okay, but that leaves underlying question: where does the documentation explain what the == operator does by default?
Answer: because we are talking about an operator, and not about a method, it's not in the section about dunder methods. It's in section 6, which talks about expressions. Specifically, 6.10.1. Value comparisons:
The default behavior for equality comparison (== and !=) is based on the identity of the objects. Hence, equality comparison of instances with the same identity results in equality, and equality comparison of instances with different identities results in inequality. A motivation for this default behavior is the desire that all objects should be reflexive (i.e. x is y implies x == y).
The docs.python.org page on the Python "Data Model" states that when both sides in a literal comparison operation implement magic methods for the operation, the method of the left operand gets used with the right operand as its argument:
x<y calls x.__lt__(y), x<=y calls x.__le__(y), x==y calls x.__eq__(y), x!=y calls x.__ne__(y), x>y calls x.__gt__(y), and x>=y calls x.__ge__(y).
The following class wraps the builtin tuple and implements a magic method for one of these comparison operators to demonstrate this:
class eqtest(tuple):
def __eq__(self, other):
print('Equivalence!')
When using instances of this class on the left side of a comparison operator, it behaves as expected:
>>> eqtest((1,2,3)) == (1,2,3)
Equivalence!
However, the comparison operator of the custom class seems to get called even when only using its instance on the right:
>>> (1,2,3) == eqtest((1,2,3))
Equivalence!
The result is also demonstrably different when the magic method of the left operand is explicitly called:
>>> (1,2,3).__eq__(eqtest2((1,2,3)))
True
It's easy to understand why this might be a deliberate design choice, especially with subclasses, in order to return the result most likely to be useful from the type that was defined later. However, since it deviates quite explicitly from the basic documented behaviour, it's quite hard to know how and why it works this way confidently enough to account for and use it in production.
In what cases do the Python language and the CPython reference implementation reverse the order of comparison operators even if both sides provide valid results, and where is this documented?
The rules on comparisons state that tuples don't know how to compare to other types. tuplerichcompare does Py_RETURN_NOTIMPLEMENTED. However, the PyObject richcompare checks for subtypes, such as your inherited class, and swaps the comparison order (applying the symmetry rule).
This is documented in the page you linked as well:
If the operands are of different types, and right operand’s type is a direct or indirect subclass of the left operand’s type, the reflected method of the right operand has priority, otherwise the left operand’s method has priority. Virtual subclassing is not considered.
This enables subclasses to implement more specific behaviours that work with the comparison written either way.
Everything in Python is an object
We all know this sentence and all Pythonistas (including me) loving it. In that regard, it is interesting to look at operators. They seem to be no objects, e.g.
>>> type(*) # or /, +, -, < ...
returns SyntaxError: invalid syntax.
However, there might be situations where it would be useful to have them treated as objects. Consider for example a function like
def operation(operand1, operand2, operator):
"""
This function returns the operation of two operands defined by the operator as parameter
"""
# The following line is invalid python code and should only describe the function
return operand1 <operator> operand2
So operation(1, 2, +) would return 3, operation(1, 2, *) would return 2, operation(1, 2, <) would return True, etc...
Why is this not implemented in python? Or is it and if, how?
Remark: I do know the operator module, which also wouldn't be applicable in the example function above. Also I am aware that one could workaround it in a way like e.g. operations(operand1, operand2, '>') and find the desired operation via the string representation of the corresponding operator. However I am asking for the reason of the non-existance of operator-objects being able to be passed as parameters in functions e.g. like every other python object.
Every value is an object. Operators are not values; they are syntax. However, they are implemented by functions, which are values. The operator module provides access to those functions.
Not at all applicable to Python, though suggestive, is that a language could provide additional syntax to convert an operator into a "name". For example, in Haskell, you can use an infix operator like + as if it were a name using parentheses. Where you wanted to write operation(3, 5, +) in Python, Haskell allows operation 3 5 (+).
There's no technical reason why something similar couldn't be added to Python, but there's also no compelling design reason to add it. The operator module is sufficient and "fits" better with the language design as a whole.
Operators tell the interpreter what underlying method to operate on the objects provided, so they are more like functions, which are still object in a sense, you just need the appropriate reference to call the type on. For instance, say you have Foo.some_method and you want to look up its type. You need the proper reference: type(Foo.some_method) instead of just type(some_method), the first of which returns <class 'function'>, the latter a NameError.
That said, you can certainly implement something like this without the operator module:
def operation(operand1, operand2, operator):
return getattr(operand1, operator)(operand2)
operation(1, 2, '__add__')
# 3
That said, the easiest way to understand your issue is that operators are part of the syntax for python to interpret your code, not an actual object. So when the interpreter sees *, +, ~ etc... it expects two operands to fetch the aliased method and execute. The method itself is an object. The syntax, not so much.
Here is an example of an excellent question.
Technically speaking to answer properly to this question one would could use the Lexical Analyser (tokenizer) approach, in terms of token categories, besides encoding declarations, comments and blank lines.
Besides Operator, also: i) NEWLINE, INDENT and DEDENT, ii) Keywords, and iii) Delimiters are not objects. Everything else is an object in Python.
So next time they tell you "Everything in Python is an object" you should answer:
"Everything in a logical line that is not NEWLINE, INDENT, DEDENT, Space bar Character, Operator, Keyword or Delimiter is an object in Python."
The rationale attempt in here: When Python designers devised the language they figured that these token categories where pretty much useless to be treated as objects, in any situation, that could not be otherwise solved by other means.
Cheers.
You can think operator as a kind of syntax sugar. For example, 3+4 is just a syntax sugar of int.__add__(3,4). And type(int.__add__) is not None but type(+) will raise error.
You say:
# The following line is invalid python code and should only describe the function
return operand1 <operator> operand2
That's not true, that code isn't invalid. And with a fitting operator, it does work as intended:
def operation(operand1, operand2, operator):
"""
This function returns the operation of two operands defined by the operator as parameter
"""
return operand1 <operator> operand2
class Subtraction:
def __gt__(self, operand):
try:
return self.operand - operand
except:
self.operand = operand
return self
print(operation(13, 8, Subtraction()))
Prints 5 as expected.
Is there any substantial difference between operators and methods?
The only difference I see is the way the are called, do they have other differences?
For example in Python concatenation, slicing, indexing are defined as operators, while (referring to strings) upper(), replace(), strip() and so on are methods.
If I understand question currectly...
In nutshell, everything is a method of object. You can find "expression operators" methods in python magic class methods, in the operators.
So, why python has "sexy" things like [x:y], [x], +, -? Because it is common things to most developers, even to unfamiliar with development people, so math functions like +, - will catch human eye and he will know what happens. Similar with indexing - it is common syntax in many languages.
But there is no special ways to express upper, replace, strip methods, so there is no "expression operators" for it.
So, what is different between "expression operators" and methods, I'd say just the way it looks.
Your question is rather broad. For your examples, concatenation, slicing, and indexing are defined on strings and lists using special syntax (e.g., []). But other types may do things differently.
In fact, the behavior of most (I think all) of the operators is constrolled by magic methods, so really when you write something like x + y a method is called under the hood.
From a practical perspective, one of the main differences is that the set of available syntactic operators is fixed and new ones cannot be added by your Python code. You can't write your own code to define a new operator called $ and then have x $ y work. On the other hand, you can define as many methods as you want. This means that you should choose carefully what behavior (if any) you assign to operators; since there are only a limited number of operators, you want to be sure that you don't "waste" them on uncommon operations.
Is there any substantial difference between operators and
methods?
Practically speaking, there is no difference because each operator is mapped to a specific Python special method. Moreover, whenever Python encounters the use of an operator, it calls its associated special method implicitly. For example:
1 + 2
implicitly calls int.__add__, which makes the above expression equivalent1 to:
(1).__add__(2)
Below is a demonstration:
>>> class Foo:
... def __add__(self, other):
... print("Foo.__add__ was called")
... return other + 10
...
>>> f = Foo()
>>> f + 1
Foo.__add__ was called
11
>>> f.__add__(1)
Foo.__add__ was called
11
>>>
Of course, actually using (1).__add__(2) in place of 1 + 2 would be inefficient (and ugly!) because it involves an unnecessary name lookup with the . operator.
That said, I do not see a problem with generally regarding the operator symbols (+, -, *, etc.) as simply shorthands for their associated method names (__add__, __sub__, __mul__, etc.). After all, they each end up doing the same thing by calling the same method.
1Well, roughly equivalent. As documented here, there is a set of special methods prefixed with the letter r that handle reflected operands. For example, the following expression:
A + B
may actually be equivalent to:
B.__radd__(A)
if A does not implement __add__ but B implements __radd__.
In python, We say everything is an object,
for instance: expression x<y internally calls x.__lt__(y)
where __lt__ is the method in object of class('int' say if values are 2 & 3) and x and y are reference variables to objects
and
User defined Function square(3) internally calls square.__call__(3)
where 'square' is the name of an object of class 'function' and reference variable 'square' points to that object name 'square'.
So,
How are if-elif-else, for, break, continue, pass, lambda statements interpreted internally? as objects?
If yes, Can you give some examples to visualise in similar above manner?
Statements are not represented as objects in Python (including the examples you give). Rather, the way things work is that language syntax often maps to "hooks" that you can define to influence what the syntax does in a particular situation. For instance, the < syntax maps to the __lt__ hook. It doesn't mean that the < symbol "is an object"; it just means that you can define methods on your objects to customize how they work with that syntax.
Some of the syntax you ask about can be influenced in a similar way via magic methods.
if checks whether its condition is boolean true. That is, if x: internally calls bool(x) to determine whether x "counts" as True or False. You can influence this decision by defining a __nonzero__ method (or __bool__ in Python 3). elif works the same way. (You cannot change what else does.) Also, of course, a lot of if conditions involve comparisons, and you can customize those comparisons in the way you already mentioned. For instance, if you want to customize how if myObject < 2 will work, you can write a __lt__ magic method.
for engages the iterator protocol. You can make your own objects iterable by defining the __iter__ and/or next methods. You can't influence break and continue exactly, but you can write iterators that "fake" continues by skipping elements, or "fake" breaks by raising StopIteration.
lambda is just another syntax for defining a function. You can't change what lambda does, and it's not clear what the point would be anyway.
pass is a do-nothing statement. You can't change what it does, and there really would be no point: it exists specifically to do nothing.