Develop Reference

Equality Comparison with NumPy Instance Invokes `__bool__`

I have defined a class where its __ge__ method returns an instance of itself, and whose __bool__ method is not allowed to be invoked (similar to a Pandas Series).
Why is X.__bool__ invoked during np.int8(0) <= x, but not for any of the other examples? Who is invoking it? I have read the Data Model docs but I haven’t found my answer there.
import numpy as np
import pandas as pd
class X:
def __bool__(self):
print(f"{self}.__bool__")
assert False
def __ge__(self, other):
print(f"{self}.__ge__")
return X()
x = X()
np.int8(0) <= x
# Console output:
# <__main__.X object at 0x000001BAC70D5C70>.__ge__
# <__main__.X object at 0x000001BAC70D5D90>.__bool__
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# File "<stdin>", line 4, in __bool__
# AssertionError
0 <= x
# Console output:
# <__main__.X object at 0x000001BAC70D5C70>.__ge__
# <__main__.X object at 0x000001BAC70D5DF0>
x >= np.int8(0)
# Console output:
# <__main__.X object at 0x000001BAC70D5C70>.__ge__
# <__main__.X object at 0x000001BAC70D5D30>
pd_ge = pd.Series.__ge__
def ge_wrapper(self, other):
print("pd.Series.__ge__")
return pd_ge(self, other)
pd.Series.__ge__ = ge_wrapper
pd_bool = pd.Series.__bool__
def bool_wrapper(self):
print("pd.Series.__bool__")
return pd_bool(self)
pd.Series.__bool__ = bool_wrapper
np.int8(0) <= pd.Series([1,2,3])
# Console output:
# pd.Series.__ge__
# 0 True
# 1 True
# 2 True
# dtype: bool

I suspect that np.int8.__le__ is defined so that instead of returning NotImplemented and letting X.__ge__ take over, it instead tries to return something like not (np.int(8) > x), and then np.int8.__gt__ raises NotImplemented. Once X.__gt__(x, np.int8(0)) returns an instance of X rather than a Boolean value, then we need to call x.__bool__() in order to compute the value of not x.
(Still trying to track down where int8.__gt__ is defined to confirm.)
(Update: not quite. int8 uses a single generic rich comparison function that simply converts the value to a 0-dimensional array, then returns the result of PyObject_RichCompare on the array and x.)
I did find this function that appears to ultimately implement np.int8.__le__:
static NPY_INLINE int
rational_le(rational x, rational y) {
return !rational_lt(y,x);
}
It's not clear to me how we avoid getting to this function if one of the arguments (like X) would not be a NumPy type. I think I give up.

TL;DR
X.__array_priority__ = 1000
The biggest hint is that it works with a pd.Series.
First I tried having X inherit from pd.Series. This worked (i.e. __bool__ no longer called).
To determine whether NumPy is using an isinstance check or duck-typing approach, I removed the explicit inheritance and added (based on this answer):
#property
def __class__(self):
return pd.Series
The operation no longer worked (i.e. __bool__ was called).
So now I think we can conclude NumPy is using a duck-typing approach. So I checked to see what attributes are being accessed on X.
I added the following to X:
def __getattribute__(self, item):
print("getattr", item)
return object.__getattribute__(self, item)
Again instantiating X as x, and invoking np.int8(0) <= x, we get:
getattr __array_priority__
getattr __array_priority__
getattr __array_priority__
getattr __array_struct__
getattr __array_interface__
getattr __array__
getattr __array_prepare__
<__main__.X object at 0x000002022AB5DBE0>.__ge__
<__main__.X object at 0x000002021A73BE50>.__bool__
getattr __array_struct__
getattr __array_interface__
getattr __array__
Traceback (most recent call last):
File "<stdin>", line 32, in <module>
np.int8(0) <= x
File "<stdin>", line 21, in __bool__
assert False
AssertionError
Ah-ha! What is __array_priority__? Who cares, really. With a little digging, all we need to know is that NDFrame (from which pd.Series inherits) sets this value as 1000.
If we add X.__array_priority__ = 1000, it works! __bool__ is no longer called.
What made this so difficult (I believe) is that the NumPy code didn't show up in the call stack because it is written in C. I could investigate further if I tried out the suggestion here.

Comparing None to several variables in Python3

Is there anyone that can tell my why I get this type of return value?
>>> a = 7
>>> b = None
>>> bool(a and b)
False
>>> bool((a is None) and (b is None))
False
>>> bool((a and b) is None) # (a and b) eq False
True
>>> bool(False is None)
False
I just can't understand why this is happening.

To explain
>>> 7 and None
None
>>> bool(None)
False
So to answer:
a and b gives None and not False as you put in comment.
bool(a and b) gives False
So then when you replace a and b by its real value:
you get bool(None is None) which is True.
I believe you had bool(bool(a and b) is None) in mind, which would give False

Python's and actually returns the last value tested. For x and y, if x is false, it will return x; if x is true, it will return y. For example:
>>> 0 and ''
0
>>> 0 and 'x'
0
>>> 1 and ''
''
>>> 1 and 'x'
'x'
So when you do a and b, you get None.
Source: Documentation: Boolean Operations — and, or, not

All Python builtins and most instances have implicit truth values thanks to their __bool__ magic methods. In your case, you are working with the builtin types of int and None. In pure python, the __bool__ method for an int would like something like the following:
class int(object):
...
def __bool__(self):
return self != 0 # True as long as not equal to 0
...
And for a None it would be the following:
class NoneType(object):
...
def __bool__(self):
return False # Always False
...
Whenever an operation is done that requires the variable to be treated as a boolean (and, or, not, bool casting), these __bool__ magic methods are used to get their corresponding instance's truth value.
So with this in mind, let's look at your expressions one-by-one.
>>> bool(a and b)
False
a is the integer 7, so by its __bool__ magic method, it has a positive truth value (True). b is None, and it has a negative truth value (False). When you and variables in Python, if the first and operand has a positive truth value, and will always return the second operand (the opposite behavior can be seen with or). More information on this can be found here. So when you do a and b, None is returned because a has a positive truth value. The resulting None is then casted to a bool which as shown in None's __bool__ method above, will be False.
>>> bool((a is None) and (b is None))
False
This evaluates to False as a is not None. Here, since you are using is in the statements, you are not comparing truth value, but you are checking if they are the same object. Since 7 and None are not the same instance, it results to False, causing the rest of the statement to evaluate to False.
>>> bool((a and b) is None) # (a and b) eq False (should be None)
True
Following up from the first one, a and b will return None. since None is the same instance as None, None is None evaluates to True. The casting to bool is redundant.
>>> bool(False is None)
False
Lastly, here we are once again checking if False and None are the same instance. As they are not, this evaluates to False and the casting to bool is once again redundant.

Consider these two facts:
None is false
and returns true if both variables are true.
In every case that misleads you, you did not consider these facts.

How to check if value is logically integer?

I have some functions doing math stuff which needs to take integer agrmuents.
I know that I can force using int by using condition isinstance(x, int)
or more strict type(x) == int, but IMO it isn't pythonic.
I think my Python code shouldn't reject 2.0 just because it's float.
What's the best way to check if value is logically integer?
By logically integer I mean value of any type, which represents integer.
It should be able to be used in arithmetic operations like int, but I don't have to check it,
because I belive that in Python any set of conditions can get fooled.
For example,
True, -2, 2.0, Decimal(2), Fraction(4, 2) are logically integers,
when '2' and 2.5 are not.
At the moment I use int(x) == x, but I'm not sure if it's the best solution.
I know I can use float(x).is_integer().
I also saw x % 1 == 0.

Normally one would check against the Integral ABC (Abstract Base Class), but floating point values are not normally meant to be treated as integers no matter their value. If you want that, take note of their is_integer property:
(1324.34).is_integer()
#>>> False
(1324.00).is_integer()
#>>> True
and the code is then just:
from numbers import Integral
def is_sort_of_integer(value):
if isinstance(value, Integral):
return True
try:
return value.is_integer()
except AttributeError:
return False
If you also want to deal with Decimals, Fractions and so on, which don't have an is_integer method, the best option would probably be just:
from numbers import Number, Integral
def is_sort_of_integer(value):
if isinstance(value, Integral):
return True
if isinstance(value, Number):
try:
return not value % 1
except TypeError:
return False
return False
The Integral check shouldn't be needed in this case, but it's probably best to keep it.

One way to solve this is by using a metaclass to define your custom implementation of __instancecheck__, then define a concrete class having the metaclass, and use isinstance with your concrete class.
This has the downside of pulling in metaclass machinery, which is often extraneous.
But it has the upside of cleanly encapsulating whatever properties you desire to use for what you mean by "logically integer" for your application.
Here's some code to show this approach:
class Integral(type):
def __instancecheck__(self, other):
try:
cond1 = int(other) == other
cond2 = (other >= 1.0) or (other < 1.0)
# ... plus whatever other properties you want to check
return all([cond1, cond2,])
except:
return False
class IntLike:
__metaclass__ = Integral
print isinstance(-1, IntLike)
print isinstance('1', IntLike)
print isinstance(27.2, IntLike)
print isinstance(27.0, IntLike)
print isinstance(fractions.Decimal(2), IntLike)
print isinstance(fractions.Fraction(4, 2), IntLike)
It prints:
True
False
False
True
True
True
Note that it is important to get rid of the idea that the mathematical concept of being logically integer should apply to your program. Unless you bring in some proof-checking machinery, you won't get that. For example, you mention properties like certain functions being available, and specifically sqrt -- but this won't be available for negative integers unless you implement custom behavior to check for complex results.
It will be application-specific. For example, someone else might pick up this code and modify it so that '1' does register as IntLike, and perhaps for the sake of their application it will be correct.
This is why I like the metaclass approach here. It lets you explicitly denote each condition that you are imposing, and store them in one location. Then using the regular isinstance machinery makes it very clear to code readers what you are trying to do.
Lastly, note that no given conditions will always be perfect. For example, the class below could be used to 'fool' the int(x) == x trick:
class MyInt(object):
def __init__(self, value):
self.value = value
def __int__(self):
return int(self.value)
def __add__(self, other):
return self.value + other
#... define all needed int operators
def __eq__(self, other):
if isinstance(other, float):
raise TypeError('How dare you compare me to a float!')
return self.value == other
# ..etc
Then you get behavior like this:
In [90]: mi = MyInt(3)
In [91]: mi + 4
Out[91]: 7
In [92]: mi == 3
Out[92]: True
In [93]: int(mi) == mi
Out[93]: True
In [94]: mi == 3.0
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-93-827bea4a197f> in <module>()
----> 1 mi == 3.0
<ipython-input-89-66fec92fab7d> in __eq__(self, other)
13 def __eq__(self, other):
14 if isinstance(other, float):
---> 15 raise TypeError('How dare you compare me to a float!')
16 return self.value == other
17
TypeError: How dare you compare me to a float!
and whether or not isinstance(mi, IntLike) returns True will be totally dependent on how you have implemented the comparison operators for MyInt and also whatever extra checks you have made in Integral's __instancecheck__.

There are some cases, which none of int(x) == x, x.isinteger() & x % 1 == 0 can handle the way I would like to.
Example:
>>> big_float = 9999999999999999.1
The big_float is big enough to ignore substracting some small number from it (AFAIK, it's called underflow):
>>> big_float -1 == big_float
True
Then
>>> def fib(n):
... current, prev = 1, 0
... while n > 0:
... current, prev, n = current+prev, current, n-1
... return prev
...
>>> fib(big_float) #unwanted infinite loop
There are some cases, which none of int(x) == x, x.isinteger() & x % 1 == 0 can handle the way I would like to.
>>> int(big_float) == big_float
True
>>> big_float.is_integer()
True
>>> big_float % 1 == 0
True
Solution
We can check if big_float in the same way as int(big_float):
>>> int(big_float) -1 == big_float -1
False
Of course, this method works also for more trivial cases, like this:
>>> x = 2.1
>>> int(x) -1 == x -1
False
Of course, you don't have to substract 1, you can use whatever mathematical operation you need.
Note that this condition may throw exception:
>>> x = '2'
>>> int(x) -1 == x -1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for -: 'str' and 'int'

Simple, try to convert it to an integer. If int() works, it is "logically an integer", otherwise it is not.
try:
int(thing)
is_integer = True
except ValueError:
is_integer = False
But, typically, rather than do it like this you would just use int(thing) in the code you needed this for, and just catch the error if it ends up not being an integer and handle that case appropriately.

Disable silent conversions in numpy

Is there a way to disable silent conversions in numpy?
import numpy as np
a = np.empty(10, int)
a[2] = 4 # OK
a[3] = 4.9 # Will silently convert to 4, but I would prefer a TypeError
a[4] = 4j # TypeError: can't convert complex to long
Can numpy.ndarray objects be configured to return a TypeError when assigning any value which is not isinstance() of the ndarray type?
If not, would the best alternative be to subclass numpy.ndarray (and override __setattr__ or __setitem__)?

Unfortunately numpy doesn't offer this feature in array creation, you can set if casting is allowed only when you are converting an array (check the documentation for numpy.ndarray.astype).
You could use that feature, or subclass numpy.ndarray, but also consider using the array module offered by python itself to create a typed array:
from array import array
a = array('i', [0] * 10)
a[2] = 4 # OK
a[3] = 4.9 # TypeError: integer argument expected, got float

Just an idea.
#Python 2.7.3
>>> def test(value):
... if '.' in str(value):
... return str(value)
... else:
... return value
...
>>> a[3]=test(4.0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for long() with base 10: '4.0'

How to handle empty (none) tuple returned from python function

I have a function that either returns a tuple or None. How is the Caller supposed to handle that condition?
def nontest():
return None
x,y = nontest()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not iterable

EAFP:
try:
x,y = nontest()
except TypeError:
# do the None-thing here or pass
or without try-except:
res = nontest()
if res is None:
....
else:
x, y = res

How about:
x,y = nontest() or (None,None)
If nontest returns a two-item tuple like it should, then x and y are assigned to the items in the tuple. Otherwise, x and y are each assigned to none. Downside to this is that you can't run special code if nontest comes back empty (the above answers can help you if that is your goal). Upside is that it is clean and easy to read/maintain.

If you can change the function itself, it's probably a better idea to make it raise a relevant exception instead of returning None to signal an error condition. The caller should then just try/except that.
If the None isn't signalling an error condition, you'll want to rethink your semantics altogether.