operator module makes it easy to avoid unnecessary functions and lambdas
in situations like this:
import operator
def mytest(op, list1, list2):
ok = [op(i1, i2) for i1, i2 in zip(list1, list2)]
return all(ok)
mytest(operator.eq, [1, 2, 3], [1, 2, 3]) # True
mytest(operator.add, [-1, 2, -3], [1, -2, 33]) # False
Well, now I need to do i1 and i2, but to my surprise, I can't find and in the operator module! And the same applies to or! I know, and is not exactly operator, it's a keyword, but not, along with is and even del, are all keywords and all are included.
So what's the story? Why are they missing?
Because you cannot convert boolean operators into python functions. Functions always evaluate their arguments, and boolean operators do not. Adding and and or to the operators module would also require adding a special kind of functions (like lisp "macros") that evaluate their arguments on demand. Obviously, this is not something python designers ever wanted. Consider:
if obj is not None and obj.is_valid():
....
you cannot write this in a functional form. An attempt like
if operator.xyz(obj is not None, obj.is_valid())
will fail if obj is actually None.
You can write these yourself, but you'll need to pass a function (e.g. lambda) for the second argument to prevent it from being evaluated at call time, assuming that the usual short-circuiting behavior is important to you.
def func_or(val1, fval2):
return val1 or fval2()
def func_and(val1, fval2):
return val1 and fval2()
Usage:
func_or(False, lambda: True)
func_and(True, lambda: False)
The reason there's no operator.and is that and is a keyword, so that would be a SyntaxError.
As tgh435 explained, the reason there's no renamed and function in operator is that it would be misleading: a function call always evaluates its operands, but the and operator doesn't. (It would also be an exception to an otherwise consistent and simple rule.)
In your case, it looks like you don't actually care about short-circuiting at all, so can build your own version trivially:
def and_(a, b):
return a and b
Or, if you're just using it once, even inline:
mytest(lambda a, b: a and b, [-1, 2, -3], [1, -2, 33])
In some cases, it's worth looking at all (and, for or, any). It is effectively short-circuited and expanded to arbitrary operands. Of course it has a different API than the operator functions, taking a single iterable of operands instead of two separate operands. And the way it short-circuits is different; it just stops iterating the iterable, which only helps if you've set things up so the iterable is only evaluating things as needed. So, it's usually not usable as a drop-in replacement—but it's sometimes usable if you refactor your code a bit.
Python's and and or syntaxes cannot directly be mapped to functions. These syntaxes are lazy evaluated: If the result of the left part of the expression allows to know the value of the whole expression, the right part is skipped. Since they introduce flow control, their behavior cannot be reproduced using an operator.
To reduce confusion, python have chosen to simply not provide these methods.
georg gives a good example of a situation where and laziness matters:
if obj is not None and obj.is_valid():
...
Now, if you don't need lazy evaluation, you can use abarnert's answer implementation:
def and_(a, b):
return a and b
def or_(a, b):
return a or b
Usage:
>>> or_(False, True)
>>> and_(True, False)
If you need lazy evaluation, you can use kindall's answer implementation:
def func_or(val1, fval2):
return val1 or fval2()
def func_and(val1, fval2):
return val1 and fval2()
Usage:
>>> func_or(False, lambda: True)
>>> func_and(True, lambda: False)
Note:
As mentioned in the comments, the functions operator.and_ and operator.or_ correspond to the bitwise operators & and |. See: https://docs.python.org/3/library/operator.html#mapping-operators-to-functions
Note that the names operators.and and operators.or aren't used: and and or are Python keywords so it would be a syntax error.
Related
We know that,
a = 1
b = 2
print(not a > b)
is the correct way of using the "not" keyword and the below throws an error
a = 1
b = 2
print(a not > b)
since "not" inverts the output Boolean.
Thus, by this logic the correct way for checking the presence of a member in a list should be
a = 1
b = [2,3,4,5]
print(not a in b)
But I find the most common way is
a = 1
b = [2,3,4,5]
print(a not in b)
which from the logic given in previous example should throw an error.
So what is the correct way of using the "not in" operator in Python3.x?
not in is a special case that simplifies to exactly what you tried first. Namely,
a not in b
literally simplifies to not (a in b). It also works (slightly differently, but same idea) for is.
a is not b`
is equivalent to not (a is b). Python added these because they flow naturally like English prose. On the other hand, a not < b doesn't look or feel natural, so it's not allowed. The not in and is not are special cases in the grammar, not small parts of a general rule about where not can go. The only general rule in play is that not can always be used as a prefix operator (like in not (a < b))
what is the correct way of using the "not in" operator
There is only one way to use the not in operator. Your not a in b instead uses the not operator and the in operator.
PEP 8 doesn't seem to have an opinion about which to use, but about the similar is not operator (thanks Silvio) it says:
Use is not operator rather than not ... is. While both expressions are functionally identical, the former is more readable and preferred:
# Correct:
if foo is not None:
# Wrong:
if not foo is None:
So I'd say not in should also be preferred, for the same reason.
not, not in and in are all valid operators. Transitively, not (in_expression) is also valid
Correct way? Refer Zen of Python.
First of all not in, is not a two separate operator, is constituently a single operator ,and also known as membership operator. There is another membership operator that is in. Membership operator has high precedence than logical NOT, AND and OR.
print(not a in b) -> This is actually first evaluating a in b then result is inverted by the logical 'not' and then result is printed.
So as per your example it should print True as a in b gives False then it is inverted to True via logical NOT operator.
print(a not in b) -> Here python checks if a is not a part of the b, if it is return 'False' else 'True` .
So as per your example it should return True as a is not a part of b.
I think a not in b is more clear than not a in b.I would suggest to use membership operator for testing the membership.
However the result will remain same for both kind of expression but the process of evaluating is completely different.
Adding or multiplying a large list of numbers in Python can elegantly be done by folding the list with the addition or multiplication operator:
import functools, operator
lst = range(1,100)
sum = functools.reduce(operator.add, lst)
prod = functools.reduce(operator.mul, lst)
This needs the function equivalents of the operators + and * which
are provided by the operator module as operator.add and
operator.mul, respectively.
If I want to use the same idiom with the operator or:
ingredients = ['onion', 'celery', 'cyanide', 'chicken stock']
soup_is_poisonous = functools.reduce(operator.or, map(is_poisonous, ingredients))
... then I discover that operator doesn't have a function equivalent of the logical and and or operators (though it has one for logical not)
Of course, I can trivially write one that works:
def operator_or(x,y):
return x or y
But I wonder: why are there no operator.or and operator.and in operator? Bitwise and and or are there, but not the logical ones.
Of course this is just a minor annoyance, and the answer may well be
the same as with the missing identity function:
that it is easy to write one. But this holds for * and + as well, so why the difference?
To wrap up all your helpful answers
and comments, in order of somewhat decreasing (to me) convincingness:
the addition of operator.or would break an important promise made by the module
For all operators <op> that have function equivalents
operator.op in the operator module, it is the case that a <op> b is equivalent to (i.e. can always, without changing program
behaviour, replace or be replaced by) operator.op(a, b). This
equivalence is actually mentioned in the module docstring. This is
impossible to do for the operators and and or as their
evaluation is short-circuiting while Python function calls are always evaluated after all of their arguments are.
On the values True and False, | and &, hence also the existing (bitwise) operator.and_ and operator.or_
already return the same results (if they return at all, that is) as or and and.
If is_poisonous() returns either True of False (not an unreasonable requirement), I could use
soup_is_poisonous = reduce(operator.or_, map(is_poisonous, ingredients), False)
in the example from the original question. However, many Python
programs conveniently use any "truthy" value as True in idioms like
your_model_T_color = "black" or any_color_you_like
using | or operator.or_ instead of or here will result in a
TypeError or, even worse, some unexpected value (if the operands
are ints)
The functions any and all can be used instead of
functools.reduce(operator.or, ....)
I'm not convinced by this
argument: operator functions are used in many more contexts than
as a first argument to reduce. Moreover, any always returns
either True or False, not the first truthy value:
any([0,0,0,5,6,7]) # returns True
reduce(lambda x, y: x or y, [0,0,0,5,6,7]) # returns 5
so any and reduce(operator.or would not really be equivalent
any([x,y]) does the same (and more, as it accepts iterables) as operator.or(x,y) would.
That is not quite true (see above), any([0,5]) returns True while operator.or(0,5) would return 5. Moreover, the number of arguments matters greatly if we use a function as an argument to another function like reduce()
all is short-circuiting logical-and.
any is short-circuiting logical-or.
No need to put versions that take exactly two arguments (instead of an iterable) into the operator module, I guess.
Python has a great syntax for null coalescing:
c = a or b
This sets c to a if a is not False, None, empty, or 0, otherwise c is set to b.
(Yes, technically this is not null coalescing, it's more like bool coalescing, but it's close enough for the purpose of this question.)
There is not an obvious way to do this for a collection of objects, so I wrote a function to do this:
from functools import reduce
def or_func(x, y):
return x or y
def null_coalesce(*a):
return reduce(or_func, a)
This works, but writing my own or_func seems suboptimal - surely there is a built-in like __or__? I've attempted to use object.__or__ and operator.__or__, but the first gives an AttributeError and the second refers to the bitwise | (or) operator.
As a result I have two questions:
Is there a built-in function which acts like a or b?
Is there a built-in implementation of such a null coalesce function?
The answer to both seems to be no, but that would be somewhat surprising to me.
It's not exactly a single built-in, but what you want to achieve can be easily done with:
def null_coalesce(*a):
return next(x for x in a if x)
It's lazy, so it does short-circuit like a or b or c, but unlike reduce.
You can also make it null-specific with:
def null_coalesce(*a):
return next(x for x in a if x is not None)
Is there a built-in function which I can use which acts like a or b?
No. Quoting from this answer on why:
The or and and operators can't be expressed as functions because of their short-circuiting behavior:
False and some_function()
True or some_function()
in these cases, some_function() is never called.
A hypothetical or_(True, some_function()), on the other hand, would have to call some_function(), because function arguments are always evaluated before the function is called.
Is there a built-in implementation of such a null coalesce function?
No, there isn't. However, the Python documentation page for itertools suggests the following:
def first_true(iterable, default=False, pred=None):
"""Returns the first true value in the iterable.
If no true value is found, returns *default*
If *pred* is not None, returns the first item
for which pred(item) is true.
"""
# first_true([a,b,c], x) --> a or b or c or x
# first_true([a,b], x, f) --> a if f(a) else b if f(b) else x
return next(filter(pred, iterable), default)
Marco has it right, there's no built-in, and itertools has a recipe. You can also pip install boltons to use the boltons.iterutils.first() utility, which is perfect if you want short-circuiting.
from boltons.iterutils import first
c = first([a, b])
There are a few other related and handy reduction tools in iterutils, too, like one().
I've done enough of the above that I actually ended up wanting a higher-level tool that could capture the entire interaction (including the a and b references) in a Python data structure, yielding glom and its Coalesce functionality.
from glom import glom, Coalesce
target = {'b': 1}
spec = Coalesce('a', 'b')
c = glom(target, spec)
# c = 1
(Full disclosure, as hinted above, I maintain glom and boltons, which is good news, because you can bug me if you find bugs.)
I have a string list
[str1, str2, str3.....]
and I also have a def to check the format of the strings, something like:
def CheckIP(strN):
if(formatCorrect(strN)):
return True
return False
Now I want to check every string in list, and of course I can use for to check one by one. But could I use map() to make code more readable...?
You can map your list to your function and then use all to check if it returns True for every item:
if all(map(CheckIP, list_of_strings)):
# All strings are good
Actually, it would be cleaner to just get rid of the CheckIP function and use formatCorrect directly:
if all(map(formatCorrect, list_of_strings)):
# All strings are good
Also, as an added bonus, all uses lazy-evaluation. Meaning, it only checks as many items as are necessary before returning a result.
Note however that a more common approach would be to use a generator expression instead of map:
if all(formatCorrect(x) for x in list_of_strings):
In my opinion, generator expressions are always better than map because:
They are slightly more readable.
They are just as fast if not faster than using map. Also, in Python 2.x, map creates a list object that is often unnecessary (wastes memory). Only in Python 3.x does map use lazy-computation like a generator expression.
They are more powerful. In addition to just mapping items to a function, generator expressions allow you to perform operations on each item as they are produced. For example:
sum(x * 2 for x in (1, 2, 3))
They are preferred by most Python programmers. Keeping with convention is important when programming because it eases maintenance and makes your code more understandable.
There is talk of removing functions like map, filter, etc. from a future version of the language. Though this is not set in stone, it has come up many times in the Python community.
Of course, if you are a fan of functional programming, there isn't much chance you'll agree to points one and four. :)
An example, how you could do:
in_str = ['str1', 'str2', 'str3', 'not']
in_str2 = ['str1', 'str2', 'str3']
def CheckIP(strN):
# different than yours, just to show example.
if 'str' in strN:
return True
else:
return False
print(all(map(CheckIP, in_str))) # gives false
print(all(map(CheckIP, in_str2))) # gives true
L = [str1, str2, str3.....]
answer = list(map(CheckIP, L))
answer is a list of booleans such that answer[i] is CheckIP(L[i]). If you want to further check if all of those values are True, you could use all:
all(answer)
This returns True if and only if all the values in answer are True. However, you may do this without listifying:
all(map(CheckIP, L)), as, in python3, `map` returns an iterator, not a list. This way, you don't waste space turning everything into a list. You also save on time, as the first `False` value makes `all` return `False`, stopping `map` from computing any remaining values
Is there a good, succinct/built-in way to see if all the values in an iterable are zeros? Right now I am using all() with a little list comprehension, but (to me) it seems like there should be a more expressive method. I'd view this as somewhat equivalent to a memcmp() in C.
values = (0, 0, 0, 0, 0)
# Test if all items in values tuple are zero
if all([ v == 0 for v in values ]) :
print 'indeed they are'
I would expect a built-in function that does something like:
def allcmp(iter, value) :
for item in iter :
if item != value :
return False
return True
Does that function exist in python and I'm just blind, or should I just stick with my original version?
Update
I'm not suggesting that allcmp() is the solution. It is an example of what I think might be more meaningful. This isn't the place where I would suggest new built-ins for Python.
In my opinion, all() isn't that meaningful. It doesn't express what "all" is checking for. You could assume that all() takes an iterable, but it doesn't express what the function is looking for (an iterable of bools that tests all of them for True). What I'm asking for is some function like my allcmp() that takes two parameters: an iterable and a comparison value. I'm asking if there is a built-in function that does something similar to my made up allcmp().
I called mine allcmp() because of my C background and memcmp(), the name of my made up function is irrelevant here.
Use generators rather than lists in cases like that:
all(v == 0 for v in values)
Edit:
all is standard Python built-in. If you want to be efficient Python programmer you should know probably more than half of them (http://docs.python.org/library/functions.html). Arguing that alltrue is better name than all is like arguing that C while should be call whiletrue. Is subjective, but i think that most of the people prefer shorter names for built-ins. This is because you should know what they do anyway, and you have to type them a lot.
Using generators is better than using numpy because generators have more elegant syntax. numpy may be faster, but you will benefit only in rare cases (generators like showed are fast, you will benefit only if this code is bottleneck in your program).
You probably can't expect nothing more descriptive from Python.
PS. Here is code if you do this in memcpm style (I like all version more, but maybe you will like this one):
list(l) == [0] * len(l)
If you know that the iterable will contain only integers then you can just do this:
if not any(values):
# etc...
If values is a numpy array you can write
import numpy as np
values = np.array((0, 0, 0, 0, 0))
all(values == 0)
The any() function may be the most simple and easy way to achieve just that. If the iterable is empty,e.g. all elements are zero, it will return False.
values = (0, 0, 0, 0, 0)
print (any(values)) # return False
The built-in set is given an iterable and returns a collection (set) of unique values.
So it can be used here as:
set(it) == {0}
assuming it is the iterable
{0} is a set containing only zero
More info on python set-types-set-frozenset here in docs.
I prefer using negation:
all(not v for v in values)