I am writing a script in which i have to test numbers against a number of conditions. If any of the conditions are met i want to return True and i want to do that the fastest way possible.
My first idea was to use any() instead of nested if statements or multiple or linking my conditions. Since i would be satisfied if any of the conditions were True i could really benefit from any() being lazy and returning True as soon as it could.
Based on the fact that the following print happens instantly and not after 10 (= 0 + 1 + 2 + 3 + 4) seconds i assume it is. Is that the case or am i somehow mistaken?
import time
def some(sec):
time.sleep(sec)
return True
print(any(some(x) for x in range(5)))
Yes, any() and all() short-circuit, aborting as soon as the outcome is clear: See the docs:
all(iterable)
Return True if all elements of the iterable are true (or if the
iterable is empty). Equivalent to:
def all(iterable):
for element in iterable:
if not element:
return False
return True
any(iterable)
Return True if any element of the iterable is true. If the iterable is
empty, return False. Equivalent to:
def any(iterable):
for element in iterable:
if element:
return True
return False
While the all() and any() functions short-circuit on the first "true" element of an iterable, the iterable itself may be constructed in a non-lazy way. Consider this example:
>> any(x == 100 for x in range(10**8))
True
This will take several seconds to execute in Python 2 as range(10**8) constructs a list of 10**8 elements. The same expression runs instantly in Python 3, where range() is lazy.
As Tim correctly mentioned, any and all do short-circuit, but in your code, what makes it lazy is the use of generators. For example, the following code would not be lazy:
print(any([slow_operation(x) for x in big_list]))
The list would be fully constructed and calculated, and only then passed as an argument to any.
Generators, on the other hand, are iterables that calculate each item on demand. They can be expressions, functions, or sometimes manually implemented as lazy iterators.
Yes, it's lazy as demonstrated by the following:
def some(x, result=True):
print(x)
return result
>>> print(any(some(x) for x in range(5)))
0
True
>>> print(any(some(x, False) for x in range(5)))
0
1
2
3
4
False
In the first run any() halted after testing the first item, i.e. it short circuited the evaluation.
In the second run any() continued testing until the sequence was exhausted.
Yes, and here is an experiment that shows it even more definitively than your timing experiment:
import random
def some(x):
print(x, end = ', ')
return random.random() < 0.25
for i in range(5):
print(any(some(x) for x in range(10)))
typical run:
0, 1, 2, True
0, 1, True
0, True
0, 1, 2, 3, True
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, False
No. All and Any support shortcircuiting but they don't make the conditionals interpretation lazy.
If you want an All or Any using lazy evaluation you need to pass them a generator. Or else the values get evaluated in the moment the list/set/iterator/whatever is constructed
JoshiRaez is the actual correct answer.
Here is an example
a = []
any([True, a[0]])
will fail
On the other side, using OR (or AND) will not fail since its not a function:
a = []
True or a[0]
Related
I have a script that checks if there are one or more of the same items in a list. Here's the code:
items = ["Blue", "Black", "Red"]
def isUnique(item):
seen = list()
return not any(i in seen or seen.append(i) for i in item)
print(isUnique(items))
It prints "True" if all the items in the given list are unique and "False" if one or more items in the list are unique. Can someone please explain the any() part of the script for me as I don't fully understand how it works?
This code is kind of a hack, since it uses a generator expression with side-effects and exploits the fact that append returns None, which is falsy.
The equivalent code written in the imperative style is like so:
def isUnique(items):
seen = list()
for i in items:
if i in seen or seen.append(i):
return False
return True
The or is still a bit strange there - it is being used for its short-circuiting behaviour, so that append is only called when i in seen is false - so we could rewrite it like this:
def isUnique(items):
seen = list()
for i in items:
if i in seen:
return False
else:
seen.append(i)
return True
This is equivalent because append is only called when i in seen is false, and the call to append returns None which means the return False line shouldn't execute in that case.
Here you need to understand first how or operator works.
or is like exp1 or exp2
it just evaluates the expression which gives True first or give true at last
eg
>>> 2 or 3
2
>>> 5 or 0.0
5
>>> [] or 3
3
>>> 0 or {}
{}
now for your list comprehension, [i in seen or seen.append(i) for i in items] i in seen evaluate false and seen.append(i) True and which return None ie list.append return None so , comprehension contain all None
>>> seen = []
>>> items = ["Blue", "Black", "Red"]
>>> res = [i in seen or seen.append(i) for i in items]
>>> res
[None, None, None]
>>> any(res)
False
as per any documentation, it is returning false beacuse as it is not getting iterable or bool.
>>> help(any)
Help on built-in function any in module builtins:
any(iterable, /)
Return True if bool(x) is True for any x in the iterable.
If the iterable is empty, return False.
the any function in python takes a list of booleans and returns the OR of all of them.
the i in seen or seen.append(i) for i in item appends i to seen if it's not in seen already. but if it is already in seen then the append() does not run since the first part is already True, and python doesn't need to know if the second part is true since True OR'd with anything is True. so it doesn't execute it. so the seen array ends up being a unique list of colours it has seen.
i in seen or seen.append(i) for i in item is also a generator expression,
which generates booleans, and any checks the booleans it generates, if even one of them evaluates to True, the whole any will return True.
so the first time an item that is already in the seen array is found, any will stop the generator and return True itself.
so if a duplicate element happens to be in the array no more conditions are evaluated and no more elements are appended to seen array
so if the array had duplicate elements, like,
items = ["Blue", "Blue", "Black", "Red"]
def isUnique(item):
seen = list()
unique = not any(i in seen or seen.append(i) for i in item)
print(seen)
return unique
isUnique(items)
would result in the output, just
['Blue']
EDIT: there are great answers. Adding some simpler ways to achieve the wanted result:
Method 1:
items = ["Blue", "Black", "Red"]
items_set = set(items)
if len(items_set) != len(items):
# there are duplications
This works because a set object ‘removes’ duplications.
Method 2:
contains_duplicates = any(items.count(element) > 1 for element in items) # true if contains duplications and false otherwise.
See https://www.kite.com/python/answers/how-to-check-for-duplicates-in-a-list-in-python
———————————————
any is a great function
Return True if any element of the iterable is true. If the iterable is empty, return False
Your function isUnique, however, does a bit more logic. Let's break it down:
First you create an empty list object and store it in 'seen' variable.
for i in item - iterates the list of items.
i in seen - This statement returns True if 'i' is a member of 'seen', and false otherwise.
seen.append(i) - add i to seen. This statement returns None if 'i' is appeneded to seen successfully.
Notice the or statement between i in seen or seen.append(i). That means, if one of the statements here is True, the or statement returns True.
At this point, I'd run [i in seen or seen.append(i) for i in item], see the result and experiment with it. The result for your example is [None, None, None].
Basically, for each item, you both add it to the list and check if it is already in the list.
Finally, you use the any() function - which returns True if the iterable has a True value. This will happen only if i in seen will return True.
Notice you are using not any(...), which returns False in case there are no repititions.
There are simpler and clearer ways to implement this. You should try!
It is quite simple: the expression inside any() is a generator. any() draws from that generator and returns True (and stops) at the first element from the generator that is True. If it exhausts the generator, then it returns False.
The expression in the generator (i in seen or seen.append(i)) is a trick to express as a one-liner the logic that: if i is in the list, the expression is True and any() stops immediately, otherwise, i is added to the list and the generator continues.
The function can be significantly improved by using a set instead of a list:
def isUnique(item):
seen = set()
return not any(i in seen or seen.add(i) for i in item)
It is much faster to test for presence of an item in a set (O[1]) than in a list (O[n]).
One interesting and perhaps underappreciated aspect of this code is that it works on a (potentially infinite) generator. It will stop drawing from the generator at the first repeated item. Subsequent items that would be obtained by the generator are not evaluated at all (with potential side-effects, desirable or not).
A different approach, suitable for known and finite collections of items, would be the following:
def isUnique(items):
items = tuple(items) # in case items is a generator
return len(set(items)) == len(items)
This assumes that all the items fit in memory. Obviously this won't work if items is a generator of a very large or infinite number of elements.
I'm trying to see if lst2 is the reverse of lst1.
For the following code, why does return True have to be outside the if statement. When I put else: return False with the if statement, both of the prints return True (which is incorrect). Thank you!
def reversed_list(lst1, lst2):
for index1 in range(len(lst1)):
if lst1[index1] != lst2[(-1 - index1)]:
return False
return True
print(reversed_list([1, 2, 3], [3, 2, 1]))
print(reversed_list([1, 5, 3], [3, 2, 1]))
If you put it into the if, i.e. into the same conditionality as the return False, but after it, then it will never be executed, because the function will always have been left with the first return. Or it will always be executed inside the if, before, leaving it and thereby unintendedly overriding the intended False. This seems is what you are observing.
If you put it into the loop (but outside the if) it will be executed during the first iteration of the loop, i.e. much too early.
If you put it into the loop, but with an else, it will still be executed too early. at the first case of not False. This is still not what you want, because you only want a True when there is no False anywhere in the loop, not already at the first case of not False.
You only want to return a true boolean if the loop gets completly through without ever triggering the False. You want that because otherwise you might miss cases of False.
This is why the position you describe and use in the shown code, outside of both, the if and the loop, is the only correct way.
This approach is a lighter method to a brute force approach.
With brute force, the second list is reversed and then all the elements of both lists are compared. That's a lot of wasted resources, especially if the lists are massive in length.
The approach provided in your Q utilizes a pointer, which essentially allows you to "stop early." Instead of sorting the second list, iterate over the elements and compare them. If they match, move on to the next elements. If they do not match, escape early and return False. With this approach, you cannot return True until all the elements in the lists are compared. (Hence, return True is outside the for loop.)
The return statement immediately stops the execution of a function or method.
In this way, your function stops at the first difference and returns False. If all tested elements are equal, then the loop finishes without that return False statement and continues with the next statement which is return True .
I am not sure if this is exactly what you need, but this is a way of checking if l1 is the same as l2 reversed.
l1 = [1, 2, 3]
l2 = [3, 2, 1]
l2.reverse()
if l1 == l2:
print("Yes")
else:
print("No")
I don´t understand why it returns False, if sequence[0] is bigger than sequence[1]
sequence=[10, 1, 2, 3, 4, 5]
a=any(q for q in range(len(sequence)-1) if sequence[q]>=sequence[q+1])
print(a)
It works for the indexes bigger than 0
Your problem is that, for this list, (q for q in range(len(sequence)-1) if sequence[q]>=sequence[q+1]) is (0), and 0 is falsey.
Putting the actual indices into any kind of iterable is a red herring here - and you probably don't realise you're actually doing it. What you want to do is merely check if the predicate sequence[q]>=sequence[q+1] is true for any q. So do this instead:
any(sequence[q]>=sequence[q+1] for q in range(len(sequence)-1))
This gives an iterable of booleans, and checks if any are True or not.
First remove the any() to see what your comprehension actually gives you:
[q for q in range(len(sequence)-1) if sequence[q]>=sequence[q+1]]
>>> [0]
That is, there is one pair of numbers where the condition is true, and it is at index 0 in the original list.
any([0]) is then False because 0 is false. any() checks each item to see whether it's truthy.
Robin's solution is the usual way to do it, by using the comparison result as the yielded value. But it can be quite a lot faster to not yield false values, which you might notice if your sequence is long enough, so you could use this form:
any(True for q in range(len(sequence)-1) if sequence[q]>=sequence[q+1])
This question already has answers here:
Python "all" function with conditional generator expression returning True. Why?
(2 answers)
Closed 9 years ago.
have a question on all() operator in Python.
say
array = ["one","one","one"]
all( x=="one" for x in array ) <<--- i want to check for all "one" in array
The above seem to work. however, if i have
array = []
all( x=="one" for x in array ) <<--- this still return true to me.
The behaviour is that i want it return false if all items are not "one". How to do it? thanks
You can read all() as if it means:
It returns False if any of the items evaluates to False. True otherwise.
So an empty set will return True, because there is none that will make it false.
Generally speaking, in an empty set, all the elements fullfill any requirement you can imagine. That's a principle of logic, not of Python, BTW.
all's implementation is equivalent to this
def all(iterable):
for element in iterable:
if not element:
return False
return True
So, it returns True till any of the elements in the iterable is Falsy. In your case that didnt happen. Thats why it returns True
all always returns True for an empty list/tuple/etc. This is because, technically, every item in an empty collection fulfills any and every condition there is.
To fix the problem, you need to add some additional code to test whether your list is empty or not. Fortunately, empty lists evaluate to False in Python, so you can just do this:
>>> array = []
>>> bool(array and all(x=="one" for x in array))
False
>>> if array and all(x=="one" for x in array):
... print True
... else:
... print False
...
False
>>>
How to do it?
array and all(x=="one" for x in array)
Empty lists are false, so the result is false and it doesn't matter that the all part is true.
If you want to deal with iterables other than containers like list then it's a bit harder. I suppose you need something like this:
set(x=="one" for x in iterable) == { True }
Although if you care about speed, the following should be faster on the whole, since the version above doesn't short-circuit like all does:
def nonempty_all(iterable):
iterator = iter(iterable)
try:
if not next(iterator):
return False
except StopIteration:
return False
return all(iterator)
Numpy has a great method .all() for arrays of booleans, that tests if all the values are true. I'd like to do the same without adding numpy to my project. Is there something similar in the standard libary? Otherwise, how would you implement it?
I can of course think of the obvious way to do it:
def all_true(list_of_booleans):
for v in list_of_booleans:
if not v:
return False
return True
Is there a more elegant way, perhaps a one-liner?
There is; it is called all(), surprisingly. It is implemented exactly as you describe, albeit in C. Quoting the docs:
Return True if all elements of the iterable are true (or if the
iterable is empty). Equivalent to:
def all(iterable):
for element in iterable:
if not element:
return False
return True
New in version 2.5.
This is not limited to just booleans. Note that this takes an iterable; passing in a generator expression means only enough of the generator expression is going to be evaluated to test the hypothesis:
>>> from itertools import count
>>> c = count()
>>> all(i < 10 for i in c)
False
>>> next(c)
11
There is an equivalent any() function as well.
There is a similar function, called all().