Related
The itertools.count counter in Python (2.7.9) is very handy for thread-safe counting. How can I get the current value of the counter though?
The counter increments and returns the last value every time you call next():
import itertools
x = itertools.count()
print x.next() # 0
print x.next() # 1
print x.next() # 2
So far, so good.
I can't find a way to get the current value of the counter without calling next(), which would have the undesirable side-effect of increasing the counter, or using the repr() function.
Following on from the above:
print repr(x) # "count(3)"
So you could parse the output of repr(). Something like
current_value = int(repr(x)[6:-1])
would do the trick, but is really ugly.
Is there a way to get the current value of the counter more directly?
Another hack to get next value without advancing iterator is to abuse copy protocol:
>>> c = itertools.count()
>>> c.__reduce__()[1][0]
0
>>> next(c)
0
>>> c.__reduce__()[1][0]
1
Or just take it from object copy:
>>> from copy import copy
>>> next(copy(c))
1
Use the source, Luke!
According to module implementation, it's not possible.
typedef struct {
PyObject_HEAD
Py_ssize_t cnt;
PyObject *long_cnt;
PyObject *long_step;
} countobject;
Current state is stored in cnt and long_cnt members, and neither of them is exposed in object API. Only place where it may be retrieved is object __repr__, as you suggested.
Note that while parsing string you have to consider a non-singular increment case. repr(itertools.count(123, 4)) is equal to 'count(123, 4)' - logic suggested by you in question would fail in that case.
According to the documentation there is no way to access the current value of the function. itertools.count() is a generator method from the itertools module. As such, it is common practice to just simply assign the value of a generator's current value to a variable.
Simply store the the result of the next call:
current_value = x.next()
or ( Built-in python method for Python version ≥ 2.6 )
current_value = next(x)
You could make a wrapper function, or a utility decorator class if you would like some added syntactic sugar, but assignment is standard.
It is a generator, it wouldn't be easy to do what you want.
If you want to use it's value in several places, I'd recommend getting a value via .next() and storing it in a variable. If you are concerned that counter may be incremented between these 2 uses, you'd need to put them both in a critical section anyway.
If you don't want to pollute that counter with additional '+1's generated by those checks, you can use one more counter to count checks (put this in critical section too). Substracting latter from the former would give you what you need.
Also, are you really sure about thread-safety? Docs page has nothing about threads.
Ran into the same thing today. Here's what I ended up with:
class alt_count:
def __init__(self, start=0, step=1):
self.current = start - step
self.step = step
def __next__(self):
self.current = self.current + self.step
return self.current
Should give you almost all the itertools.count functionality plus the current property.
i = alt_count()
print(next(i)) # 0
print(next(i)) # 1
print(i.current) # 1
If the current value is not needed, I found using this simple closure also works. Note that nonlocal only works for Python version > 3.
def alt_next_maker(start=0, step=1):
res = start - step
def alt_next():
nonlocal res, step
res = res + step
return res
return alt_next
Can be used as a simple alternative if you don't want to use the itertools module.
alt_next = alt_next_maker()
print(alt_next()) # 0
print(alt_next()) # 1
The docs also mention the following as equivalent:
def count(start=0, step=1):
# count(10) --> 10 11 12 13 14 ...
# count(2.5, 0.5) -> 2.5 3.0 3.5 ...
n = start
while True:
yield n
n += step
I came across an interesting issue while trying to achieve dynamic sort.
Given the following code:
>>> l = []
>>> for i in range(2):
>>> def f():
>>> return f.v
>>> f.v = i
>>> l.append(f)
You have to be careful about how to use the functions in l:
>>> l[0]()
1
>>> l[1]()
1
>>> [h() for h in l]
[1, 1]
>>> [f() for f in l]
[0, 1]
>>> f = l[0]
>>> f()
0
>>> k = l[1]
>>> k()
0
>>> f = l[1]
>>> k()
1
>>> del f
>>> k()
NameError: global name 'f' is not defined
The behavior of the function depends on what f currently is.
What should I do to avoid this issue? How can I set a function attribute that does not depends on the function's name?
Update
Reading your comments and answers, here is my actual problem.
I have some data that I want to sort according to user input (so I don't know sorting criteria in advance). User can choose on which part of the data to apply successive sorts, and these sorts can be ascending or descending.
So my first try was to loop over the user inputs, define a function for each criterion, store this function in a list and then use this list for sorted's key like this: key=lambda x: [f(x) for f in functions]. To avoid multiplying conditions into functions themselves, I was computing some needed values before the function definition and binding them to the function (different functions with different pre-computed values).
While debugging, I understood that function attribute was not the solution here, so I indeed wrote a class with a __call__ method.
The issue is due to the fact that return f.v loads the global f, and not the one you intend.1 You can see this by disassembling the code:
>>> dis.dis(l[0])
3 0 LOAD_GLOBAL 0 (f)
3 LOAD_ATTR 1 (v)
6 RETURN_VALUE
After the loop that populates l, f is a reference to the last closure created, as you can see here:
>>> l
[<function f at 0x02594170>, <function f at 0x02594130>]
>>> f
<function f at 0x02594130>
Thus, when you call l[0](), it still loads the f that points to the last function created, and it returns 1. When you redefined f by doing f = l[0], then the global f now points to the first function.
What you seem to want is a function that has a state, which really is a class. You could therefore do something like this:
class MyFunction:
def __init__(self, v):
self.v = v
def __call__(self):
return self.v
l = [MyFunction(i) for i in range(2)]
l[0]() # 0
l[1]() # 1
Though it may be a good idea to explain your actual problem first, as there might be a better solution.
1: Why doesn't it load the global f and not the current instance, you may ask?
Recall that when you create a class, you need to pass a self argument, like so:
# ...
def my_method(self):
return self.value
self is actually a reference to the current instance of your object. That's how Python knows where to load the attribute value. It knows it has to look into the instance referenced by self. So when you do:
a.value = 1
a.my_method()
self is now a reference to a.
So when you do:
def f():
return f.v
There's no way for Python to know what f actually is. It's not a parameter, so it has to load it from elsewhere. In your case, it's loaded from the global variables.
Thus, when you do f.v = i, while you do set an attribute v for the instance of f, there's no way to know which instance you are referring to in the body of your function.
Note that what you are doing here:
def f():
return f.v
is not making a function which returns whatever its own v attribute is. It's returning whatever the f object's v attribute is. So it necessarily depends on the value of f. It's not that your v attribute "depends on the function's name". It really has nothing at all to do with the function's name.
Later, when you do
>>> f = l[0]
>>> k = l[1]
>>> k()
0
What you have done is bound k to the function at l[1]. When you call it, you of course get f.v, because that's what the function does.
But notice:
>>> k.v
1
>>> [h.v for h in l]
[0, 1]
So, a function is an object, and just like most objects, it can have attributes assigned to it (which you can access using dot notation, or the getattr() function, or inspecting the object's dictionary, etc.). But a function is not designed to access its own attributes from within its own code. For that, you want to use a class (as demonstrated by #VincentSavard).
In your particular case, the effect you seem to be after doesn't really need an "attribute" per se; you are apparently looking for a closure. You can implement a closure using a class, but a lighter-weight way is a nested function (one form of which is demonstrated by #TomKarzes; you could also use a named inner function instead of lambda).
Try this:
l = []
for i in range(2):
def f(n):
return lambda: n
l.append(f(i))
This doesn't use attributes, but creates a closure for each value of i. The value of n is then locked once f returns. Here's some sample output:
>>> [f() for f in l]
[0, 1]
As others said, return f.v looks for f name in the current scope which is equal to the last defined function.
To work around this you can simulate functions:
>>> class Function(object):
... def __init__(self, return_value):
... self.return_value = return_value
... def __call__(self):
... return self.return_value
...
>>> l = []
>>> for i in range(2):
... l.append(Function(i))
...
>>> l[0]()
>>> 0
>>> l[1]()
>>> 1
Is there a way to use variable like in operator overloading.
e.g.
a += 1
Instead of a = a + 1
in
a = max(a, some_other_variable)
The max() function is just an example.
NOTE:
My intent here is not to use the variable 'a' again, if possible. These two examples are different and not related to each other.
e.g.
a = some_function(a, b)
Here, the values returned from some_function() is assigned back to variable 'a' again.
Unless variable 'a' is a class variable I cannot access variable inside function some_function(), although if there is a way so that I can use it only once?
You cannot supplement Python's set of operators and statements directly in the Python code. However, you can write a wrapper that uses Python's language services to write a Pythonesque DSL which includes the operators you want.
I feel like you want something along these lines ...
>>> class Foo(object):
... def __iadd__(self, other):
... return max(self.num, other)
... def __init__(self, num):
... self.num = num
...
>>> a = Foo(5)
>>> a += 4
>>> print a
5
>>> a = Foo(4)
>>> a += 6
>>> a
6
But please note that I would consider this use of __iadd__ to be very impolite. Having __iadd__ return something other than self is generally inconsiderate if the type is mutable.
Instead of overloading an operator in a like the other answer, you could create a partial-like object for the second part. (I used the left shift operator for "coolness")
class partial(functools.partial):
def __rlshift__(self, val):
return self(val)
and use like this:
>>> a = 10
>>> a <<= partial(max, 20)
>>> a
20
So you don't need to mess with your variable types to execute the operation. Also you will not need to declare a new class for every function.
PS: Beware that the actual execution is max(20, a).
How do you make a repeating generator, like xrange, in Python? For instance, if I do:
>>> m = xrange(5)
>>> print list(m)
>>> print list(m)
I get the same result both times — the numbers 0..4. However, if I try the same with yield:
>>> def myxrange(n):
... i = 0
... while i < n:
... yield i
... i += 1
>>> m = myxrange(5)
>>> print list(m)
>>> print list(m)
The second time I try to iterate over m, I get nothing back — an empty list.
Is there a simple way to create a repeating generator like xrange with yield, or generator comprehensions? I found a workaround on a Python tracker issue, which uses a decorator to transform a generator into an iterator. This restarts every time you start using it, even if you didn't use all the values last time through, just like xrange. I also came up with my own decorator, based on the same idea, which actually returns a generator, but one which can restart after throwing a StopIteration exception:
#decorator.decorator
def eternal(genfunc, *args, **kwargs):
class _iterable:
iter = None
def __iter__(self): return self
def next(self, *nargs, **nkwargs):
self.iter = self.iter or genfunc(*args, **kwargs):
try:
return self.iter.next(*nargs, **nkwargs)
except StopIteration:
self.iter = None
raise
return _iterable()
Is there a better way to solve the problem, using only yield and/or generator comprehensions? Or something built into Python? So I don't need to roll my own classes and decorators?
Update
The comment by u0b34a0f6ae nailed the source of my misunderstanding:
xrange(5) does not return an iterator, it creates an xrange object. xrange objects can be iterated, just like dictionaries, more than once.
My "eternal" function was barking up the wrong tree entirely, by acting like an iterator/generator (__iter__ returns self) rather than like a collection/xrange (__iter__ returns a new iterator).
Not directly. Part of the flexibility that allows generators to be used for implementing co-routines, resource management, etc, is that they are always one-shot. Once run, a generator cannot be re-run. You would have to create a new generator object.
However, you can create your own class which overrides __iter__(). It will act like a reusable generator:
def multigen(gen_func):
class _multigen(object):
def __init__(self, *args, **kwargs):
self.__args = args
self.__kwargs = kwargs
def __iter__(self):
return gen_func(*self.__args, **self.__kwargs)
return _multigen
#multigen
def myxrange(n):
i = 0
while i < n:
yield i
i += 1
m = myxrange(5)
print list(m)
print list(m)
Using itertools its super easy.
import itertools
alist = [1,2,3]
repeatingGenerator = itertools.cycle(alist)
print(next(generatorInstance)) #=> yields 1
print(next(generatorInstance)) #=> yields 2
print(next(generatorInstance)) #=> yields 3
print(next(generatorInstance)) #=> yields 1 again!
If you write a lot of these, John Millikin's answer is the cleanest it gets.
But if you don't mind adding 3 lines and some indentation, you can do it without a custom decorator. This composes 2 tricks:
[Generally useful:] You can easily make a class iterable without implementing
.next() - just use a generator for __iter__(self)!
Instead of bothering with a constructor, you can define a one-off class inside a function.
=>
def myxrange(n):
class Iterable(object):
def __iter__(self):
i = 0
while i < n:
yield i
i += 1
return Iterable()
Small print: I didn't test performance, spawning classes like this might be wasteful. But awesome ;-)
I think the answer to that is "No". I'm possibly wrong. It may be that with some of the funky new things you can do with generators in 2.6 involving arguments and exception handling that would allow something like what you want. But those features are mostly intended for implementing semi-continuations.
Why do you want to not have your own classes or decorators? And why did you want to create a decorator that returned a generator instead of a class instance?
You can reset iterators with more_itertools.seekable, a third-party tool.
Install via > pip install more_itertools.
import more_itertools as mit
def myxrange(n):
"""Yield integers."""
i = 0
while i < n:
yield i
i += 1
m = mit.seekable(myxrange(5))
print(list(m))
m.seek(0) # reset iterator
print(list(m))
# [0, 1, 2, 3, 4]
# [0, 1, 2, 3, 4]
Note: memory consumption grows while advancing an iterator, so be wary wrapping large iterables.
use this solution:
>>> myxrange_ = lambda x: myxrange(x)
>>> print list(myxrange_(5))
... [0, 1, 2, 3, 4]
>>> print list(myxrange_(5))
... [0, 1, 2, 3, 4]
>>> for number in myxrange_(5):
... print number
...
0
1
2
3
4
>>>
and with a decorator:
>>> def decorator(generator):
... return lambda x: generator(x)
...
>>> #decorator
>>> def myxrange(n):
... i = 0
... while i < n:
... yield i
... i += 1
...
>>> print list(myxrange(5))
... [0, 1, 2, 3, 4]
>>> print list(myxrange(5))
... [0, 1, 2, 3, 4]
>>>
Simple.
Is it possible to do following without the i?
for i in range(some_number):
# do something
If you just want to do something N amount of times and don't need the iterator.
Off the top of my head, no.
I think the best you could do is something like this:
def loop(f,n):
for i in xrange(n): f()
loop(lambda: <insert expression here>, 5)
But I think you can just live with the extra i variable.
Here is the option to use the _ variable, which in reality, is just another variable.
for _ in range(n):
do_something()
Note that _ is assigned the last result that returned in an interactive python session:
>>> 1+2
3
>>> _
3
For this reason, I would not use it in this manner. I am unaware of any idiom as mentioned by Ryan. It can mess up your interpreter.
>>> for _ in xrange(10): pass
...
>>> _
9
>>> 1+2
3
>>> _
9
And according to Python grammar, it is an acceptable variable name:
identifier ::= (letter|"_") (letter | digit | "_")*
You may be looking for
for _ in itertools.repeat(None, times): ...
this is THE fastest way to iterate times times in Python.
The general idiom for assigning to a value that isn't used is to name it _.
for _ in range(times):
do_stuff()
What everyone suggesting you to use _ isn't saying is that _ is frequently used as a shortcut to one of the gettext functions, so if you want your software to be available in more than one language then you're best off avoiding using it for other purposes.
import gettext
gettext.bindtextdomain('myapplication', '/path/to/my/language/directory')
gettext.textdomain('myapplication')
_ = gettext.gettext
# ...
print _('This is a translatable string.')
Here's a random idea that utilizes (abuses?) the data model (Py3 link).
class Counter(object):
def __init__(self, val):
self.val = val
def __nonzero__(self):
self.val -= 1
return self.val >= 0
__bool__ = __nonzero__ # Alias to Py3 name to make code work unchanged on Py2 and Py3
x = Counter(5)
while x:
# Do something
pass
I wonder if there is something like this in the standard libraries?
You can use _11 (or any number or another invalid identifier) to prevent name-colision with gettext. Any time you use underscore + invalid identifier you get a dummy name that can be used in for loop.
May be answer would depend on what problem you have with using iterator?
may be use
i = 100
while i:
print i
i-=1
or
def loop(N, doSomething):
if not N:
return
print doSomething(N)
loop(N-1, doSomething)
loop(100, lambda a:a)
but frankly i see no point in using such approaches
Instead of an unneeded counter, now you have an unneeded list.
Best solution is to use a variable that starts with "_", that tells syntax checkers that you are aware you are not using the variable.
x = range(5)
while x:
x.pop()
print "Work!"
I generally agree with solutions given above. Namely with:
Using underscore in for-loop (2 and more lines)
Defining a normal while counter (3 and more lines)
Declaring a custom class with __nonzero__ implementation (many more lines)
If one is to define an object as in #3 I would recommend implementing protocol for with keyword or apply contextlib.
Further I propose yet another solution. It is a 3 liner and is not of supreme elegance, but it uses itertools package and thus might be of an interest.
from itertools import (chain, repeat)
times = chain(repeat(True, 2), repeat(False))
while next(times):
print 'do stuff!'
In these example 2 is the number of times to iterate the loop. chain is wrapping two repeat iterators, the first being limited but the second is infinite. Remember that these are true iterator objects, hence they do not require infinite memory. Obviously this is much slower then solution #1. Unless written as a part of a function it might require a clean up for times variable.
We have had some fun with the following, interesting to share so:
class RepeatFunction:
def __init__(self,n=1): self.n = n
def __call__(self,Func):
for i in xrange(self.n):
Func()
return Func
#----usage
k = 0
#RepeatFunction(7) #decorator for repeating function
def Job():
global k
print k
k += 1
print '---------'
Job()
Results:
0
1
2
3
4
5
6
---------
7
If do_something is a simple function or can be wrapped in one, a simple map() can do_something range(some_number) times:
# Py2 version - map is eager, so it can be used alone
map(do_something, xrange(some_number))
# Py3 version - map is lazy, so it must be consumed to do the work at all;
# wrapping in list() would be equivalent to Py2, but if you don't use the return
# value, it's wastefully creating a temporary, possibly huge, list of junk.
# collections.deque with maxlen 0 can efficiently run a generator to exhaustion without
# storing any of the results; the itertools consume recipe uses it for that purpose.
from collections import deque
deque(map(do_something, range(some_number)), 0)
If you want to pass arguments to do_something, you may also find the itertools repeatfunc recipe reads well:
To pass the same arguments:
from collections import deque
from itertools import repeat, starmap
args = (..., my args here, ...)
# Same as Py3 map above, you must consume starmap (it's a lazy generator, even on Py2)
deque(starmap(do_something, repeat(args, some_number)), 0)
To pass different arguments:
argses = [(1, 2), (3, 4), ...]
deque(starmap(do_something, argses), 0)
We can use the while & yield, we can create our own loop function like this. Here you can refer to the official documentation.
def my_loop(start,n,step = 1):
while start < n:
yield start
start += step
for x in my_loop(0,15):
print(x)
#Return first n items of the iterable as a list
list(itertools.islice(iterable, n))
Taken from http://docs.python.org/2/library/itertools.html
If you really want to avoid putting something with a name (either an iteration variable as in the OP, or unwanted list or unwanted generator returning true the wanted amount of time) you could do it if you really wanted:
for type('', (), {}).x in range(somenumber):
dosomething()
The trick that's used is to create an anonymous class type('', (), {}) which results in a class with empty name, but NB that it is not inserted in the local or global namespace (even if a nonempty name was supplied). Then you use a member of that class as iteration variable which is unreachable since the class it's a member of is unreachable.
What about:
while range(some_number):
#do something