When a function is called by unpacking arguments, it seems to increase the recursion depth twice. I would like to know why this happens.
Normally:
depth = 0
def f():
global depth
depth += 1
f()
try:
f()
except RuntimeError:
print(depth)
#>>> 999
With an unpacking call:
depth = 0
def f():
global depth
depth += 1
f(*())
try:
f()
except RuntimeError:
print(depth)
#>>> 500
In theory both should reach about 1000:
import sys
sys.getrecursionlimit()
#>>> 1000
This happens on CPython 2.7 and CPython 3.3.
On PyPy 2.7 and PyPy 3.3 there is a difference, but it is much smaller (1480 vs 1395 and 1526 vs 1395).
As you can see from the disassembly, there is little difference between the two, other than the type of call (CALL_FUNCTION vs CALL_FUNCTION_VAR):
import dis
def f():
f()
dis.dis(f)
#>>> 34 0 LOAD_GLOBAL 0 (f)
#>>> 3 CALL_FUNCTION 0 (0 positional, 0 keyword pair)
#>>> 6 POP_TOP
#>>> 7 LOAD_CONST 0 (None)
#>>> 10 RETURN_VALUE
def f():
f(*())
dis.dis(f)
#>>> 47 0 LOAD_GLOBAL 0 (f)
#>>> 3 BUILD_TUPLE 0
#>>> 6 CALL_FUNCTION_VAR 0 (0 positional, 0 keyword pair)
#>>> 9 POP_TOP
#>>> 10 LOAD_CONST 0 (None)
#>>> 13 RETURN_VALUE
The exception message actually offers you a hint. Compare the non-unpacking option:
>>> import sys
>>> sys.setrecursionlimit(4) # to get there faster
>>> def f(): f()
...
>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in f
File "<stdin>", line 1, in f
File "<stdin>", line 1, in f
RuntimeError: maximum recursion depth exceeded
with:
>>> def f(): f(*())
...
>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in f
File "<stdin>", line 1, in f
RuntimeError: maximum recursion depth exceeded while calling a Python object
Note the addition of the while calling a Python object. This exception is specific to the PyObject_CallObject() function. You won't see this exception when you set an odd recursion limit:
>>> sys.setrecursionlimit(5)
>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in f
File "<stdin>", line 1, in f
RuntimeError: maximum recursion depth exceeded
because that is the specific exception raised in the ceval.c frame evaluation code inside PyEval_EvalFrameEx():
/* push frame */
if (Py_EnterRecursiveCall(""))
return NULL;
Note the empty message there. This is a crucial difference.
For your 'regular' function (no variable arguments), what happens is that an optimized path is picked; a Python function that doesn't need tuple or keyword argument unpacking support is handled directly in the fast_function() function of the evaluation loop. A new frameobject with the Python bytecode object for the function is created, and run. This is one recursion check.
But for a function call with variable arguments (tuple or dictionary or both), the fast_function() call cannot be used. Instead, ext_do_call() (extended call) is used, which handles the argument unpacking, then uses PyObject_Call() to invoke the function. PyObject_Call() does a recursion limit check, and 'calls' the function object. The function object is invoked via the function_call() function, which calls PyEval_EvalCodeEx(), which calls PyEval_EvalFrameEx(), which makes the second recursion limit check.
TL;DR version
Python functions calling Python functions are optimised and bypass the PyObject_Call() C-API function, unless argument unpacking takes place. Both Python frame execution and PyObject_Call() make recursion limit tests, so bypassing PyObject_Call() avoids incrementing the recursion limit check per call.
More places with 'extra' recursion depth checks
You can grep the Python source code for Py_EnterRecursiveCall for other locations where recursion depth checks are made; various libraries, such as json and pickle use it to avoid parsing structures that are too deeply nested or recursive, for example. Other checks are placed in the list and tuple __repr__ implementations, rich comparisons (__gt__, __lt__, __eq__, etc.), handling the __call__ callable object hook and handling __str__ calls.
As such, you can hit the recursion limit much faster still:
>>> class C:
... def __str__(self):
... global depth
... depth += 1
... return self()
... def __call__(self):
... global depth
... depth += 1
... return str(self)
...
>>> depth = 0
>>> sys.setrecursionlimit(10)
>>> C()()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in __call__
File "<stdin>", line 5, in __str__
RuntimeError: maximum recursion depth exceeded while calling a Python object
>>> depth
2
Related
I was revisiting some dynamic programming concepts and I wrote a code to calculate Fibonacci with memoization.
Here is the code:
def fib(n,memo={}):
if(n in memo):
return memo[n]
if(n <= 2):
return 1
memo[n]=fib(n-1,memo) + fib(n-2,memo)
return memo[n]
Now I ran some test cases and here are the results
>>> fib(2)
1
>>> fib(1000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in fib
File "<stdin>", line 6, in fib
File "<stdin>", line 6, in fib
[Previous line repeated 995 more times]
File "<stdin>", line 4, in fib
RecursionError: maximum recursion depth exceeded in comparison
>>> fib(6)
8
>>> fib(10)
55
>>> fib(100)
354224848179261915075
.
.
.
.
>>> fib(980)
2873442049110331124686847975839596483580184681281842823699466692000268066325404550898791458706068536914736664630537864515125212890415097163803163111745199726085365105
>>> fib(990)
3534100091787525753399448335204590682849450463581549776041091752538906966342713601215835661100647255108360758515849851434123968685864251091027232911065706187500753920
>>> fib(999)
2686381002448535938614672720214292396761660931898695234012317599761798170024788168933836965448335656419182785616144335631297667364221035032463485041037768036733415116
>>> fib(1000)
4346655768693745643568852767504062580256466051737178040248172908953655541794905189040387984007925516929592259308032263477520968962323987332247116164299644090653318795
When I ran fib(1000) the first time, it said maximum recursion depth exceeded. However, when I gradually increased n, fib(1000) worked fine.
Then I tried fib(2000) and got the same exception.
>>> fib(2000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in fib
File "<stdin>", line 6, in fib
File "<stdin>", line 6, in fib
[Previous line repeated 995 more times]
File "<stdin>", line 4, in fib
RecursionError: maximum recursion depth exceeded in comparison
I tried gradually increasing n and it worked fine again:
>>> fib(1200)
2726988445540627015799161531364219870500077999291772582118050289497472647637302680948250928456231003117017238012762721449359761674385644301603997220584740591763466070
>>> fib(1500)
1355112566856310195163693686714840837778601071241849724213354315322148731087352875061225935403571726530037377881434732025769925708235655004534991410292424959599748390
>>> fib(1700)
8501653935514177120246392248625833924052052390491381030300605977750345588982825628424071479174753549360050542305550855066813804919653208931716726270523366654632196915
>>> fib(1900)
5333735470177196739708654380013216364182711606231750028692155598599810955874132791398352277818697705852238294681640540003099177608752396895596802978549351480795061055
>>> fib(2000)
4224696333392304878706725602341482782579852840250681098010280137314308584370130707224123599639141511088446087538909603607640194711643596029271983312598737326253555805
>>> fib(2500)
1317090516751949629522763087125316412066606964992507141887746936727530870405038425764503130123186407746570862185871925952766836352119119528156315582632460790383834605
>>> fib(2900)
5184080332847202181832545365520373859688699234105705045492742368770388504951261158081878962852500283133276036303031796698449718008155302155556519351587134410081144235
>>> fib(3000)
4106158863079712603335683787192671052201251086373692524088854309269055842741134037313304916608500445608300368357069422745885693621454765026743730454468521604866062920
Same thing happens if I run fib(4000) immediately afterwards, but it works fine if I gradually increase. I am basically trying to understand why is that the case. The memo object is not global and should be initialized in the first call to the function, so successively increasing n to 1000 should, in theory, be no different than directly calling fib(1000).
This is because if memo is empty, the recursion needs to go all the way to the base case of n <= 2. So if you immediately start with a first call that is fib(1000) you may bump into a stack overflow.
However, when you start with smaller values, like the call of fib(10), memo will collect lots of results, including for 10 and 9. And so the next time you make a call increasing the argument that you pass, it doesn't have to recur all the way to 2, but can already back track when it reaches 9 or 10, as it will find it already available in memo.
Note that memo only initialises to {} at the moment the function is defined, so you just keep extending it, thereby reducing the need to use the call stack for deep recursions.
As trincot mentioned, there's no data present, so the code will just go and find another call to the function instead of the value until the value is computable within the stack limit. A call to the function, in theory, reserves some space somewhere (stack) where there is a returning point from the function, func's arguments and perhaps something else. What's happening for you is, that you repeat the storing or return point + args + other stuff N times when the memo is empty AND you haven't started with any computation yet, therefore the only output is either recursion exhaustion or storage (stack) exhaustion, depending on the two sizes.
However, you get a nice exception, so when such a case occurs, simply catch it, split the number into parts (e.g. divide by two/four/...) and return again (recursively). This way, even if you blow the stack with your recursion calls, you'll recursively find a smaller number which will fill the memo and then you slowly bootstrap your way to the larger numbers.
When wrapping an (internal) iterator one often has to reroute the __iter__ method to the underlying iterable. Consider the following example:
class FancyNewClass(collections.Iterable):
def __init__(self):
self._internal_iterable = [1,2,3,4,5]
# ...
# variant A
def __iter__(self):
return iter(self._internal_iterable)
# variant B
def __iter__(self):
yield from self._internal_iterable
Is there any significant difference between variant A and B?
Variant A returns an iterator object that has been queried via iter() from the internal iterable. Variant B returns a generator object that returns values from the internal iterable. Is one or the other preferable for some reason? In collections.abc the yield from version is used. The return iter() variant is the pattern that I have used until now.
The only significant difference is what happens when an exception is raised from within the iterable. Using return iter() your FancyNewClass will not appear on the exception traceback, whereas with yield from it will. It is generally a good thing to have as much information on the traceback as possible, although there could be situations where you want to hide your wrapper.
Other differences:
return iter has to load the name iter from globals - this is potentially slow (although unlikely to significantly affect performance) and could be messed with (although anyone who overwrites globals like that deserves what they get).
With yield from you can insert other yield expressions before and after (although you could equally use itertools.chain).
As presented, the yield from form discards any generator return value (i.e. raise StopException(value). You can fix this by writing instead return (yield from iterator).
Here's a test comparing the disassembly of the two approaches and also showing exception tracebacks: http://ideone.com/1YVcSe
Using return iter():
3 0 LOAD_GLOBAL 0 (iter)
3 LOAD_FAST 0 (it)
6 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
9 RETURN_VALUE
Traceback (most recent call last):
File "./prog.py", line 12, in test
File "./prog.py", line 10, in i
RuntimeError
Using return (yield from):
5 0 LOAD_FAST 0 (it)
3 GET_ITER
4 LOAD_CONST 0 (None)
7 YIELD_FROM
8 RETURN_VALUE
Traceback (most recent call last):
File "./prog.py", line 12, in test
File "./prog.py", line 5, in bar
File "./prog.py", line 10, in i
RuntimeError
I've been playing around with memoization and recursion in python 3.3
Ignoring the fact that python is the wrong language to be doing this in, I've found that I get inconsistent results between using functools.lru_cache to memoize, and not using functools.lru_cache
I'm not changing the recursion limit - it stays at the default, which for me is 1000.
To test the problem, I've written up a simple recursive function to sum numbers from 1 through i
#!/usr/bin/python
def sumtil(i):
"""Recursive function to sum all numbers from 1 through i"""
# Base case, the sum of all numbers from 1 through 1 is 1...
if i == 1:
return 1
else:
return i+sumtil(i-1)
# This will not throw an exception
sumtil(998)
# This will throw an exception
sumtil(999)
Running this function normally, I can run sumtil(998) comfortably without hitting the recursion limit. sumtil(999) or above will throw an exception.
However, if I try decorating this function with #functools.lru_cache(), the recursion limit exception is thrown 3 times earlier, when running sumtil(333)
#!/usr/bin/python
import functools
#functools.lru_cache(maxsize=128)
def sumtil(i):
"""Recursive function to sum all numbers from 1 through i"""
# Base case, the sum of all numbers from 1 through 1 is 1...
if i == 1:
return 1
else:
return i+sumtil(i-1)
# This will not throw an exception
sumtil(332)
# This will throw an exception
sumtil(333)
Being that 332*3 = 996, but 333*3 = 999, it appears to me that the lru_cache decorator is causing each level of recursion in my function to become three levels of recursion.
Why do I get three times as many levels of recursion when using functools.lru_cache to memoize a function?
Because a decorator is an extra function, so it "uses" one level in the stack. Example:
>>> def foo(f):
... def bar(i):
... if i == 1:
... raise Exception()
... return f(i)
... return bar
...
>>> #foo
... def sumtil(i):
... if i == 1:
... return 1
... else:
... return i+sumtil(i-1)
...
>>> sumtil(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in bar
File "<stdin>", line 6, in sumtil
File "<stdin>", line 5, in bar
File "<stdin>", line 6, in sumtil
File "<stdin>", line 4, in bar
Exception
>>>
Besides, if the decorator uses argument packing/unpacking, then an extra level is used (though I'm not knowledgeable enough about the Python runtime to explain why that happens).
def foo(f):
def bar(*args,**kwargs):
return f(*args,**kwargs)
return bar
Max. recursion depth exceeded:
undecorated: 1000
w/o packing: 500
with packing: 334
In Python it is possible to call either del x or del (x) . I know how to define a function called F(x) , but I do not know how to define a function that cal be called like del, without a tuple as parameters.
What is the difference between F x and F(x), and how can I define a function that can be called without parenthesis ?
>>> a = 10
>>> a
10
>>> del a <------------ can be called without parenthesis
>>> a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> a = 1
>>> del (a)
>>> a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> def f(x): 1
...
>>> f (10)
>>> print f (10)
None
>>> def f(x): return 1
...
>>> print f (10)
1
>>> f 1 <------ cannot be called so
File "<stdin>", line 1
f 1
^
SyntaxError: invalid syntax
>>>
The main reason is that del is actually a statement and therefore has special behavior in Python. Therefore you cannot actually define these (and this behavior) yourself* - it is a built-in part of the language for a set of reserved keywords.
**I guess you could potentially edit the source of Python itself and build your own in, but I don't think that is what you're after :)*
Why/how does this create a seemingly infinite loop? Incorrectly, I assumed this would cause some form of a stack overflow type error.
i = 0
def foo () :
global i
i += 1
try :
foo()
except RuntimeError :
# This call recursively goes off toward infinity, apparently.
foo()
foo()
print i
The RuntimeError exception will be raised if the recursion limit is exceeded.
Since you're catching this exception, your machine is going to continue on, but you're only adding to a single global int value, which doesn't use much memory.
You can set the recursion limit with sys.setrecursionlimit().
The current limit can be found with sys.getrecursionlimit().
>>> import sys
>>> sys.setrecursionlimit(100)
>>>
>>> def foo(i):
... i += 1
... foo(i)
...
>>> foo(1)
Traceback (most recent call last):
...
File "<stdin>", line 3, in foo
RuntimeError: maximum recursion depth exceeded
>>>
If you want to run out of memory try consuming more of it.
>>> def foo(l):
... l = l * 100
... foo(l)
...
>>> foo(["hello"])
Traceback (most recent call last):
...
File "<stdin>", line 2, in foo
MemoryError
>>>
If you change the code to
i = 0
def foo ():
global i
i += 1
print i
try :
foo()
except RuntimeError :
# This call recursively goes off toward infinity, apparently.
foo()
finally:
i -= 1
print i
foo()
You'll observe that the output oscillates short below 999 (1000 being Python's default recursion limit). That means, when the limit is hit (RuntimeError) that last call of foo() is terminated, and another one is set off to replace it immediately.
If you raise a KeyboardInterrupt you'll observe how the entire trace is being terminated at once.
UPDATE
Interestingly the second call of foo() is not protected by the try ... except-block anymore. Therefore the application will in fact terminate eventually. This becomes apparant if you set the recursion limit to a smaller number, e.g. the output for sys.setrecursionlimit(3):
$ python test.py
1
2
1
2
1
0
Traceback (most recent call last):
File "test.py", line 19, in <module>
foo()
File "test.py", line 14, in foo
foo()
File "test.py", line 14, in foo
foo()
RuntimeError