Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I’m trying to build a scoring matrix using recursive,
for i in range(1, len(str1))
for j in range(1, len(str))
#do something
my code:
def matrix_bulid(index_1, index_2):
print(index_1, index_2)
if index_1 == len(first_dna) and index_2 == len(second_dna):
return
elif index_2 == len(second_dna):
return matrix_bulid(index_1 + 1, 1)
else:
#do something
matrix_bulid(index_1, index_2 + 1)
but in really long strings , i get max depth error thing.
does anyone has any idea how to do it?
If your goal is to turn your nested for loop into a simple recursive function, you've done that successfully. There are some bugs (bad indentation, not returning anything in the second recursive case, …), but the basic structure is sound.
Unfortunately, recursive functions are bounded by the depth of the stack. Some languages have tail call elimination, which means tail-recursive functions aren't bounded. And your implementation is tail-recursive. But Python (intentionally) does not have that feature, so that doesn't matter; you can still only deal with strings up to sys.getrecursionlimit()+1 in length.
If your strings are bounded, but just a little too big for the default recursion limit (which is 1000 in CPython), you can use sys.setrecursionlimit to set it higher.
There are also tricks to simulate tail call elimination in Python, which I'll discuss below. But even with the best possible trick, your recursive code is still longer, less obvious, less Pythonic, and slower than your nested loop.
If you're doing this for a homework assignment, you're probably done. Unless you were given a choice of programming language, in which case you will need to choose a language with tail call elimination. If you're doing this for real code, you should stick with your original nested loops—the code is simpler, it will run faster, and there is no stack limit, without needing anything hacky or complicated.
The most efficient way to implement tail-call optimization in CPython is by modifying the bytecode of compiled functions, as here. But that's also the hackiest way, and it only works in a restricted set of circumstances.
The simplest way is to modify your function to return a function that returns a value, instead of a value, and then use a trampoline to chain the evaluations together (there are at least two blog posts that show this technique, here and here). But, while that's simplest to implement, it requires changes through your actual recursive functions, which make them more complicated and less readable.
The best tradeoff is probably to use a decorator that inserts a trampoline in the middle of each recursive step, so you don't need to delay function calls explicitly, as seen here. This can get a bit tricky if you have mutually-recursive functions, but it's rarely hard to figure out. And the implementation is pretty easy to understand. And it's not much slower than the bytecode hackery.
Related
I've seen a few examples of getting Python to do tail call optimization by using a while True loop. E.g.
def tail_exp(base, exponent, acc=1):
if exponent == 0:
return acc
return tail_exp(base, exponent - 1, acc * base)
becomes
def tail_exp_2(base, exponent, acc=1):
while True:
if exponent == 0:
return acc
exponent, acc = exponent - 1, acc * base
I'm curious to know if this technique is applicable to all/most recursive algorithms in Python, and if there are any downsides or "gotchas" to look out for when optimizing recursive algorithms in this way?
Any recursive algorithm can be replaced by an iterative one. However, some examples will require an additional stack be added to the code, to manage state that is handled by the recursive calls in the original form. With tail recursion, there is no state to be managed, so no separate stack is needed.
Some programming languages take advantage of that fact and design their compilers to optimize out tail calls in recursive code, producing machine code that is equivalent to a loop. Python does not do tail call optimization, so this isn't really relevant to your question. Rewriting code by hand is not tail call optimization, it's just a particular sort of refactoring.
There are a few reasons Python chooses not to do tail call optimization. It's not because it's impossible. Python code is compiled into byte code, so at least theoretically there's an opportunity to translate a recursive call into a loop if that was desired by the developers (in practice it's a little more complicated, since Python variable names are dynamic, you can't necessarily tell if a function name refers to what you expect it to at runtime, a fact use by techniques like monkeypatching). However, the biggest problem with tail call optimization is that it generally overwrites useful debugging information that would usually be preserved by a call stack, like exactly how deep in the recursion you are and the exact state of those previous function calls. The Python developers have decided that they prefer the simplicity and debuggability of normal recursion over performance benefits of tail call optimization, even when the latter is possible.
If you want to rewrite an algorithm from a recursive implementation into an iterative one, you can always do so. In some cases, though, it may get a lot more complicated. Recursive implementations of some algorithms can be a lot shorter, simpler, and easier to reason about, even though iterative equivalents may be faster (and won't hit the recursion limit for large inputs). Converting tail calls into a loop is usually quite simple though. The complicated cases are generally not amenable to tail call optimization either, since they're doing complicated stuff with the values returned by their recursion.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Imagine I am defining a function (in this case it is in Python), which does some simple manipulations with a string and returns a string too. I can, for example, define it in two different ways, that will produce the same output:
def signature(s):
"""Returns the signature of this string.
Signature is a string that contains all of the letters in order.
s: str
"""
1) return ''.join(sorted(list(s)))
2) t = list(s)
t.sort()
t = ''.join(t)
return t
Is there a benefit to stick to any of these two ways in the long run? I have seen different developers practising both methods, but is it just a matter of taste?
Would appreciate any feedback to help me establish better programming habits.
Simply put, it's up to your project style guide.
This isn't a performance question, as both these statements take the same time to run, benchmark it if you'd like. Therefore, the difference is in syntax and syntax alone; and when choosing your syntax, it's best to conform to your styleguide.
If you're working in a corporate production setting, you could look around your project to find best practices. If you're working alone, you can decide which one you like better. Either way, consistency is the way to go when aiming for sustainable readability on your code.
The first statement reminds me personally of Linq syntax, which aims make your code readable as is by chaining functions together. The second statement would be easier for a beginner to read, as the different functions are separated in their own lines.
As John specified in your comments, return ''.join(sorted(s)) would suffice for what you're trying to do, and it's likely one of the shortest ways to implement this code.
If you want to read more about different style guides and creating your own, I'd recommend further reading about linters.
There is a basic canon hidden in the this module. Execute
import this
to get it printed:
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
See https://www.python.org/dev/peps/pep-0020/
Your example seems to fall under
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
and would lead me to return ''.join(sorted(s)) because list(s) is not needed as strings are iterables and can be fed into sorted() as is.
If I defined a function like this:
def ccid_year(seq):
year, prefix, index, suffix = seq
return year
Is Python allowed to optimize it to be effectively:
def ccid_year(seq):
return seq[0]
I'd prefer to write the first function because it documents the format of the data being passed in but would hope that Python would generate code that is effectively as efficient as the second definition.
The two functions are not equivalent:
def ccid_year_1(seq):
year, prefix, index, suffix = seq
return year
def ccid_year_2(seq):
return seq[0]
arg = {1:'a', 2:'b', 0:'c', 3:'d'}
print ccid_year_1(arg)
print ccid_year_2(arg)
The first call prints 0 and the second prints c.
I'll answer the question at face value later, but first: When in doubt, benchmark it! But first, recall that most time is spent in a small portion of the code (i.e., most code is irrelevant to performance!) and, in CPython, function call overhead usually dominates small inefficiencies. Not to mention that large-scale algorithmic inefficiencies (a.k.a. freaking stupid code) dwarfs micro-optimization concerns.
So either don't worry about this at all, or if you have reason to worry about it, first benchmark alternatives and second don't put it in a function. Note that "reasons to worry about it" must be weighted against the time spent worrying, and the maintenance burden (if there is one) of the manual optimization.
CPython, the reference implementation you most like use, is very conservative about optimizing at this level. While there is a peephole optimizer operating on bytecode, it is limited in scale. More generally, you can't expect much optimization crossing a single statement. The problem with statically optimizing Python code is that there's a billion ways even the most innocently-looking program frament can call into arbitrary code, which might do anything at all, so you can't omit these calls.
While we're at it, your proposed optimization is invalid (in the sense that the program doesn't have the same behavior) if seq is of the wrong type (not a sequence, or a very weird sequence) or length (not exactly three items long)! Any program claiming to implement Python must maintain such differences, so it won't do the transformation you suggest literally. I assume this was just an off-hand illustration, but it does indicate you seriously underestimate how complex Python is (to implement, and doubly so to optimize). I and others have written about this at length before, so I'll stop now before this post becomes even larger.
PyPy on the other hand will, if this function is indeed called from a hot loop, probably optimize this and a million other things you didn't even think of, while compiling it down to a machine code loop that iterates faster than any Python loop could ever iterate on CPython. It will still contain a few checks to break out of the loop and take the proper action (e.g. raise an exception) if necessary, but they'll also be highly efficient if not triggered.
I do not know much about IronPython and Jython and other implementations, but if their lack of consistent several-times-faster-than-CPython benchmark results is any indicator, they do not perform significant optimizations. While the VMs IronPython and Jython include JIT compilers (not - but not quite - entirely unlike PyPy's), these JIT compilers are built for very different languages, and I'd be very surprised if they could look through the mess of code IronPython/Jython must execute to achieve Python semantics and perform such optimizations on it.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I've been working on one of the coding challenges on InterviewStreet.com and I've run into a bit of an efficiency problem. Can anyone suggest where I might change the code to make it faster and more efficient?
Here's the code
Here's the problem statement if you're interested
If your question is about optimising python code generally (which I think it should be ;) then there are all sorts of intesting things you can do, but first:
You probably shouldn't be obsessively optimising python code! If you're using the fastest algorithm for the problem you're trying to solve and python doesn't do it fast enough you should probably be using a different language.
That said, there are several approaches you can take (because sometimes, you really do want to make python code faster):
Profile (do this first!)
There are lots of ways of profiling python code, but there are two that I'll mention: cProfile (or profile) module, and PyCallGraph.
cProfile
This is what you should actually use, though interpreting the results can be a bit daunting.
It works by recording when each function is entered or exited, and what the calling function was (and tracking exceptions).
You can run a function in cProfile like this:
import cProfile
cProfile.run('myFunction()', 'myFunction.profile')
Then to view the results:
import pstats
stats = pstats.Stats('myFunction.profile')
stats.strip_dirs().sort_stats('time').print_stats()
This will show you in which functions most of the time is spent.
PyCallGraph
PyCallGraph provides a prettiest and maybe the easiest way of profiling python programs -- and it's a good introduction to understanding where the time in your program is spent, however it adds significant execution overhead
To run pycallgraph:
pycallgraph graphviz ./myprogram.py
Simple! You get a png graph image as output (perhaps after a while...)
Use Libraries
If you're trying to do something in python that a module already exists for (maybe even in the standard library), then use that module instead!
Most of the standard library modules are written in C, and they will execute hundreds of times faster than equivilent python implementations of, say, bisection search.
Make the Interpreter do as Much of Your Work as You Can
The interpreter will do some things for you, like looping. Really? Yes! You can use the map, reduce, and filter keywords to significantly speed up tight loops:
consider:
for x in xrange(0, 100):
doSomethingWithX(x)
vs:
map(doSomethingWithX, xrange(0,100))
Well obviously this could be faster because the interpreter only has to deal with a single statement, rather than two, but that's a bit vague... in fact, this is faster for two reasons:
all flow control (have we finished looping yet...) is done in the interpreter
the doSomethingWithX function name is only resolved once
In the for loop, each time around the loop python has to check exactly where the doSomethingWithX function is! even with cacheing this is a bit of an overhead.
Remember that Python is an Interpreted Language
(Note that this section really is about tiny tiny optimisations that you shouldn't let affect your normal, readable coding style!)
If you come from a background of a programming in a compiled language, like c or Fortran, then some things about the performance of different python statements might be surprising:
try:ing is cheap, ifing is expensive
If you have code like this:
if somethingcrazy_happened:
uhOhBetterDoSomething()
else:
doWhatWeNormallyDo()
And doWhatWeNormallyDo() would throw an exception if something crazy had happened, then it would be faster to arrange your code like this:
try:
doWhatWeNormallyDo()
except SomethingCrazy:
uhOhBetterDoSomething()
Why? well the interpreter can dive straight in and start doing what you normally do; in the first case the interpreter has to do a symbol look up each time the if statement is executed, because the name could refer to something different since the last time the statement was executed! (And a name lookup, especially if somethingcrazy_happened is global can be nontrivial).
You mean Who??
Because of cost of name lookups it can also be better to cache global values within functions, and bake-in simple boolean tests into functions like this:
Unoptimised function:
def foo():
if condition_that_rarely_changes:
doSomething()
else:
doSomethingElse()
Optimised approach, instead of using a variable, exploit the fact that the interpreter is doing a name lookup on the function anyway!
When the condition becomes true:
foo = doSomething # now foo() calls doSomething()
When the condition becomes false:
foo = doSomethingElse # now foo() calls doSomethingElse()
PyPy
PyPy is a python implementation written in python. Surely that means it will run code infinitely slower? Well, no. PyPy actually uses a Just-In-Time compiler (JIT) to run python programs.
If you don't use any external libraries (or the ones you do use are compatible with PyPy), then this is an extremely easy way to (almost certainly) speed up repetitive tasks in your program.
Basically the JIT can generate code that will do what the python interpreter would, but much faster, since it is generated for a single case, rather than having to deal with every possible legal python expression.
Where to look Next
Of course, the first place you should have looked was to improve your algorithms and data structures, and to consider things like caching, or even whether you need to be doing so much in the first place, but anyway:
This page of the python.org wiki provides lots of information about how to speed up python code, though some of it is a bit out of date.
Here's the BDFL himself on the subject of optimising loops.
There are quite a few things, even from my own limited experience that I've missed out, but this answer was long enough already!
This is all based on my own recent experiences with some python code that just wasn't fast enough, and I'd like to stress again that I don't really think any of what I've suggested is actually a good idea, sometimes though, you have to....
First off, profile your code so you know where the problems lie. There are many examples of how to do this, here's one: https://codereview.stackexchange.com/questions/3393/im-trying-to-understand-how-to-make-my-application-more-efficient
You do a lot of indexed access as in:
for pair in range(i-1, j):
if coordinates[pair][0] >= 0 and coordinates[pair][1] >= 0:
Which could be written more plainly as:
for coord in coordinates[i-1:j]:
if coord[0] >= 0 and cood[1] >= 0:
List comprehensions are cool and "pythonic", but this code would probably run faster if you didn't create 4 lists:
N = int(raw_input())
coordinates = []
coordinates = [raw_input() for i in xrange(N)]
coordinates = [pair.split(" ") for pair in coordinates]
coordinates = [[int(pair[0]), int(pair[1])] for pair in coordinates]
I would instead roll all those together into one simple loop or if you're really dead set on list comprehensions, encapsulate the multiple transformations into a function which operates on the raw_input().
This answer shows how I locate code to optimize.
Suppose there is some line of code you could replace, and it is costing, say, 40% of the time.
Then it resides on the call stack 40% of the time.
If you take 10 samples of the call stack, it will appear on 4 of them, give or take.
It really doesn't matter how many samples show it.
If it appears on two or more, and if you can replace it, you will save whatever time it costs.
Most of the interview street problems seem to be tested in a way that will verify that you have found an algorithm with the right big O complexity rather than that you have coded the solution in the most optimal way possible.
In other words if you are failing some of the test cases due to running out of time the problem is likely that you need to figure out a solution with lower algorithmic complexity rather than micro-optimize the algorithm you have. This is why they generally state that N can be quite large.
Apparently, there's been a big brouhaha over whether or not Python needs tail-call optimization (TCO). This came to a head when someone shipped Guido a copy of SICP, because he didn't "get it." I'm in the same boat as Guido. I understand the concept of tail-call optimization. I just can't think of any reason why Python really needs it.
To make this easier for me to understand, what would be a snippet of code that would be greatly simplified using TCO?
Personally, I put great value on tail call optimization; but mainly because it makes recursion as efficient as iteration (or makes iteration a subset of recursion). In minimalistic languages you get huge expressive power without sacrificing performance.
In a 'practical' language (like Python), OTOH, you usually have a lot of other constructions for almost every situation imaginable, so it's less critical. It is always a good thing to have, to allow for unforeseen situations, of course.
Personally, I put great value on tail call optimization; but mainly because it makes recursion as efficient as iteration (or makes iteration a subset of recursion). In minimalistic languages you get huge expressive power without sacrificing performance.
In a 'practical' language (like Python), OTOH, you usually have a lot of other constructions for almost every situation imaginable, so it's less critical. It is always a good thing to have, to allow for unforeseen situations, of course.
If you intensely want to use recursion for things that might alternatively be expressed as loops, then "tail call optimization" is really a must. However, Guido, Python's Benevolent Dictator For Life (BDFL), strongly believes in loops being expressed as loops -- so he's not going to special-case tail calls (sacrificing stack-trace dumps and debugging regularity).
Tail-call optimization makes it easier to write recursive functions without worrying about a stack overflow:
def fac(n, result=1):
if n > 1:
return fac(n - 1, n * result)
return result
Without tail-call optimization, calling this with a big number could overflow the stack.
Guido recognized in a follow up post that TCO allowed a cleaner the implementation of state machine as a collection of functions recursively calling each other. However in the same post he proposes an alternative equally cleaner solution without TCO.