I know this question is kind of stupid, maybe it's a just a part of writing code but it seems defining simple functions can really hurt performance severely... I've tried this simple test:
def make_legal_foo_string(x):
return "This is a foo string: " + str(x)
def sum_up_to(x):
return x*(x+1)/2
def foo(x):
return [make_legal_foo_string(x),sum_up_to(x),x+1]
def bar(x):
return ''.join([str(foo(x))," -- bar !! "])
it's very good style and makes code clear but it can be three times as slow as just writing it literally. It's inescapable for functions that can have side effects but it's actually almost trivial to define some functions that just should literally be replaced with lines of code every time they appear, translate the source code into that and only then compile. Same I think for magic numbers, it doesn't take a lot of time to read from memory but if they're not supposed to be changed then why not just replace every instance of 'magic' with a literal before the code compiles?
Function call overheads are not big; you won't normally notice them. You only see them in this case because your actual code (x*x) is itself so completely trivial. In any real program that does real work, the amount of time spent in function-calling overhead will be negligably small.
(Not that I'd really recommend using foo, identity and square in the example, in any case; they're so trivial it's just as readable to have them inline and they don't really encapsulate or abstract anything.)
if they're not supposed to be changed then why not just replace every instance of 'magic' with a literal before the code compiles?
Because programs are written for to be easy for you to read and maintain. You could replace constants with their literal values, but it'd make the program harder to work with, for a benefit that is so tiny you'll probably never even be able to measure it: the height of premature optimisation.
Encapsulation is about one thing and one thing only: Readability. If you're really so worried about performance that you're willing to start stripping out encapsulated logic, you may as well just start coding in assembly.
Encapsulation also assists in debugging and feature adding. Consider the following: lets say you have a simple game and need to add code that depletes the players health under some circumstances. Easy, yes?
def DamagePlayer(dmg):
player.health -= dmg;
This IS very trivial code, so it's very tempting to simply scatter "player.health -=" everywhere. But what if later you want to add a powerup that halves damage done to the player while active? If the logic is still encapsulated, it's easy:
def DamagePlayer(dmg):
if player.hasCoolPowerUp:
player.health -= dmg / 2
else
player.health -= dmg
Now, consider if you had neglected to encapulate that bit of logic because of it's simplicity. Now you're looking at coding the same logic into 50 different places, at least one of which you are almost certain to forget, which leads to weird bugs like: "When player has powerup all damage is halved except when hit by AlienSheep enemies..."
Do you want to have problems with Alien Sheep? I don't think so. :)
In all seriousness, the point I'm trying to make is that encapsulation is a very good thing in the right circumstances. Of course, it's also easy to over-encapsulate, which can be just as problematic. Also, there are situations where the speed really truly does matter (though they are rare), and that extra few clock cycles is worth it. About the only way to find the right balance is practice. Don't shun encapsulation because it's slower, though. The benefits usually far outweigh the costs.
I don't know how good python compilers are, but the answer to this question for many languages is that the compiler will optimize calls to small procedures / functions / methods by inlining them. In fact, in some language implementations you generally get better performance by NOT trying to "micro-optimize" the code yourself.
What you are talking about is the effect of inlining functions for gaining efficiency.
It is certainly true in your Python example, that encapsulation hurts performance. But there are some counter example to it:
In Java, defining getter&setter instead of defining public member variables does not result in performance degradation as the JIT inline the getters&setters.
sometimes calling a function repetedly may be better than performing inlining as the code executed may then fit in the cache. Inlining may cause code explosion...
Figuring out what to make into a function and what to just include inline is something of an art. Many factors (performance, readability, maintainability) feed into the equation.
I actually find your example kind of silly in many ways - a function that just returns it's argument? Unless it's an overload that's changing the rules, it's stupid. A function to square things? Again, why bother. Your function 'foo' should probably return a string, so that it can be used directly:
''.join(foo(x)," -- bar !! "])
That's probably a more correct level of encapsulation in this example.
As I say, it really depends on the circumstances. Unfortunately, this is the sort of thing that doesn't lend itself well to examples.
IMO, this is related to Function Call Costs. Which are negligible usually, but not zero. Splitting the code in a lot of very small functions may hurt. Especially in interpreted languages where full optimization is not available.
Inlining functions may improve performance but it may also deteriorate. See, for example, C++ FQA Lite for explanations (“Inlining can make code faster by eliminating function call overhead, or slower by generating too much code, causing instruction cache misses”). This is not C++ specific. Better leave optimizations for compiler/interpreter unless they are really necessary.
By the way, I don't see a huge difference between two versions:
$ python bench.py
fine-grained function decomposition: 5.46632194519
one-liner: 4.46827578545
$ python --version
Python 2.5.2
I think this result is acceptable. See bench.py in the pastebin.
There is a performance hit for using functions, since there is the overhead of jumping to a new address, pushing registers on the stack, and returning at the end. This overhead is very small, however, and even in peformance critial systems worrying about such overhead is most likely premature optimization.
Many languages avoid these issues in small frequenty called functions by using inlining, which is essentially what you do above.
Python does not do inlining. The closest thing you can do is use macros to replace the function calls.
This sort of performance issue is better served by another language, if you need the sort of speed gained by inlining (mostly marginal, and sometimes detremental) then you need to consider not using python for whatever you are working on.
There is a good technical reason why what you suggested is impossible. In Python, functions, constants, and everything else are accessible at runtime and can be changed at any time if necessary; they could also be changed by outside modules. This is an explicit promise of Python and one would need some extremely important reasons to break it.
For example, here's the common logging idiom:
# beginning of the file xxx.py
log = lambda *x: None
def something():
...
log(...)
...
(here log does nothing), and then at some other module or at the interactive prompt:
import xxx
xxx.log = print
xxx.something()
As you see, here log is modified by completely different module --- or by the user --- so that logging now works. That would be impossible if log was optimized away.
Similarly, if an exception was to happen in make_legal_foo_string (this is possible, e.g. if x.__str__() is broken and returns None) you'd be hit with a source quote from a wrong line and even perhaps from a wrong file in your scenario.
There are some tools that in fact apply some optimizations to Python code, but I don't think of the kind you suggested.
Related
Suppose you have a function like f that calls a function m.g:
def f(x):
return m.g(x, 2*x, x+1)
and f gets called a lot, so PyPy JITs it and inlines m.g into it. What if later, due to the "dynamic" nature of Python, m.g gets replaced by something else: Will the old JITed version of f be discarded right away, or could it still be called accidentally?
Also, what if your program does these redefinitions a lot, can the discarded JITed versions cause a memory leak?
To answer your last question: "if your program does these redefinitions a lot, can the discarded JITed versions cause a memory leak?" This is a good question, and the answer might be yes in some cases. The bad case might be if g is a function you just created with exec or eval, so that there is an unbounded number of function objects that end up being called here. This is a problem that we thought about fixing in the past, but never got around to do it. If you are experiencing what looks like a leak and have isolated this part of the code, then I'd say chances are that it is exactly what you are fearing. In that case, I'd recommend to write a mail to pypy-dev#python.org or come to #pypy on irc.freenode.net to describe your case.
You should have nothing to worry about. If what you're talking about, the wrong code being executed, were to happen, that would be a bug in the PyPy environment. That's very unlikely. Also, don't worry about a memory leak. Even if there was one, it wouldn't amount to enough memory that you'd ever notice or care. The only way you could possibly be affected by some sort of memory bug like that would be if you change that definition 1000s of times per execution of your code. I doubt that's the case.
“Premature optimization is the root of all evil”. Maybe you've heard of this famous quote. I think it also applies to dealing with problems that might occur in the future, but that are unlikely. Don't worry about any of this unless you see bad behavior. I very much doubt that you will. Trust in your tools!
I don't know if this will be useful to the community or not, as it might be unique to my situation. I'm working with a senior programmer who, in his code, has this peculiar habit of turning all strings into constants before using them. And I just don't get why. It doesn't make any sense to me. 99% of the time, we are gaining no abstraction or expressive power from the conversion, as it's done like this:
URL_CONVERTER = "url_converter"
URL_TYPE_LONG = "url_type_long"
URL_TYPE_SHORT = "url_type_short"
URL_TYPE_ARRAY = [URL_TYPE_LONG, URL_TYPE_SHORT]
for urltype in URL_TYPE_ARRAY:
outside_class.validate(urltype)
Just like that. As a rule, the constant name is almost always simply the string's content, capitalized, and these constants are seldom referenced more than once anyway. Perhaps less than 5% of the constants thus created are referenced twice or more during runtime.
Is this some programming technique that I just don't understand? Or is it just a bad habit? The other programmers are beginning to mimic this (possibly bad) form, and I want to know if there is a reason for me to as well before blindly following.
Thanks!
Edit: Updated the example. In addition, I understand everyone's points, and would add that this is a fairly small shop, at most two other people will ever see anyone's code, never mind work on it, and these are pretty simple one-offs we're building, not complicated workers or anything. I understand why this would be good practice in a large project, but in the context of our work, it comes across as too much overhead for a very simple task.
It helps to keep your code congruent. E.g. if you use URL_TYPE_LONG in both your client and your server and for some reason you need to change its string value, you just change one line. And you don't run the risk of forgetting to change one instance in the code, or to change one string in your code which just hazardly has the same value.
Even if those constants are only referenced once now, who are we to foresee the future...
I think this also arises from a time (when dinosaurs roamed the earth) when you tried to (A) keep data and code seperated and (B) you were concerned about how many strings you had allocated.
Below are a few scenarios where this would be a good practice:
You have a long string that will be used in a lot of places. Thus, you put it in a (presumably shorter) variable name so that you can use it easily. This keeps the lines of your code from becoming overly long/repetitive.
(somewhat similar to #1) You have a long sting that can't fit on a certain line without sending it way off the screen. So, you put the string in a variable to keep the lines concise.
You want to save a string and alter (add/remove characters from) it latter on. Only a variable will give you this functionality.
You want to have the ability to change multiple lines that use the same string by just altering one variable's value. This is a lot easier than having to go through numerous lines looking for occurrences of the string. It also keeps you from possibly missing some and thereby introducing bugs to your code.
Basically, the answer is to be smart about it. Ask yourself: will making the string into a variable save typing? Will it improve efficiency? Will it make your code easier to work with and/or maintain? If you answered "yes" to any of these, then you should do it.
Doing this is a really good idea.. I work on a fairly large python codebase with 100+ other engineers, and vouch for the fact that this makes collaboration much easier.
If you directly used the underlying strings everywhere, it would make it easier for you to make a typo when referencing it in one particular module and could lead to hard-to-catch bugs.
Its easier for modern IDE's to provide autocomplete and refactoring support when you are using a variable like this. You can easily change the underlying identifier to a different string or even a number later; This makes it easier for you to track down all modules referencing a particular identifier.
I have a Python unittest, with some tests having the same type object tested. The basic outline in one test-class is:
class TestClass(unittest.TestCase):
def setup(self):
...
def checkObjects(self, obj):
for i in [...values...]:
self.assertEqual(starttags(i,obj), endtags(i,obj))
def testOne(self):
#Get object one.
checkObjects(objone)
def testAnother(self):
#Access another object.
checkObjects(another)
... various tests for similar objects.
Although it's modular, I noticed that any failures will give an error like AssertionError: number != anothernumber, and the line of code generating the error, self.assertEqual(starttags(i,obj), endtags(i,obj)). If I had listed the tests out instead of placing in a for loop, I would have something like:
self.assertEqual(starttags(value1,obj), endtags(value1,obj))
self.assertEqual(starttags(value2,obj), endtags(value2,obj))
Which shows exactly which case causes the error, but is copy-paste code, which I thought was generally not recommended. I noticed the problem recently when a contributor reworked a more clean unit-test, that unfortunately would give little debug-info on assertion failures. So, what is the best-practice in these cases? Something like a list of tuples, fed into a for-loop with assertEquals is "cleaner", but copy-paste with different values on different lines gives useful stack traces.
If by cleaner you mean less code, then this kind of cleaner code is not always more readable code. In fact, it usually is less readable (especially when you come back to it). You can always go with fancy refactorings but you need to know when to stop. In longer run, it's always better to use more obvious, simple code than trying to squeeze one less line of code for artificial gains - not just in terms of unit testing.
Unit tests go with their own rules. For example they usually allow different naming conventions than what your regular code standards say, you hardly ever document it - they're kind of special part of your codebase. Also, having repeated code is not that uncommon. Actually, it's rather typical to have many similar looking, small tests.
Design your tests (code) with simplicity in mind
And at the moment, your tests are confusing even at the stage of writing them - imagine coming back to that code in 3 months from now. Imagine one of those tests breaking as a result of somebody else doing some change in some other place. It won't get any better.
Design your tests in such a way, that when one of them breaks, you immediately know why it did so and where. Not only that - design them in such a way, that you can tell what they are doing in a blink of an eye. Having for loops, ifs and basically any other kind of control flow mechanism (or too extensive refactoring) in test code, usually results in one question springing to your mind - What are we doing here? This is the kind of question you don't want to find yourself asking.
To summarize this longish post with words of people smarter than myself:
Any fool can write code that a computer can understand. Good programmers write code that humans can understand.
-Martin Fowler et al, Refactoring: Improving the Design of Existing Code, 1999
Do yourself a favor and stick to that rule.
Use nosetests, it makes your tests so much cleaner:
#!/usr/bin/python
def test_one():
for i in [...]:
assert xxx(i) == yyy
def test_two():
...
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Does creating separate functions instead of one big one slow processing time?
OOP Bloop boop!
Yes, OOP is great, keeps code clean and well organized. That's why I like to use it. I"m referring to to OOP very primitively specifically in using functions (defs).
Will taking out my function calls and sticking the content of the function straight into my algo increase speed of execution of the overall code?
Yes, I know I can run a test myself, but I'm choosing to ask it here in the forum of my fellow coder colleagues, because I know this is a question that floats around in many heads....
def myFunc(var):
return var*3243 #perhaps more complicated function code will make a difference?
i = 0
hungry = True
while hungry:
i = i + 1
x = myFunc(i)
if i > 50:
hungry = False
Write it correctly (i.e. with concerns properly separated into distinct functions and classes), then use PyPy to make it fast.
PyPy uses function inlining and assorted other tricks in its Just-In-Time compiler to speed up code execution without having to make it unmaintainable in the name of speed.
Note that this solution only works if you're using the Python 2.x series with any C extensions you use being compatible with cpyext.
Apparently there is very high function call overhead in Python.
http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Data_Aggregation
Is it worth keeping in mind while choosing the best way to write a piece of code?
Sure.
Is the speed up worth making your coder harder to understand?
Probably not.
Python won't "inline" the function for you, so of course there is overhead. Usually the code in the function will take enough time that the call overhead is not significant.
Bear in mind that it's much easier to test,debug and profile programs that are broken up into functions.
If you really really need more performance it's usually a better idea to write the function in C or Cython than to eliminate the call overhead
It's also worth noting that the usual code set up like this
def main():
...
if __name__=="main":
main()
is faster than just running code at the top level as identifier lookups are faster in functions/methods than in the global namespace
I've got a few questions about best practices in Python. Not too long ago I would do something like this with my code:
...
junk_block = "".join(open("foo.txt","rb").read().split())
...
I don't do this anymore because I can see that it makes code harder to read, but would the code run slower if I split the statements up like so:
f_obj = open("foo.txt", "rb")
f_data = f_obj.read()
f_data_list = f_data.split()
junk_block = "".join(f_data_list)
I also noticed that there's nothing keeping you from doing an 'import' within a function block, is there any reason why I should do that?
As long as you're inside a function (not at module top level), assigning intermediate results to local barenames has an essentially-negligible cost (at module top level, assigning to the "local" barenames implies churning on a dict -- the module's __dict__ -- and is measurably costlier than it would be within a function; the remedy is never to have "substantial" code at module top level... always stash substantial code within a function!-).
Python's general philosophy includes "flat is better than nested" -- and that includes highly "nested" expressions. Looking at your original example...:
junk_block = "".join(open("foo.txt","rb").read().split())
presents another important issues: when is that file getting closed? In CPython today, you need not worry -- reference counting in practice does ensure timely closure. But most other Python implementations (Jython on the JVM, IronPython on .NET, PyPy on all sorts of backends, pynie on Parrot, Unladen Swallow on LLVM if and when it matures per its published roadmap, ...) do not guarantee the use of reference counting -- many garbage collection strategies may be involved, with all sort of other advantages.
Without any guarantee of reference counting (and even in CPython it's always been deemed an implementation artifact, not part of the language semantics!), you might be exhausting resources, by executing such "open but no close" code in a tight loop -- garbage collection is triggered by scarcity of memory, and does not consider other limited resources such as file descriptors. Since 2.6 (and 2.5, with an "import from the future"), Python has a great solution via the RAII ("resource acquisition is initialization") approach supported by the with statement:
with open("foo.txt","rb") as f:
junk_block = "".join(f.read().split())
is the least-"unnested" way that will ensure timely closure of the file across all compliant versions of Python. The stronger semantics make it preferable.
Beyond ensuring the correct, and prudent;-), semantics, there's not that much to choose between nested and flattened versions of an expression such as this. Given the task "remove all runs of whitespace from the file's contents", I would be tempted to benchmark alternative approaches based on re and on the .translate method of strings (the latter, esp. in Python 2.*, is often the fastest way to delete all characters from a certain set!), before settling on the "split and rejoin" approach if it proves to be faster -- but that's really a rather different issue;-).
First of all, there's not really a reason you shouldn't use the first example - it'd quite readable in that it's concise about what it does. No reason to break it up since it's just a linear combination of calls.
Second, import within a function block is useful if there's a particular library function that you only need within that function - since the scope of an imported symbol is only the block within which it is imported, if you only ever use something once, you can just import it where you need it and not have to worry about name conflicts in other functions. This is especially handy with from X import Y statements, since Y won't be qualified by its containing module name and thus might conflict with a similarly named function in a different module being used elsewhere.
from PEP 8 (which is worth reading anyway)
Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants
That line has the same result as this:
junk_block = open("foo.txt","rb").read().replace(' ', '')
In your example you are splitting the words of the text into a list of words, and then you are joining them back together with no spaces. The above example instead uses the str.replace() method.
The differences:
Yours builds a file object into memory, builds a string into memory by reading it, builds a list into memory by splitting the string, builds a new string by joining the list.
Mine builds a file object into memory, builds a string into memory by reading it, builds a new string into memory by replacing spaces.
You can see a bit less RAM is used in the new variation but more processor is used. RAM is more valuable in some cases and so memory waste is frowned upon when it can be avoided.
Most of the memory will be garbage collected immediately but multiple users at the same time will hog RAM.
If you want to know if your second code fragment is slower, the quick way to find out would be to just use timeit. I wouldn't expect there to be that much difference though, since they seem pretty equivalent.
You should also ask if a performance difference actually matters in the code in question. Often readability is of more value than performance.
I can't think of any good reasons for importing a module in a function, but sometimes you just don't know you'll need to do something until you see the problem. I'll have to leave it to others to point out a constructive example of that, if it exists.
I think the two codes are readable. I (and that's just a question of personal style) will probably use the first, adding a coment line, something like: "Open the file and convert the data inside into a list"
Also, there are times when I use the second, maybe not so separated, but something like
f_data = open("foo.txt", "rb").read()
f_data_list = f_data.split()
junk_block = "".join(f_data_list)
But then I'm giving more entity to each operation, which could be important in the flow of the code. I think it's important you are confortable and don't think that the code is difficult to understand in the future.
Definitly, the code will not be (at least, much) slower, as the only "overload" you're making is to asing the results to values.