When I run
my_str = "res = f(content1, content2, reso)"
cProfile.runctx(my_str, globals(), locals())
I get:
3 function calls in 0.038 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.037 0.037 0.037 0.037 <string>:1(<module>)
1 0.000 0.000 0.038 0.038 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
Which is Nice, however, when I run:
with cProfile.Profile() as pr:
f(content1, content2, reso)
pr.print_stats()
I get something different (and all times are = to 0):
9 function calls in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 cProfile.py:40(print_stats)
1 0.000 0.000 0.000 0.000 cProfile.py:50(create_stats)
1 0.000 0.000 0.000 0.000 pstats.py:107(__init__)
1 0.000 0.000 0.000 0.000 pstats.py:117(init)
1 0.000 0.000 0.000 0.000 pstats.py:136(load_stats)
1 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
What's the difference between these 2? I would expect them printing the same result. Am I missing something?
Let's say we have an integer n = 123 and want to convert it into constituent digits d = [1, 2, 3]. Among two possible options,
def to_digits1(n):
return [int(c) for c in str(n)]
and
def to_digits2(n):
d = []
while n > 10:
n, m = divmod(n, 10)
d.insert(0, m)
d.insert(0, n)
return d
the former appears to be significantly faster on large inputs. For instance,
from cProfile import run
run('to_digits1(9999 ** 2048)')
run('to_digits2(9999 ** 2048)')
would yield something like:
5 function calls in 0.002 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.002 0.002 <string>:1(<module>)
1 0.001 0.001 0.002 0.002 to_digits.py:16(to_digits1)
1 0.001 0.001 0.001 0.001 to_digits.py:17(<listcomp>)
1 0.000 0.000 0.002 0.002 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
and
16387 function calls in 0.049 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.049 0.049 <string>:1(<module>)
1 0.002 0.002 0.049 0.049 to_digits.py:20(to_digits2)
8191 0.038 0.000 0.038 0.000 {built-in method builtins.divmod}
1 0.000 0.000 0.049 0.049 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
8192 0.009 0.000 0.009 0.000 {method 'insert' of 'list' objects}
respectively. Why would that be, considering that the first function converts n to a string and then converts each character back into integers, while the second funciton seems more concise and direct?
EDIT: As suggested in the comments, a list insert in the second example is expensive at O(n). This can be fixed by using a deque instead of a list, like so:
from collections import deque
def to_digits3(n):
d = deque()
while n > 10:
n, m = divmod(n, 10)
d.insert(0, m)
d.insert(0, n)
return list(d)
and while that minimizes the cost of the insert stage, it doesn't seem to give a substantial speed-up:
run('to_digits3(9999 ** 2048)')
16387 function calls in 0.038 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.038 0.038 <string>:1(<module>)
1 0.002 0.002 0.038 0.038 to_digits.py:30(to_digits3)
8191 0.035 0.000 0.035 0.000 {built-in method builtins.divmod}
1 0.000 0.000 0.038 0.038 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
8192 0.001 0.000 0.001 0.000 {method 'insert' of 'collections.deque' objects}
Can you help identify the bottleneck of this code? I am solving problem #7 of project Euler, and am failing to understand why this solution takes so long (30s). I know there are better solutions, I just want to understand more about why this, specifically is so bad.
def primes(n):
primes = set([2])
count = 1
i = 1
while count < n:
i += 2
if not any([i % num == 0 for num in primes]):
primes.add(i)
count += 1
print i
cProfile.run("primes(10001)") #slow! 30s
The profile is here below:
62374 function calls in 34.605 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 34.605 34.605 <string>:1(<module>)
1 34.273 34.273 34.605 34.605 problem_7.py:8(primes)
52371 0.328 0.000 0.328 0.000 {any}
10000 0.004 0.000 0.004 0.000 {method 'add' of 'set' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
I implemented a couple tweaks to help out. See my comments and results below. In all cases I changed the accumulator from being named primes (shadowing its function name) to being named ps. I didn't test to see if this improved performance, I just hate shadowing names :o)
# Your original code
def prime_orig(n):
ps = set([2])
count = 1
i = 1
while count < n:
i += 2
if not any([i % num == 0 for num in ps]):
ps.add(i)
count += 1
# replace the set accum with a list, per #Sheshnath's comment
def prime_list(n):
ps = [2]
count = 1
i = 1
while count < n:
i += 2
if not any([i % num == 0 for num in ps]):
ps.append(i)
count += 1
# replace the listcomp with a genexp
def prime_genexp(n):
ps = set([2])
count = 1
i = 1
while count < n:
i += 2
if not any(i % num == 0 for num in ps):
ps.add(i)
count += 1
# both optimizations at once
def prime_genexp_list(n):
ps = [2]
count = 1
i = 1
while count < n:
i += 2
if not any(i % num == 0 for num in ps):
ps.append(i)
count += 1
RESULTS:
cProfile.run('prime_orig(10001)')
114746 function calls in 27.283 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.701 0.701 27.283 27.283 <ipython-input-2-3059f1e23ab4>:1(prime_orig)
52371 26.330 0.001 26.330 0.001 <ipython-input-2-3059f1e23ab4>:7(<listcomp>)
1 0.000 0.000 27.283 27.283 <string>:1(<module>)
52371 0.250 0.000 0.250 0.000 {built-in method builtins.any}
1 0.000 0.000 27.283 27.283 {built-in method builtins.exec}
10000 0.003 0.000 0.003 0.000 {method 'add' of 'set' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
cProfile.run('prime_list(10001)')
114746 function calls in 24.523 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.666 0.666 24.523 24.523 <ipython-input-2-3059f1e23ab4>:11(prime_list)
52371 23.625 0.000 23.625 0.000 <ipython-input-2-3059f1e23ab4>:17(<listcomp>)
1 0.000 0.000 24.523 24.523 <string>:1(<module>)
52371 0.231 0.000 0.231 0.000 {built-in method builtins.any}
1 0.000 0.000 24.523 24.523 {built-in method builtins.exec}
10000 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
cProfile.run('prime_genexp(10001)')
50627376 function calls in 10.577 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.040 0.040 10.577 10.577 <ipython-input-2-3059f1e23ab4>:21(prime_genexp)
50565001 7.060 0.000 7.060 0.000 <ipython-input-2-3059f1e23ab4>:27(<genexpr>)
1 0.000 0.000 10.577 10.577 <string>:1(<module>)
52371 3.475 0.000 10.530 0.000 {built-in method builtins.any}
1 0.000 0.000 10.577 10.577 {built-in method builtins.exec}
10000 0.002 0.000 0.002 0.000 {method 'add' of 'set' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
cProfile.run('prime_genexp_list(10001)')
50400891 function calls in 9.781 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.040 0.040 9.781 9.781 <ipython-input-2-3059f1e23ab4>:31(prime_genexp_list)
50338516 6.272 0.000 6.272 0.000 <ipython-input-2-3059f1e23ab4>:37(<genexpr>)
1 0.000 0.000 9.781 9.781 <string>:1(<module>)
52371 3.468 0.000 9.735 0.000 {built-in method builtins.any}
1 0.000 0.000 9.781 9.781 {built-in method builtins.exec}
10000 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
Profiling Django app to figure out slow functions.
I just added some middleware to track function calls, following this blog: http://agiliq.com/blog/2015/07/profiling-django-middlewares/ and I see that the entry of cProfile stats for {posix.write} is one of the longest.
Any idea what that is, and where that comes from?
Other functions are referenced by their name and package path, so I'm not sure what {posix.write} means.
the log looks like this:
204051 function calls (197141 primitive calls) in 0.997 seconds
Ordered by: internal time
List reduced from 1204 to 50 due to restriction <50>
ncalls tottime percall cumtime percall filename:lineno(function)
35 0.305 0.009 0.305 0.009 {posix.write}
95 0.206 0.002 0.207 0.002 {method 'execute' of 'psycopg2.extensions.cursor' objects}
73 0.088 0.001 0.088 0.001 {select.select}
898 0.023 0.000 0.047 0.000 /.venv/lib/python2.7/site-packages/django/db/models/base.py:388(__init__)
1642 0.012 0.000 0.371 0.000 /.venv/lib/python2.7/site-packages/django/template/base.py:806(_resolve_lookup)
1 0.010 0.010 0.011 0.011 {_sass.compile_filename}
1 0.009 0.009 0.009 0.009 {psycopg2._psycopg._connect}
34 0.009 0.000 0.009 0.000 {method 'recv' of '_socket.socket' objects}
39 0.007 0.000 0.007 0.000 {posix.read}
9641/6353 0.006 0.000 0.321 0.000 {getattr}
173 0.006 0.000 0.026 0.000 /.venv/lib/python2.7/site-packages/django/core/urlresolvers.py:425(_reverse_with_prefix)
25769 0.006 0.000 0.007 0.000 {isinstance}
EDIT:
I understand that posix.write is the write function of posix. That I need to understand I guess is what part of Django uses that a lot and why it is showing up as taking 300+ms.
How would I go about tracking this down?
Thanks
I want to imitate a normal python list, except whenever elements are added or removed via slicing, I want to 'save' the list. Is this possible? This was my attempt but it will never print 'saving'.
class InterceptedList(list):
def addSave(func):
def newfunc(self, *args):
func(self, *args)
print 'saving'
return newfunc
__setslice__ = addSave(list.__setslice__)
__delslice__ = addSave(list.__delslice__)
>>> l = InterceptedList()
>>> l.extend([1,2,3,4])
>>> l
[1, 2, 3, 4]
>>> l[3:] = [5] # note: 'saving' is not printed
>>> l
[1, 2, 3, 5]
This does work for other methods like append and extend, just not for the slice operations.
EDIT: The real problem is I'm using Jython and not Python and forgot it. The comments on the question are correct. This code does work fine in Python (2.6). However, the code nor the answers work in Jython.
From the Python 3 docs:
__getslice__(), __setslice__() and __delslice__() were killed.
The syntax a[i:j] now translates to a.__getitem__(slice(i, j))
(or __setitem__() or __delitem__(), when used as an assignment
or deletion target, respectively).
That's enough speculation. Let's start using facts instead shall we?
As far as I can tell, the bottom line is that you must override both set of methods.
If you want to implement undo/redo you probably should try using undo stack and set of actions that can do()/undo() themselves.
Code
import profile
import sys
print sys.version
class InterceptedList(list):
def addSave(func):
def newfunc(self, *args):
func(self, *args)
print 'saving'
return newfunc
__setslice__ = addSave(list.__setslice__)
__delslice__ = addSave(list.__delslice__)
class InterceptedList2(list):
def __setitem__(self, key, value):
print 'saving'
list.__setitem__(self, key, value)
def __delitem__(self, key):
print 'saving'
list.__delitem__(self, key)
print("------------Testing setslice------------------")
l = InterceptedList()
l.extend([1,2,3,4])
profile.run("l[3:] = [5]")
profile.run("l[2:6] = [12, 4]")
profile.run("l[-1:] = [42]")
profile.run("l[::2] = [6,6]")
print("-----------Testing setitem--------------------")
l2 = InterceptedList2()
l2.extend([1,2,3,4])
profile.run("l2[3:] = [5]")
profile.run("l2[2:6] = [12,4]")
profile.run("l2[-1:] = [42]")
profile.run("l2[::2] = [6,6]")
Jython 2.5
C:\Users\wuu-local.pyza\Desktop>c:\jython2.5.0\jython.bat intercept.py
2.5.0 (Release_2_5_0:6476, Jun 16 2009, 13:33:26)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)]
------------Testing setslice------------------
saving
3 function calls in 0.035 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:0(<module>)
1 0.000 0.000 0.000 0.000 intercept.py:9(newfunc)
1 0.034 0.034 0.035 0.035 profile:0(l[3:] = [5])
0 0.000 0.000 profile:0(profiler)
saving
3 function calls in 0.005 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.001 0.001 <string>:0(<module>)
1 0.001 0.001 0.001 0.001 intercept.py:9(newfunc)
1 0.004 0.004 0.005 0.005 profile:0(l[2:6] = [12, 4])
0 0.000 0.000 profile:0(profiler)
saving
3 function calls in 0.012 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:0(<module>)
1 0.000 0.000 0.000 0.000 intercept.py:9(newfunc)
1 0.012 0.012 0.012 0.012 profile:0(l[-1:] = [42])
0 0.000 0.000 profile:0(profiler)
2 function calls in 0.004 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:0(<module>)
1 0.004 0.004 0.004 0.004 profile:0(l[::2] = [6,6])
0 0.000 0.000 profile:0(profiler)
-----------Testing setitem--------------------
2 function calls in 0.004 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:0(<module>)
1 0.004 0.004 0.004 0.004 profile:0(l2[3:] = [5])
0 0.000 0.000 profile:0(profiler)
2 function calls in 0.006 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:0(<module>)
1 0.006 0.006 0.006 0.006 profile:0(l2[2:6] = [12,4])
0 0.000 0.000 profile:0(profiler)
2 function calls in 0.004 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:0(<module>)
1 0.004 0.004 0.004 0.004 profile:0(l2[-1:] = [42])
0 0.000 0.000 profile:0(profiler)
saving
3 function calls in 0.007 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.002 0.002 <string>:0(<module>)
1 0.001 0.001 0.001 0.001 intercept.py:20(__setitem__)
1 0.005 0.005 0.007 0.007 profile:0(l2[::2] = [6,6])
0 0.000 0.000 profile:0(profiler)
Python 2.6.2
C:\Users\wuu-local.pyza\Desktop>python intercept.py
2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit (Intel)]
------------Testing setslice------------------
saving
4 function calls in 0.002 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.002 0.002 0.002 0.002 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 intercept.py:9(newfunc)
1 0.000 0.000 0.002 0.002 profile:0(l[3:] = [5])
0 0.000 0.000 profile:0(profiler)
saving
4 function calls in 0.000 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 intercept.py:9(newfunc)
1 0.000 0.000 0.000 0.000 profile:0(l[2:6] = [12, 4])
0 0.000 0.000 profile:0(profiler)
saving
4 function calls in 0.000 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 intercept.py:9(newfunc)
1 0.000 0.000 0.000 0.000 profile:0(l[-1:] = [42])
0 0.000 0.000 profile:0(profiler)
3 function calls in 0.000 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 profile:0(l[::2] = [6,6])
0 0.000 0.000 profile:0(profiler)
-----------Testing setitem--------------------
3 function calls in 0.000 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 profile:0(l2[3:] = [5])
0 0.000 0.000 profile:0(profiler)
3 function calls in 0.000 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 profile:0(l2[2:6] = [12,4])
0 0.000 0.000 profile:0(profiler)
3 function calls in 0.000 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 profile:0(l2[-1:] = [42])
0 0.000 0.000 profile:0(profiler)
saving
4 function calls in 0.003 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.003 0.003 <string>:1(<module>)
1 0.002 0.002 0.002 0.002 intercept.py:20(__setitem__)
1 0.000 0.000 0.003 0.003 profile:0(l2[::2] = [6,6])
0 0.000 0.000 profile:0(profiler)
"setslice" and "delslice" are deprecated, if you want to do the interception you need to work with python slice objects passed to "setitem" and "delitem". If you want to intecept both slices and ordinary accesses this code works perfectly in python 2.6.2.
class InterceptedList(list):
def addSave(func):
def newfunc(self, *args):
func(self, *args)
print 'saving'
return newfunc
def __setitem__(self, key, value):
print 'saving'
list.__setitem__(self, key, value)
def __delitem__(self, key):
print 'saving'
list.__delitem__(self, key)
the circumstances where __getslice__ and __setslice__ are called are pretty narrow. Specifically, slicing only occurs when you use a regular slice, where the first and end elements are mentioned exactly once. for any other slice syntax, or no slices at all, __getitem__ or __setitem__ is called.