I need to display a complex Table and decided to use stacked QTableWidgets. With an increasing number of rows the program needs a lot of time for creating all the widgets and almost the same time for displaying.
The maintable looks like this: MainTable
The stacked TableWidget in the table:
StackedTables
if the cell contains data, there is at least one TableWidget in one cell of the MainTable and in the worst case there are 2 more TableWidgets in that one. That means I could have 3 TableWidgets in one cell.
Time measurement with cProfile and time.time for 80rows (with 48 of the complex cells for each row):
complete update time: 15s (manually stopped)
time to create the table: 7.548534870147705s (time.time over complete function)
display time: 7.5s (complete update time - function time)
rows: 80
63600 function calls in 7.462 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1896 2.455 0.001 3.882 0.002 DigitalePlanungstafel.py:6054(grundWidgetErstellen) -- (create table in cell)
3936 2.027 0.001 2.027 0.001 {built-in method setCellWidget}
2535 1.306 0.001 1.306 0.001 {built-in method setColumnCount}
630 0.770 0.001 1.183 0.002 DigitalePlanungstafel.py:6035(obenWidgetErstellen) -- (create table in table in cell)
2607 0.674 0.000 0.674 0.000 {built-in method setRowCount}
2528 0.059 0.000 0.059 0.000 {built-in method horizontalHeader}
2526 0.021 0.000 0.021 0.000 {built-in method verticalHeader}
163 0.019 0.000 0.019 0.000 {method 'execute' of 'sqlite3.Cursor' objects}
2526 0.016 0.000 0.016 0.000 {built-in method setFrameShape}
1410 0.014 0.000 0.014 0.000 {built-in method setStyleSheet}
4502 0.013 0.000 0.013 0.000 {built-in method setRowHeight}
2526 0.009 0.000 0.009 0.000 {built-in method setFixedSize}
2546 0.009 0.000 0.009 0.000 {built-in method setColumnWidth}
5052 0.009 0.000 0.009 0.000 {built-in method setVisible}
1329 0.007 0.000 0.007 0.000 {built-in method setItem}
2181 0.006 0.000 0.006 0.000 {built-in method cellWidget}
80 0.005 0.000 0.005 0.000 {built-in method addWidget}
2526 0.004 0.000 0.004 0.000 {built-in method setEditTriggers}
929 0.004 0.000 0.004 0.000 {built-in method setBackground}
1330 0.003 0.000 0.003 0.000 {method 'format' of 'str' objects}
414 0.003 0.000 0.003 0.000 {built-in method _pickle.loads}
336 0.003 0.000 0.003 0.000 {method 'strftime' of 'datetime.date' objects}
2526 0.003 0.000 0.003 0.000 {built-in method setHorizontalScrollBarPolicy}
1410 0.002 0.000 0.002 0.000 {built-in method setFixedHeight}
1377 0.002 0.000 0.002 0.000 {built-in method setTextAlignment}
83 0.002 0.000 0.004 0.000 _strptime.py:321(_strptime)
2 0.002 0.001 0.002 0.001 {built-in method setSortingEnabled}
2526 0.001 0.000 0.001 0.000 {built-in method setShowGrid}
1570 0.001 0.000 0.001 0.000 {built-in method rowHeight}
2526 0.001 0.000 0.001 0.000 {built-in method setSelectionMode}
163 0.001 0.000 0.001 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
240 0.001 0.000 0.001 0.000 DigitalePlanungstafel.py:7494(__init__)
2526 0.001 0.000 0.001 0.000 {built-in method setVerticalScrollBarPolicy}
1 0.001 0.001 0.001 0.001 {built-in method sortByColumn}
80 0.001 0.000 0.001 0.000 {built-in method setLayout}
83 0.001 0.000 0.001 0.000 {built-in method _locale.setlocale}
1 0.001 0.001 0.001 0.001 {built-in method _sqlite3.connect}
89 0.000 0.000 0.000 0.000 {built-in method setForeground}
83 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
83 0.000 0.000 0.004 0.000 _strptime.py:562(_strptime_datetime)
48 0.000 0.000 0.000 0.000 {built-in method setHorizontalHeaderItem}
80 0.000 0.000 0.000 0.000 {built-in method setContentsMargins}
83 0.000 0.000 0.005 0.000 {built-in method strptime}
83 0.000 0.000 0.000 0.000 locale.py:379(normalize)
88 0.000 0.000 0.000 0.000 {built-in method setFont}
160 0.000 0.000 0.000 0.000 {built-in method setData}
83 0.000 0.000 0.000 0.000 {method 'groupdict' of '_sre.SRE_Match' objects}
80 0.000 0.000 0.000 0.000 {built-in method setAlignment}
83 0.000 0.000 0.001 0.000 _strptime.py:29(_getlang)
83 0.000 0.000 0.001 0.000 locale.py:565(getlocale)
83 0.000 0.000 0.001 0.000 locale.py:462(_parse_localename)
80 0.000 0.000 0.000 0.000 {built-in method setUnderline}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
249 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of 'sqlite3.Connection' objects}
160 0.000 0.000 0.000 0.000 {built-in method item}
742 0.000 0.000 0.000 0.000 DigitalePlanungstafel.py:7499(__lt__)
48 0.000 0.000 0.000 0.000 {built-in method today}
165 0.000 0.000 0.000 0.000 {method 'toordinal' of 'datetime.date' objects}
475 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
167 0.000 0.000 0.000 0.000 {built-in method builtins.len}
166 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
83 0.000 0.000 0.000 0.000 {method 'end' of '_sre.SRE_Match' objects}
84 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.TextIOWrapper' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.print}
83 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
47 0.000 0.000 0.000 0.000 {built-in method columnCount}
83 0.000 0.000 0.000 0.000 {method 'weekday' of 'datetime.date' objects}
96 0.000 0.000 0.000 0.000 {method 'date' of 'datetime.datetime' objects}
20 0.000 0.000 0.000 0.000 {built-in method columnWidth}
1 0.000 0.000 0.000 0.000 {built-in method _locale._getdefaultlocale}
3 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
1 0.000 0.000 0.000 0.000 _strptime.py:284(_calc_julian_from_U_or_W)
1 0.000 0.000 0.000 0.000 _bootlocale.py:11(getpreferredencoding)
1 0.000 0.000 0.000 0.000 {built-in method sortIndicatorOrder}
2 0.000 0.000 0.000 0.000 {built-in method time.time}
1 0.000 0.000 0.000 0.000 {method 'cursor' of 'sqlite3.Connection' objects}
2 0.000 0.000 0.000 0.000 {method 'index' of 'list' objects}
1 0.000 0.000 0.000 0.000 {built-in method fromordinal}
1 0.000 0.000 0.000 0.000 {built-in method sortIndicatorSection}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 codecs.py:259(__init__)
The time for calling the function and creating the Table is okay but the program needs almost the same time again until it reacts.
The Goal would be to reduce the update time by approx. 50%. I need to display around 200 rows.
Is the stacked QTableWidget the right approach - If so what do I have to do to optimize the update times?
I already thought about changing the presenatation from QTableWidget to a QGraphicsView and simply drawing the rectangles.
Or maybe a combnation, for example: Using the QTableWidget for the Header and the first columns and then merging all the complex cells and inserting a QGraphicsView but I am not sure if I am able to get the right size for the drawed cells and I do not know if the displaying time will be shorter.
What do you guys think is the right approch for a table like this?
If you need I can append the function, which is updating the Table.
EDIT:
I am using now 2 rows in the MainTable for one Block which reduced the number of stacked widgets from 2.526 to 7!
The time measurement for the same rows now looks like this:
complete update time: ~2s (manually stopped)
time to create the table: 0.572490930557251 (time.time over complete function)
display time: ~1.5s (complete update time - function time)
rows: 160
20912 function calls in 0.534 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1417 0.258 0.000 0.258 0.000 {built-in method setCellWidget}
88 0.209 0.002 0.209 0.002 {built-in method setRowCount}
163 0.017 0.000 0.017 0.000 {method 'execute' of 'sqlite3.Cursor' objects}
1410 0.010 0.000 0.010 0.000 {built-in method setStyleSheet}
80 0.005 0.000 0.005 0.000 {built-in method addWidget}
1336 0.004 0.000 0.004 0.000 {built-in method setItem}
7 0.003 0.000 0.004 0.001 DigitalePlanungstafel.py:6037(obenWidgetErstellen)
501 0.003 0.000 0.003 0.000 {built-in method _pickle.loads}
1990 0.003 0.000 0.003 0.000 {built-in method cellWidget}
336 0.002 0.000 0.002 0.000 {method 'strftime' of 'datetime.date' objects}
83 0.002 0.000 0.004 0.000 _strptime.py:321(_strptime)
1330 0.002 0.000 0.002 0.000 {method 'format' of 'str' objects}
1410 0.002 0.000 0.002 0.000 {built-in method setFixedHeight}
929 0.001 0.000 0.001 0.000 {built-in method setBackground}
1377 0.001 0.000 0.001 0.000 {built-in method setTextAlignment}
240 0.001 0.000 0.001 0.000 DigitalePlanungstafel.py:7452(__init__)
163 0.001 0.000 0.001 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
16 0.001 0.000 0.001 0.000 {built-in method setColumnCount}
770 0.001 0.000 0.001 0.000 {built-in method setSpan}
2127 0.001 0.000 0.001 0.000 {built-in method item}
1570 0.001 0.000 0.001 0.000 {built-in method rowHeight}
80 0.001 0.000 0.001 0.000 {built-in method setLayout}
83 0.000 0.000 0.000 0.000 {built-in method _locale.setlocale}
1 0.000 0.000 0.000 0.000 {built-in method _sqlite3.connect}
167 0.000 0.000 0.000 0.000 {built-in method setRowHeight}
89 0.000 0.000 0.000 0.000 {built-in method setForeground}
83 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
48 0.000 0.000 0.000 0.000 {built-in method setHorizontalHeaderItem}
83 0.000 0.000 0.004 0.000 _strptime.py:562(_strptime_datetime)
7 0.000 0.000 0.000 0.000 {built-in method takeItem}
80 0.000 0.000 0.000 0.000 {built-in method setContentsMargins}
83 0.000 0.000 0.004 0.000 {built-in method strptime}
88 0.000 0.000 0.000 0.000 {built-in method setFont}
160 0.000 0.000 0.000 0.000 {built-in method setData}
83 0.000 0.000 0.000 0.000 locale.py:379(normalize)
80 0.000 0.000 0.000 0.000 {built-in method setAlignment}
83 0.000 0.000 0.001 0.000 _strptime.py:29(_getlang)
83 0.000 0.000 0.000 0.000 {method 'groupdict' of '_sre.SRE_Match' objects}
83 0.000 0.000 0.001 0.000 locale.py:565(getlocale)
1 0.000 0.000 0.000 0.000 {method 'close' of 'sqlite3.Connection' objects}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
7 0.000 0.000 0.000 0.000 {built-in method horizontalHeader}
80 0.000 0.000 0.000 0.000 {built-in method setUnderline}
249 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
83 0.000 0.000 0.000 0.000 locale.py:462(_parse_localename)
48 0.000 0.000 0.000 0.000 {built-in method today}
7 0.000 0.000 0.000 0.000 {built-in method verticalHeader}
475 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
167 0.000 0.000 0.000 0.000 {built-in method builtins.len}
165 0.000 0.000 0.000 0.000 {method 'toordinal' of 'datetime.date' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.TextIOWrapper' objects}
166 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
7 0.000 0.000 0.000 0.000 {built-in method setFrameShape}
27 0.000 0.000 0.000 0.000 {built-in method setColumnWidth}
84 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'sort' of 'list' objects}
83 0.000 0.000 0.000 0.000 {method 'end' of '_sre.SRE_Match' objects}
87 0.000 0.000 0.000 0.000 DigitalePlanungstafel.py:6059(<lambda>)
83 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.print}
7 0.000 0.000 0.000 0.000 {built-in method setFixedSize}
14 0.000 0.000 0.000 0.000 {built-in method setVisible}
83 0.000 0.000 0.000 0.000 {method 'weekday' of 'datetime.date' objects}
96 0.000 0.000 0.000 0.000 {method 'date' of 'datetime.datetime' objects}
7 0.000 0.000 0.000 0.000 {built-in method setEditTriggers}
20 0.000 0.000 0.000 0.000 {built-in method columnWidth}
7 0.000 0.000 0.000 0.000 {built-in method setHorizontalScrollBarPolicy}
1 0.000 0.000 0.000 0.000 {built-in method _locale._getdefaultlocale}
The updateing time is more than good but I have 2 problems because of the splitting.
Sorting (over the header) is not working anymore. This is caused by the merging, it messes up everything. Is there a way to freeze two associated rows before sorting?
I only want one row to be selected at a time. Cause of the splitting I need to select the two associated rows no matter which one gets selected. Not really a big deal but doesen´t really look good (see pictures).
MainTable selected row
MainTable selected row
You do not want to do any of this - instead:
Use a model for your data.
Use a viewmodel that adapts the model to the view you wish to have.
Use a QTableView to display the viewmodel.
Related
I was solving leetcode 1155 which is about number of dice rolls with target sum. I was using dictionary-based memorization. Here's the exact code:
class Solution:
def numRollsToTarget(self, dices: int, faces: int, target: int) -> int:
dp = {}
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
if dp.get((t,rd)): return dp[(t,rd)]
dp[(t,rd)] = sum(ways(t-i, rd-1) for i in range(1,faces+1))
return dp[(t,rd)]
return ways(target, dices)
But this solution is invariably timing out for a combination of face and dices around 15*15
Then I found this solution which uses functools.lru_cache and the rest of it is exactly the same. This solution works very fast.
class Solution:
def numRollsToTarget(self, dices: int, faces: int, target: int) -> int:
from functools import lru_cache
#lru_cache(None)
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
return sum(ways(t-i, rd-1) for i in range(1,faces+1))
return ways(target, dices)
Earlier, I have compared and found that in most cases, lru_cache does not outperform dictionary-based cache by such a margin.
Can someone explain the reason why there is such a drastic performance difference between the two approaches?
First, running your OP code with cProfile and this is the report:
with print(numRollsToTarget2(4, 6, 20)) (OP version)
You can spot right away there're some heavy calls in ways genexpr and sum. That's prob. need close examinations and try to improve/reduce. Next posting is for similar memo version, but the calls is much less. And that version has passed w/o timeout.
35
2864 function calls (366 primitive calls) in 0.018 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.018 0.018 <string>:1(<module>)
1 0.000 0.000 0.001 0.001 dice_rolls.py:23(numRollsToTarget2)
1075/1 0.001 0.000 0.001 0.001 dice_rolls.py:25(ways)
1253/7 0.001 0.000 0.001 0.000 dice_rolls.py:30(<genexpr>)
1 0.000 0.000 0.018 0.018 dice_rolls.py:36(main)
21 0.000 0.000 0.000 0.000 rpc.py:153(debug)
3 0.000 0.000 0.017 0.006 rpc.py:216(remotecall)
3 0.000 0.000 0.000 0.000 rpc.py:226(asynccall)
3 0.000 0.000 0.016 0.005 rpc.py:246(asyncreturn)
3 0.000 0.000 0.000 0.000 rpc.py:252(decoderesponse)
3 0.000 0.000 0.016 0.005 rpc.py:290(getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:298(_proxify)
3 0.000 0.000 0.016 0.005 rpc.py:306(_getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:328(newseq)
3 0.000 0.000 0.000 0.000 rpc.py:332(putmessage)
2 0.000 0.000 0.001 0.000 rpc.py:559(__getattr__)
3 0.000 0.000 0.000 0.000 rpc.py:57(dumps)
1 0.000 0.000 0.001 0.001 rpc.py:577(__getmethods)
2 0.000 0.000 0.000 0.000 rpc.py:601(__init__)
2 0.000 0.000 0.016 0.008 rpc.py:606(__call__)
4 0.000 0.000 0.000 0.000 run.py:412(encoding)
4 0.000 0.000 0.000 0.000 run.py:416(errors)
2 0.000 0.000 0.017 0.008 run.py:433(write)
6 0.000 0.000 0.000 0.000 threading.py:1306(current_thread)
3 0.000 0.000 0.000 0.000 threading.py:222(__init__)
3 0.000 0.000 0.016 0.005 threading.py:270(wait)
3 0.000 0.000 0.000 0.000 threading.py:81(RLock)
3 0.000 0.000 0.000 0.000 {built-in method _struct.pack}
3 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
6 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
1 0.000 0.000 0.018 0.018 {built-in method builtins.exec}
6 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
9 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1 0.000 0.000 0.017 0.017 {built-in method builtins.print}
179/1 0.000 0.000 0.001 0.001 {built-in method builtins.sum}
3 0.000 0.000 0.000 0.000 {built-in method select.select}
3 0.000 0.000 0.000 0.000 {method '_acquire_restore' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_is_owned' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_release_save' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
6 0.016 0.003 0.016 0.003 {method 'acquire' of '_thread.lock' objects}
3 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects}
2 0.000 0.000 0.000 0.000 {method 'decode' of 'bytes' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
3 0.000 0.000 0.000 0.000 {method 'dump' of '_pickle.Pickler' objects}
2 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
201 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {method 'getvalue' of '_io.BytesIO' objects}
3 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'send' of '_socket.socket' objects}
Then I tried to run modified/simplified version, and compare the results.
35
387 function calls (193 primitive calls) in 0.006 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.006 0.006 <string>:1(<module>)
1 0.000 0.000 0.006 0.006 dice_rolls.py:36(main)
1 0.000 0.000 0.000 0.000 dice_rolls.py:5(numRollsToTarget)
195/1 0.000 0.000 0.000 0.000 dice_rolls.py:8(dp)
21 0.000 0.000 0.000 0.000 rpc.py:153(debug)
3 0.000 0.000 0.006 0.002 rpc.py:216(remotecall)
3 0.000 0.000 0.000 0.000 rpc.py:226(asynccall)
3 0.000 0.000 0.006 0.002 rpc.py:246(asyncreturn)
3 0.000 0.000 0.000 0.000 rpc.py:252(decoderesponse)
3 0.000 0.000 0.006 0.002 rpc.py:290(getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:298(_proxify)
3 0.000 0.000 0.006 0.002 rpc.py:306(_getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:328(newseq)
3 0.000 0.000 0.000 0.000 rpc.py:332(putmessage)
2 0.000 0.000 0.001 0.000 rpc.py:559(__getattr__)
3 0.000 0.000 0.000 0.000 rpc.py:57(dumps)
1 0.000 0.000 0.001 0.001 rpc.py:577(__getmethods)
2 0.000 0.000 0.000 0.000 rpc.py:601(__init__)
2 0.000 0.000 0.005 0.003 rpc.py:606(__call__)
4 0.000 0.000 0.000 0.000 run.py:412(encoding)
4 0.000 0.000 0.000 0.000 run.py:416(errors)
2 0.000 0.000 0.006 0.003 run.py:433(write)
6 0.000 0.000 0.000 0.000 threading.py:1306(current_thread)
3 0.000 0.000 0.000 0.000 threading.py:222(__init__)
3 0.000 0.000 0.006 0.002 threading.py:270(wait)
3 0.000 0.000 0.000 0.000 threading.py:81(RLock)
3 0.000 0.000 0.000 0.000 {built-in method _struct.pack}
3 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
6 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
1 0.000 0.000 0.006 0.006 {built-in method builtins.exec}
6 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
9 0.000 0.000 0.000 0.000 {built-in method builtins.len}
34 0.000 0.000 0.000 0.000 {built-in method builtins.max}
1 0.000 0.000 0.006 0.006 {built-in method builtins.print}
3 0.000 0.000 0.000 0.000 {built-in method select.select}
3 0.000 0.000 0.000 0.000 {method '_acquire_restore' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_is_owned' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_release_save' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
6 0.006 0.001 0.006 0.001 {method 'acquire' of '_thread.lock' objects}
3 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects}
2 0.000 0.000 0.000 0.000 {method 'decode' of 'bytes' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
3 0.000 0.000 0.000 0.000 {method 'dump' of '_pickle.Pickler' objects}
2 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {method 'getvalue' of '_io.BytesIO' objects}
3 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'send' of '_socket.socket' objects}
The profiling codes are here:
import cProfile
from typing import List
def numRollsToTarget(d, f, target):
memo = {}
def dp(d, target):
if d == 0:
return 0 if target > 0 else 1
if (d, target) in memo:
return memo[(d, target)]
result = 0
for k in range(max(0, target-f), target):
result += dp(d-1, k)
memo[(d, target)] = result
return result
return dp(d, target) % (10**9 + 7)
def numRollsToTarget2(dices: int, faces: int, target: int) -> int:
dp = {}
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
if dp.get((t,rd)): return dp[(t,rd)]
dp[(t,rd)] = sum(ways(t-i, rd-1) for i in range(1,faces+1))
return dp[(t,rd)]
return ways(target, dices)
def numRollsToTarget3(dices: int, faces: int, target: int) -> int:
from functools import lru_cache
#lru_cache(None)
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
return sum(ways(t-i, rd-1) for i in range(1,faces+1))
return ways(target, dices)
def main():
print(numRollsToTarget(4, 6, 20))
#print(numRollsToTarget2(4, 6, 20))
#print(numRollsToTarget3(4, 6, 20)) # not faster than first
if __name__ == '__main__':
cProfile.run('main()')
I have a single-threaded Python 3 program that's CPU bound, the only IO is printing a couple of lines to output (no reading/writing of files).
On my desktop machine (AMD Ryzen 1700x 3.8 GHz, 16GB 3000 MHz DDR4) it performs (consistently) at 3400 episodes/second where a run takes around 60 seconds.
On my laptop (Intel i7-6600U 2.8 GHz, 16GB 2000 MHz DDR3) the performance is doubled at 7000 episodes/second, and a run coming in at just under 30 seconds.
Both machines run the same operating systems (Fedora 26) and the same python version (not built from source).
What's more, when profiling, there's a line showing
10.999 tottime, 28.814 cumtime for arrayprint.py:557(fillFormat)
but only when the code is run on the desktop. On the laptop, the particular function does not appear at all (and none of the arrayprint functions use more than 1 second tottime).
Not only is it strange that the performance differs between the machines, but no arrays or lists are ever printed to the screen, converted to strings, or saved to files during the execution of the program.
Here's the full profile for the desktop:
54499635 function calls (53787999 primitive calls) in 58.746 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
533727 0.359 0.000 0.514 0.000 <frozen importlib._bootstrap>:402(parent)
533727 0.469 0.000 0.697 0.000 <frozen importlib._bootstrap>:989(_handle_fromlist)
1 0.000 0.000 58.746 58.746 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 __init__.py:120(getLevelName)
567524 0.237 0.000 0.727 0.000 __init__.py:1284(debug)
2 0.000 0.000 0.000 0.000 __init__.py:1296(info)
2 0.000 0.000 0.000 0.000 __init__.py:1308(warning)
2 0.000 0.000 0.000 0.000 __init__.py:1320(warn)
4 0.000 0.000 0.000 0.000 __init__.py:1374(findCaller)
4 0.000 0.000 0.000 0.000 __init__.py:1404(makeRecord)
4 0.000 0.000 0.000 0.000 __init__.py:1419(_log)
4 0.000 0.000 0.000 0.000 __init__.py:1444(handle)
4 0.000 0.000 0.000 0.000 __init__.py:1498(callHandlers)
567528 0.175 0.000 0.175 0.000 __init__.py:1528(getEffectiveLevel)
567528 0.315 0.000 0.490 0.000 __init__.py:1542(isEnabledFor)
4 0.000 0.000 0.000 0.000 __init__.py:157(<lambda>)
4 0.000 0.000 0.000 0.000 __init__.py:251(__init__)
4 0.000 0.000 0.000 0.000 __init__.py:329(getMessage)
4 0.000 0.000 0.000 0.000 __init__.py:387(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:390(format)
4 0.000 0.000 0.000 0.000 __init__.py:540(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:546(formatMessage)
4 0.000 0.000 0.000 0.000 __init__.py:562(format)
8 0.000 0.000 0.000 0.000 __init__.py:703(filter)
8 0.000 0.000 0.000 0.000 __init__.py:807(acquire)
8 0.000 0.000 0.000 0.000 __init__.py:814(release)
4 0.000 0.000 0.000 0.000 __init__.py:827(format)
4 0.000 0.000 0.000 0.000 __init__.py:850(handle)
4 0.000 0.000 0.000 0.000 __init__.py:969(flush)
4 0.000 0.000 0.000 0.000 __init__.py:980(emit)
289159 0.101 0.000 1.491 0.000 _methods.py:31(_sum)
533727 0.175 0.000 1.613 0.000 _methods.py:37(_any)
177909 0.862 0.000 33.737 0.000 arrayprint.py:237(_get_formatdict)
177909 0.370 0.000 34.214 0.000 arrayprint.py:273(_get_format_function)
177909 0.686 0.000 39.971 0.000 arrayprint.py:315(_array2string)
533727/177909 0.674 0.000 40.351 0.000 arrayprint.py:340(array2string)
1224652 0.960 0.000 1.554 0.000 arrayprint.py:467(_extendLine)
177909 1.671 0.000 4.320 0.000 arrayprint.py:475(_formatArray)
533727 0.682 0.000 29.496 0.000 arrayprint.py:543(__init__)
533727 10.999 0.000 28.814 0.000 arrayprint.py:557(fillFormat)
355336 1.600 0.000 5.432 0.000 arrayprint.py:589(<listcomp>)
2416068 2.677 0.000 3.832 0.000 arrayprint.py:642(_digits)
177909 0.720 0.000 2.378 0.000 arrayprint.py:652(__init__)
1224652 1.057 0.000 1.057 0.000 arrayprint.py:665(__call__)
533727 0.147 0.000 0.147 0.000 arrayprint.py:674(__init__)
177909 0.227 0.000 0.319 0.000 arrayprint.py:702(__init__)
177909 0.415 0.000 17.986 0.000 arrayprint.py:713(__init__)
177909 0.166 0.000 0.166 0.000 arrayprint.py:730(__init__)
177909 0.046 0.000 0.046 0.000 arrayprint.py:751(__init__)
1 0.000 0.000 0.000 0.000 enum.py:265(__call__)
1 0.000 0.000 0.000 0.000 enum.py:515(__new__)
1 0.000 0.000 0.000 0.000 enum.py:544(_missing_)
177909 0.206 0.000 0.206 0.000 enum.py:552(__str__)
177909 0.269 0.000 0.475 0.000 enum.py:564(__format__)
755408 0.248 0.000 0.366 0.000 enum.py:579(__hash__)
200000 0.037 0.000 0.037 0.000 enum.py:592(name)
27976 0.005 0.000 0.005 0.000 enum.py:597(value)
200524 0.443 0.000 0.641 0.000 eventgen.py:115(_push)
200001 0.492 0.000 0.885 0.000 eventgen.py:122(pop)
200000 0.892 0.000 1.036 0.000 eventgen.py:137(ce_str)
13988 0.017 0.000 0.034 0.000 eventgen.py:15(__lt__)
99676 0.168 0.000 0.911 0.000 eventgen.py:44(event_new)
79335 0.096 0.000 0.520 0.000 eventgen.py:52(event_end)
11689 0.078 0.000 0.261 0.000 eventgen.py:61(event_new_handoff)
9824 0.014 0.000 0.098 0.000 eventgen.py:90(event_end_handoff)
77441 0.295 0.000 0.380 0.000 eventgen.py:94(reassign)
177909 0.177 0.000 0.555 0.000 fromnumeric.py:1364(ravel)
200001 0.093 0.000 0.464 0.000 fromnumeric.py:1471(nonzero)
289159 0.542 0.000 2.148 0.000 fromnumeric.py:1710(sum)
533727 0.637 0.000 2.956 0.000 fromnumeric.py:1866(any)
200001 0.120 0.000 0.372 0.000 fromnumeric.py:55(_wrapfunc)
4 0.000 0.000 0.000 0.000 genericpath.py:117(_splitext)
49 0.000 0.000 0.001 0.000 grid.py:172(neighbors1)
49 0.001 0.000 0.001 0.000 grid.py:195(neighbors2)
533727/177909 0.311 0.000 40.472 0.000 numeric.py:1927(array_str)
1067454 1.724 0.000 4.091 0.000 numeric.py:2692(seterr)
1067454 1.466 0.000 1.603 0.000 numeric.py:2792(geterr)
533727 0.299 0.000 0.422 0.000 numeric.py:3085(__init__)
533727 0.411 0.000 2.588 0.000 numeric.py:3089(__enter__)
533727 0.461 0.000 2.374 0.000 numeric.py:3094(__exit__)
177909 0.064 0.000 0.151 0.000 numeric.py:463(asarray)
711636 0.223 0.000 0.503 0.000 numeric.py:534(asanyarray)
4 0.000 0.000 0.000 0.000 posixpath.py:119(splitext)
4 0.000 0.000 0.000 0.000 posixpath.py:142(basename)
4 0.000 0.000 0.000 0.000 posixpath.py:39(_get_sep)
6 0.000 0.000 0.000 0.000 posixpath.py:50(normcase)
4 0.000 0.000 0.000 0.000 process.py:137(name)
4 0.000 0.000 0.000 0.000 process.py:35(current_process)
1 0.000 0.000 0.000 0.000 signal.py:25(_int_to_enum)
2 0.000 0.000 0.000 0.000 signal.py:35(_enum_to_int)
1 0.000 0.000 0.000 0.000 signal.py:45(signal)
99627 0.062 0.000 0.062 0.000 stats.py:38(new)
20292 0.028 0.000 0.039 0.000 stats.py:42(new_rej)
88750 0.047 0.000 0.047 0.000 stats.py:48(end)
11623 0.005 0.000 0.005 0.000 stats.py:51(hoff_new)
1799 0.001 0.000 0.002 0.000 stats.py:54(hoff_rej)
22091 0.012 0.000 0.012 0.000 stats.py:58(rej)
200000 0.234 0.000 1.513 0.000 stats.py:64(iter)
1 0.000 0.000 0.000 0.000 stats.py:69(n_iter)
1 0.000 0.000 0.000 0.000 stats.py:86(endsim)
1 0.000 0.000 0.001 0.001 strats.py:189(get_init_action)
200000 1.070 0.000 49.964 0.000 strats.py:193(get_action)
177909 1.348 0.000 1.937 0.000 strats.py:220(execute_action)
200001 4.572 0.000 47.626 0.000 strats.py:243(optimal_ch)
89158 0.071 0.000 0.958 0.000 strats.py:299(reward)
89158 0.018 0.000 0.018 0.000 strats.py:308(discount)
1242355 0.944 0.000 0.944 0.000 strats.py:333(get_qval)
89158 0.160 0.000 0.160 0.000 strats.py:336(update_qval)
1 0.000 0.000 58.746 58.746 strats.py:40(init_sim)
1 1.271 1.271 58.745 58.745 strats.py:49(_simulate)
4 0.000 0.000 0.000 0.000 threading.py:1076(name)
4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread)
227976 0.120 0.000 0.162 0.000 types.py:135(__get__)
177909 0.079 0.000 0.079 0.000 {built-in method _functools.reduce}
200001 0.192 0.000 0.222 0.000 {built-in method _heapq.heappop}
200524 0.084 0.000 0.088 0.000 {built-in method _heapq.heappush}
310143 0.064 0.000 0.064 0.000 {built-in method _operator.gt}
843054 0.152 0.000 0.152 0.000 {built-in method _operator.lt}
1 0.000 0.000 0.000 0.000 {built-in method _signal.signal}
8 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
2 0.000 0.000 0.000 0.000 {built-in method _warnings.warn}
1 0.000 0.000 58.746 58.746 {built-in method builtins.exec}
200001 0.056 0.000 0.056 0.000 {built-in method builtins.getattr}
1067468 0.228 0.000 0.228 0.000 {built-in method builtins.hasattr}
755408 0.118 0.000 0.118 0.000 {built-in method builtins.hash}
467082 0.164 0.000 0.164 0.000 {built-in method builtins.isinstance}
533727 0.107 0.000 0.107 0.000 {built-in method builtins.issubclass}
10361766 1.076 0.000 1.076 0.000 {built-in method builtins.len}
533441 0.304 0.000 0.304 0.000 {built-in method builtins.max}
533923 0.198 0.000 0.198 0.000 {built-in method builtins.min}
889545 0.368 0.000 0.368 0.000 {built-in method numpy.core.multiarray.array}
111251 0.101 0.000 0.101 0.000 {built-in method numpy.core.multiarray.where}
2134908 0.377 0.000 0.377 0.000 {built-in method numpy.core.umath.geterrobj}
1067454 0.524 0.000 0.524 0.000 {built-in method numpy.core.umath.seterrobj}
14 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getpid}
4 0.000 0.000 0.000 0.000 {built-in method sys._getframe}
5 0.000 0.000 0.000 0.000 {built-in method time.time}
8 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
533727 0.299 0.000 1.912 0.000 {method 'any' of 'numpy.ndarray' objects}
875 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
16544 0.119 0.000 0.119 0.000 {method 'choice' of 'mtrand.RandomState' objects}
533727 0.872 0.000 0.872 0.000 {method 'compress' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
188835 0.641 0.000 0.641 0.000 {method 'exponential' of 'mtrand.RandomState' objects}
4 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'flush' of '_io.TextIOWrapper' objects}
8 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
355818 0.069 0.000 0.069 0.000 {method 'item' of 'numpy.ndarray' objects}
200001 0.196 0.000 0.196 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
533727 0.123 0.000 0.123 0.000 {method 'pop' of 'dict' objects}
11689 0.053 0.000 0.053 0.000 {method 'randint' of 'mtrand.RandomState' objects}
168494 0.128 0.000 0.128 0.000 {method 'random_sample' of 'mtrand.RandomState' objects}
177909 0.232 0.000 0.232 0.000 {method 'ravel' of 'numpy.ndarray' objects}
1889376 5.023 0.000 5.023 0.000 {method 'reduce' of 'numpy.ufunc' objects}
8 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
533727 0.155 0.000 0.155 0.000 {method 'rpartition' of 'str' objects}
4865786 1.100 0.000 1.100 0.000 {method 'rstrip' of 'str' objects}
8 0.000 0.000 0.000 0.000 {method 'write' of '_io.TextIOWrapper' objects}
And here's for the laptop:
27738517 function calls (26673571 primitive calls) in 28.612 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 28.612 28.612 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 __init__.py:120(getLevelName)
566894 0.244 0.000 0.720 0.000 __init__.py:1284(debug)
2 0.000 0.000 0.000 0.000 __init__.py:1296(info)
2 0.000 0.000 0.000 0.000 __init__.py:1308(warning)
2 0.000 0.000 0.000 0.000 __init__.py:1320(warn)
4 0.000 0.000 0.000 0.000 __init__.py:1374(findCaller)
4 0.000 0.000 0.000 0.000 __init__.py:1404(makeRecord)
4 0.000 0.000 0.000 0.000 __init__.py:1419(_log)
4 0.000 0.000 0.000 0.000 __init__.py:1444(handle)
4 0.000 0.000 0.000 0.000 __init__.py:1498(callHandlers)
566898 0.166 0.000 0.166 0.000 __init__.py:1528(getEffectiveLevel)
566898 0.309 0.000 0.476 0.000 __init__.py:1542(isEnabledFor)
4 0.000 0.000 0.000 0.000 __init__.py:157(<lambda>)
4 0.000 0.000 0.000 0.000 __init__.py:251(__init__)
4 0.000 0.000 0.000 0.000 __init__.py:329(getMessage)
4 0.000 0.000 0.000 0.000 __init__.py:387(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:390(format)
4 0.000 0.000 0.000 0.000 __init__.py:540(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:546(formatMessage)
4 0.000 0.000 0.000 0.000 __init__.py:562(format)
8 0.000 0.000 0.000 0.000 __init__.py:703(filter)
8 0.000 0.000 0.000 0.000 __init__.py:807(acquire)
8 0.000 0.000 0.000 0.000 __init__.py:814(release)
4 0.000 0.000 0.000 0.000 __init__.py:827(format)
4 0.000 0.000 0.000 0.000 __init__.py:850(handle)
4 0.000 0.000 0.000 0.000 __init__.py:969(flush)
4 0.000 0.000 0.000 0.000 __init__.py:980(emit)
288946 0.112 0.000 1.643 0.000 _methods.py:31(_sum)
177491 0.330 0.000 0.330 0.000 arrayprint.py:256(_get_formatdict)
177491 0.169 0.000 3.542 0.000 arrayprint.py:259(<lambda>)
177491 0.465 0.000 4.419 0.000 arrayprint.py:299(_get_format_function)
177491 0.623 0.000 9.729 0.000 arrayprint.py:343(_array2string)
532473/177491 0.987 0.000 10.679 0.000 arrayprint.py:381(wrapper)
532473/177491 0.721 0.000 10.150 0.000 arrayprint.py:399(array2string)
1225350 0.971 0.000 1.470 0.000 arrayprint.py:527(_extendLine)
177491 1.458 0.000 3.920 0.000 arrayprint.py:535(_formatArray)
177491 0.768 0.000 3.373 0.000 arrayprint.py:712(__init__)
1225350 0.960 0.000 0.960 0.000 arrayprint.py:725(__call__)
1 0.000 0.000 0.000 0.000 enum.py:265(__call__)
1 0.000 0.000 0.000 0.000 enum.py:515(__new__)
1 0.000 0.000 0.000 0.000 enum.py:544(_missing_)
177491 0.209 0.000 0.209 0.000 enum.py:552(__str__)
177491 0.316 0.000 0.525 0.000 enum.py:564(__format__)
755255 0.238 0.000 0.352 0.000 enum.py:579(__hash__)
200000 0.039 0.000 0.039 0.000 enum.py:592(name)
28626 0.005 0.000 0.005 0.000 enum.py:597(value)
200505 0.443 0.000 0.643 0.000 eventgen.py:115(_push)
200001 0.474 0.000 0.863 0.000 eventgen.py:122(pop)
200000 0.834 0.000 0.983 0.000 eventgen.py:137(ce_str)
14313 0.017 0.000 0.035 0.000 eventgen.py:15(__lt__)
99673 0.186 0.000 0.939 0.000 eventgen.py:44(event_new)
78949 0.094 0.000 0.500 0.000 eventgen.py:52(event_end)
11887 0.078 0.000 0.261 0.000 eventgen.py:61(event_new_handoff)
9996 0.017 0.000 0.103 0.000 eventgen.py:90(event_end_handoff)
77374 0.284 0.000 0.364 0.000 eventgen.py:94(reassign)
177491 0.195 0.000 0.595 0.000 fromnumeric.py:1380(ravel)
200001 0.098 0.000 0.490 0.000 fromnumeric.py:1487(nonzero)
288946 0.590 0.000 2.352 0.000 fromnumeric.py:1730(sum)
200001 0.130 0.000 0.392 0.000 fromnumeric.py:55(_wrapfunc)
4 0.000 0.000 0.000 0.000 genericpath.py:117(_splitext)
49 0.000 0.000 0.001 0.000 grid.py:172(neighbors1)
49 0.001 0.000 0.001 0.000 grid.py:195(neighbors2)
532473/177491 0.365 0.000 10.826 0.000 numeric.py:1905(array_str)
177491 0.062 0.000 0.151 0.000 numeric.py:463(asarray)
177491 0.051 0.000 0.104 0.000 numeric.py:534(asanyarray)
4 0.000 0.000 0.000 0.000 posixpath.py:119(splitext)
4 0.000 0.000 0.000 0.000 posixpath.py:142(basename)
4 0.000 0.000 0.000 0.000 posixpath.py:39(_get_sep)
6 0.000 0.000 0.000 0.000 posixpath.py:50(normcase)
4 0.000 0.000 0.000 0.000 process.py:137(name)
4 0.000 0.000 0.000 0.000 process.py:35(current_process)
1 0.000 0.000 0.000 0.000 signal.py:25(_int_to_enum)
2 0.000 0.000 0.000 0.000 signal.py:35(_enum_to_int)
1 0.000 0.000 0.000 0.000 signal.py:45(signal)
99624 0.066 0.000 0.066 0.000 stats.py:38(new)
20675 0.028 0.000 0.040 0.000 stats.py:42(new_rej)
88545 0.045 0.000 0.045 0.000 stats.py:48(end)
11831 0.006 0.000 0.006 0.000 stats.py:51(hoff_new)
1835 0.001 0.000 0.002 0.000 stats.py:54(hoff_rej)
22510 0.013 0.000 0.013 0.000 stats.py:58(rej)
200000 0.261 0.000 1.490 0.000 stats.py:64(iter)
1 0.000 0.000 0.000 0.000 stats.py:69(n_iter)
1 0.000 0.000 0.000 0.000 stats.py:86(endsim)
1 0.000 0.000 0.000 0.000 strats.py:189(get_init_action)
200000 1.234 0.000 19.760 0.000 strats.py:193(get_action)
177490 1.294 0.000 1.860 0.000 strats.py:220(execute_action)
200001 3.897 0.000 17.128 0.000 strats.py:243(optimal_ch)
88945 0.074 0.000 1.112 0.000 strats.py:299(reward)
88945 0.017 0.000 0.017 0.000 strats.py:308(discount)
1241938 0.681 0.000 0.681 0.000 strats.py:333(get_qval)
88945 0.167 0.000 0.167 0.000 strats.py:336(update_qval)
1 0.000 0.000 28.612 28.612 strats.py:40(init_sim)
1 1.383 1.383 28.611 28.611 strats.py:49(_simulate)
4 0.000 0.000 0.000 0.000 threading.py:1076(name)
4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread)
228626 0.122 0.000 0.166 0.000 types.py:135(__get__)
177491 0.075 0.000 0.075 0.000 {built-in method _functools.reduce}
200001 0.203 0.000 0.234 0.000 {built-in method _heapq.heappop}
200505 0.079 0.000 0.083 0.000 {built-in method _heapq.heappush}
320262 0.068 0.000 0.068 0.000 {built-in method _operator.gt}
832731 0.136 0.000 0.136 0.000 {built-in method _operator.lt}
1 0.000 0.000 0.000 0.000 {built-in method _signal.signal}
532481 0.090 0.000 0.090 0.000 {built-in method _thread.get_ident}
2 0.000 0.000 0.000 0.000 {built-in method _warnings.warn}
1 0.000 0.000 28.612 28.612 {built-in method builtins.exec}
200001 0.066 0.000 0.066 0.000 {built-in method builtins.getattr}
14 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
755255 0.113 0.000 0.113 0.000 {built-in method builtins.hash}
532473 0.092 0.000 0.092 0.000 {built-in method builtins.id}
466451 0.166 0.000 0.166 0.000 {built-in method builtins.isinstance}
532473 0.083 0.000 0.083 0.000 {built-in method builtins.issubclass}
3750044 0.325 0.000 0.325 0.000 {built-in method builtins.len}
177687 0.091 0.000 0.091 0.000 {built-in method builtins.max}
196 0.000 0.000 0.000 0.000 {built-in method builtins.min}
354982 0.142 0.000 0.142 0.000 {built-in method numpy.core.multiarray.array}
111456 0.095 0.000 0.095 0.000 {built-in method numpy.core.multiarray.where}
14 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getpid}
4 0.000 0.000 0.000 0.000 {built-in method sys._getframe}
5 0.000 0.000 0.000 0.000 {built-in method time.time}
8 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
532473 0.089 0.000 0.089 0.000 {method 'add' of 'set' objects}
875 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
16345 0.110 0.000 0.110 0.000 {method 'choice' of 'mtrand.RandomState' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
532473 0.097 0.000 0.097 0.000 {method 'discard' of 'set' objects}
188618 0.633 0.000 0.633 0.000 {method 'exponential' of 'mtrand.RandomState' objects}
4 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'flush' of '_io.TextIOWrapper' objects}
8 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
354982 0.066 0.000 0.066 0.000 {method 'item' of 'numpy.ndarray' objects}
200001 0.196 0.000 0.196 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
11887 0.052 0.000 0.052 0.000 {method 'randint' of 'mtrand.RandomState' objects}
167895 0.157 0.000 0.157 0.000 {method 'random_sample' of 'mtrand.RandomState' objects}
177491 0.251 0.000 0.251 0.000 {method 'ravel' of 'numpy.ndarray' objects}
643928 2.511 0.000 2.511 0.000 {method 'reduce' of 'numpy.ufunc' objects}
8 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
2451118 0.328 0.000 0.328 0.000 {method 'rstrip' of 'str' objects}
8 0.000 0.000 0.000 0.000 {method 'write' of '_io.TextIOWrapper' objects}
numpy was installed through pip on the laptop, and through the Fedora repositories on the desktop. Removing the package and installing it through pip removed arrayprint (fillFormat) from the profiling results and the runtime is now very much the same (which is still a bit weird). It's also strange that the other arrayprint functions are still being called, with 10 seconds of cumulative time.
I am storing an index in a compressed zip on disk and wanted to extract a single file from this zip. Doing this in python seems to be very slow, is it possible to solve this.
with zipfile.ZipFile("testoutput/index_doc.zip", mode='r') as myzip:
with myzip.open("c0ibtxf_i.txt") as mytxt:
txt = mytxt.read()
txt = codecs.decode(txt, "utf-8")
print(txt)
Is the python code I use. Running this script in python takes a noticably long time
python3 testunzip.py 1.22s user 0.06s system 98% cpu 1.303 total
Which is annoying, especially since I know it can go much faster:
unzip -p testoutput/index_doc.zip c0ibtxf_i.txt 0.01s user 0.00s system 69% cpu 0.023 total
as per request: profiling
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.051 0.051 1.492 1.492 <string>:1(<module>)
127740 0.043 0.000 0.092 0.000 cp437.py:14(decode)
1 0.000 0.000 1.441 1.441 testunzip.py:69(toprofile)
1 0.000 0.000 0.000 0.000 threading.py:72(RLock)
1 0.000 0.000 0.000 0.000 utf_8.py:15(decode)
1 0.000 0.000 0.000 0.000 zipfile.py:1065(__enter__)
1 0.000 0.000 0.000 0.000 zipfile.py:1068(__exit__)
1 0.692 0.692 1.441 1.441 zipfile.py:1085(_RealGetContents)
1 0.000 0.000 0.000 0.000 zipfile.py:1194(getinfo)
1 0.000 0.000 0.000 0.000 zipfile.py:1235(open)
1 0.000 0.000 0.000 0.000 zipfile.py:1591(__del__)
2 0.000 0.000 0.000 0.000 zipfile.py:1595(close)
2 0.000 0.000 0.000 0.000 zipfile.py:1713(_fpclose)
1 0.000 0.000 0.000 0.000 zipfile.py:191(_EndRecData64)
1 0.000 0.000 0.000 0.000 zipfile.py:234(_EndRecData)
127739 0.180 0.000 0.220 0.000 zipfile.py:320(__init__)
127739 0.046 0.000 0.056 0.000 zipfile.py:436(_decodeExtra)
1 0.000 0.000 0.000 0.000 zipfile.py:605(_check_compression)
1 0.000 0.000 0.000 0.000 zipfile.py:636(_get_decompressor)
1 0.000 0.000 0.000 0.000 zipfile.py:654(__init__)
3 0.000 0.000 0.000 0.000 zipfile.py:660(read)
1 0.000 0.000 0.000 0.000 zipfile.py:667(close)
1 0.000 0.000 0.000 0.000 zipfile.py:708(__init__)
1 0.000 0.000 0.000 0.000 zipfile.py:821(read)
1 0.000 0.000 0.000 0.000 zipfile.py:854(_update_crc)
1 0.000 0.000 0.000 0.000 zipfile.py:901(_read1)
1 0.000 0.000 0.000 0.000 zipfile.py:937(_read2)
1 0.000 0.000 0.000 0.000 zipfile.py:953(close)
1 0.000 0.000 1.441 1.441 zipfile.py:981(__init__)
127740 0.049 0.000 0.049 0.000 {built-in method _codecs.charmap_decode}
1 0.000 0.000 0.000 0.000 {built-in method _codecs.decode}
1 0.000 0.000 0.000 0.000 {built-in method _codecs.utf_8_decode}
127743 0.058 0.000 0.058 0.000 {built-in method _struct.unpack}
127739 0.016 0.000 0.016 0.000 {built-in method builtins.chr}
1 0.000 0.000 1.492 1.492 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
2 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
255484 0.020 0.000 0.020 0.000 {built-in method builtins.len}
1 0.000 0.000 0.000 0.000 {built-in method builtins.max}
1 0.000 0.000 0.000 0.000 {built-in method builtins.min}
1 0.000 0.000 0.000 0.000 {built-in method builtins.print}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
2 0.000 0.000 0.000 0.000 {built-in method zlib.crc32}
1 0.000 0.000 0.000 0.000 {function ZipExtFile.close at 0x101975620}
127741 0.011 0.000 0.011 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.BufferedReader' objects}
127740 0.224 0.000 0.317 0.000 {method 'decode' of 'bytes' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
127739 0.024 0.000 0.024 0.000 {method 'find' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
7 0.006 0.001 0.006 0.001 {method 'read' of '_io.BufferedReader' objects}
510956 0.071 0.000 0.071 0.000 {method 'read' of '_io.BytesIO' objects}
8 0.000 0.000 0.000 0.000 {method 'seek' of '_io.BufferedReader' objects}
4 0.000 0.000 0.000 0.000 {method 'tell' of '_io.BufferedReader' objects}
it seems to be something that happens in the constructor? Can I avoid this overhead somehow?
I figured out what the problem was:
pythons zipfile library builds a list of information object for each file in the zip
this causes zipfile to be quite fast once it's loaded.
but when there are a lot of files in the zip and you only need a small portion of this files each time you load the zip, the overhead of creating the info-list costs a lot of time.
To solve this, I adapted the source of python's zipfile. It has all the default functionalities you need, but when you give the constructor a list of the filenames to extract, it will not build the entire information list.
In the particular use case that you only need a few files from a zip, this will make a big difference in performance and memory usage.
for the particular case in the example above (namely extracting only one file from a zip containing 128K files, the speed of the new implementation now approaches the speed of the unzip method)
A test case:
def original_zipfile():
import zipfile
with zipfile.ZipFile("testoutput/index_doc.zip", mode='r') as myzip:
with myzip.open("c6kn5pu_i.txt") as mytxt:
txt = mytxt.read()
def my_zipfile():
import zipfile2
with zipfile2.ZipFile("testoutput/index_doc.zip", to_extract=["c6kn5pu_i.txt"], mode='r') as myzip:
with myzip.open("c6kn5pu_i.txt") as mytxt:
txt = mytxt.read()
if __name__ == "__main__":
import time
time1 = time.time()
original_zipfile()
print("running time of original_zipfile = "+str(time.time()-time1))
time1 = time.time()
my_zipfile()
print("running time of my_new_zipfile = "+str(time.time()-time1))
print(myStopwatch.getPretty())
resulted in the following time readings
running time of original_zipfile = 1.0871901512145996
running time of my_new_zipfile = 0.07036209106445312
I will include the source code, but notice that there are 2 small flaws to my implementation (once you give an extract list, when you don't the behaviour will be the same as mentioned before):
it assumes all filenames to be encoded in the same encoding (which is an optimisation I included for my own purposes)
other functionality might be altered (for example extract_all might fail or only extract the files you gave to the the constructor)
github link
I'm implementing a RANSAC algorithm for circle detection in images. I profiled the execution and I get:
13699392 function calls in 799.981 seconds
Random listing order was used
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 {time.time}
579810 0.564 0.000 0.564 0.000 {getattr}
289905 2.343 0.000 8.661 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/blas.py:226(_get_funcs)
579810 0.124 0.000 0.124 0.000 {method 'get' of 'dict' objects}
289905 0.645 0.000 2.676 0.000 {map}
2954 0.005 0.000 0.005 0.000 {method 'transpose' of 'numpy.ndarray' objects}
2954 0.023 0.000 0.464 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/shape_base.py:179(vstack)
2954 2.373 0.001 2.373 0.001 {method 'read' of 'cv2.VideoCapture' objects}
579810 0.966 0.000 2.031 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/function_base.py:550(asarray_chkfinite)
289905 10.164 0.000 24.316 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/basic.py:456(lstsq)
2954 1.090 0.000 1.090 0.000 {normalize}
1455433 3.827 0.000 3.827 0.000 {numpy.core.multiarray.array}
579810 2.899 0.000 3.148 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numerictypes.py:949(_can_coerce_all)
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.empty}
2954 32.544 0.011 795.875 0.269 git/tra-python-processer/tra/ransac.py:31(image_search)
289905 0.714 0.000 38.644 0.000 git/tra-python-processer/tra/features.py:44(__init__)
289905 2.157 0.000 2.157 0.000 {method 'randint' of 'mtrand.RandomState' objects}
1 0.005 0.005 0.005 0.005 {VideoCapture}
289905 1.026 0.000 1.026 0.000 {method 'astype' of 'numpy.generic' objects}
2954 0.006 0.000 0.010 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/fromnumeric.py:495(transpose)
289905 11.303 0.000 37.930 0.000 git/tra-python-processer/tra/features.py:48(__gen)
3496584 0.343 0.000 0.343 0.000 {len}
2954 0.344 0.000 0.344 0.000 {numpy.core.multiarray.concatenate}
2954 3.214 0.001 3.214 0.001 {numpy.core.multiarray.where}
869715 0.575 0.000 0.575 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2514(size)
869715 0.778 0.000 2.278 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py:394(asarray)
289905 716.946 0.002 716.946 0.002 git/tra-python-processer/tra/features.py:89(points_distance)
5908 0.015 0.000 0.031 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py:464(asanyarray)
289905 0.275 0.000 0.275 0.000 {isinstance}
289905 0.342 0.000 9.003 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/lapack.py:255(get_lapack_funcs)
5908 0.058 0.000 0.097 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/shape_base.py:60(atleast_2d)
295813 0.089 0.000 0.089 0.000 {method 'append' of 'list' objects}
289905 0.645 0.000 3.793 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numerictypes.py:970(find_common_type)
2954 0.221 0.000 0.221 0.000 {threshold}
1 0.000 0.000 0.000 0.000 {method 'get' of 'cv2.VideoCapture' objects}
1 0.000 0.000 0.000 0.000 git/tra-python-processer/tra/ransac.py:24(__init__)
2954 0.009 0.000 0.009 0.000 {numpy.core.multiarray.zeros}
579810 0.143 0.000 0.143 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/misc.py:126(_datacopied)
1 0.201 0.201 799.981 799.981 git/tra-python-processer/tra/ransac.py:122(video_processing)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
2954 1.528 0.001 1.528 0.001 {cvtColor}
289905 1.280 0.000 5.346 0.000 /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/blas.py:182(find_best_blas_type)
289905 0.198 0.000 0.198 0.000 {method 'index' of 'list' objects}
It's first time I use profiler, however for what I can understand the function that is most heavy is features.py:89(points_distance) that comes out to be a very easy implementation:
def points_distance(self,points):
d = n.abs(\
n.sqrt(\
n.power(self.xc - points[:,0],2) + n.power(self.yc - points[:,1],2)
)\
- self.radius
)
return d
Any suggestions? Maybe cython?
Use scipy.spatial.distance.cdist for the distance calculation in points_distance.
First, optimize your code in pure Python and numpy. Then if necessary port the critical parts to Cython. Since a number of functions are called repeatedly a few ~100000 times, you should get some speed up from Cython for those parts. Unless, of course, the computational bottleneck is in the distance calculation, which will then limit the overall execution time.
By the way, you should sort your profiler results by tottime so they are easier to read.
I am profiling a python code ; why does it spend more time in the user space ?
user#terminal$ time python main.py
1964 function calls in 0.003 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.003 0.003 :1()
1 0.000 0.000 0.000 0.000 ConfigParser.py:218(init)
1 0.000 0.000 0.001 0.001 ConfigParser.py:266(read)
30 0.000 0.000 0.000 0.000 ConfigParser.py:354(optionxform)
1 0.000 0.000 0.000 0.000 ConfigParser.py:434(_read)
15 0.000 0.000 0.000 0.000 ConfigParser.py:515(get)
15 0.000 0.000 0.000 0.000 ConfigParser.py:611(_interpolate)
15 0.000 0.000 0.000 0.000 ConfigParser.py:619(_interpolate_some)
1 0.000 0.000 0.000 0.000 config.py:32(read_config_data)
1 0.000 0.000 0.001 0.001 config.py:9(init)
6 0.000 0.000 0.000 0.000 entity.py:108(add_to_filter)
1 0.000 0.000 0.002 0.002 entity.py:24(init)
1 0.001 0.001 0.002 0.002 entity.py:39(create_inverted_index)
493 0.000 0.000 0.001 0.000 entity.py:80(beautify)
1 0.000 0.000 0.000 0.000 entity.py:84(create_bucket_lookup)
1 0.000 0.000 0.000 0.000 main.py:15()
2 0.000 0.000 0.000 0.000 main.py:18()
1 0.000 0.000 0.003 0.003 main.py:23(main)
1 0.000 0.000 0.000 0.000 main.py:9(get_bag_of_words)
19 0.000 0.000 0.000 0.000 {built-in method group}
34 0.000 0.000 0.000 0.000 {built-in method match}
1 0.000 0.000 0.000 0.000 {isinstance}
2 0.000 0.000 0.000 0.000 {len}
28 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
15 0.000 0.000 0.000 0.000 {method 'copy' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
15 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
19 0.000 0.000 0.000 0.000 {method 'isspace' of 'str' objects}
24 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects}
49 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
20 0.000 0.000 0.000 0.000 {method 'readline' of 'file' objects}
6 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects}
24 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
47 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
9 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
1030 0.000 0.000 0.000 0.000 {method 'strip' of 'str' objects}
15 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'write' of 'file' objects}
10 0.000 0.000 0.000 0.000 {open}
2 0.000 0.000 0.000 0.000 {range}
3 0.000 0.000 0.000 0.000 {reduce}
Done
real 0m0.063s
user 0m0.050s
sys 0m0.010s
While the cProfile says it took only 0.003 seconds, why is unix (sys) time saying it runs in 0.01 seconds?
time(1) is measuring the execution time of the whole process, whereas the profiler excludes Python interpreter startup time, bytecode compilation time, etc.