Develop Reference

Leetcode 1155. dictionary based memoization vs LRU Cache

I was solving leetcode 1155 which is about number of dice rolls with target sum. I was using dictionary-based memorization. Here's the exact code:
class Solution:
def numRollsToTarget(self, dices: int, faces: int, target: int) -> int:
dp = {}
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
if dp.get((t,rd)): return dp[(t,rd)]
dp[(t,rd)] = sum(ways(t-i, rd-1) for i in range(1,faces+1))
return dp[(t,rd)]
return ways(target, dices)
But this solution is invariably timing out for a combination of face and dices around 15*15
Then I found this solution which uses functools.lru_cache and the rest of it is exactly the same. This solution works very fast.
class Solution:
def numRollsToTarget(self, dices: int, faces: int, target: int) -> int:
from functools import lru_cache
#lru_cache(None)
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
return sum(ways(t-i, rd-1) for i in range(1,faces+1))
return ways(target, dices)
Earlier, I have compared and found that in most cases, lru_cache does not outperform dictionary-based cache by such a margin.
Can someone explain the reason why there is such a drastic performance difference between the two approaches?

First, running your OP code with cProfile and this is the report:
with print(numRollsToTarget2(4, 6, 20)) (OP version)
You can spot right away there're some heavy calls in ways genexpr and sum. That's prob. need close examinations and try to improve/reduce. Next posting is for similar memo version, but the calls is much less. And that version has passed w/o timeout.
35
2864 function calls (366 primitive calls) in 0.018 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.018 0.018 <string>:1(<module>)
1 0.000 0.000 0.001 0.001 dice_rolls.py:23(numRollsToTarget2)
1075/1 0.001 0.000 0.001 0.001 dice_rolls.py:25(ways)
1253/7 0.001 0.000 0.001 0.000 dice_rolls.py:30(<genexpr>)
1 0.000 0.000 0.018 0.018 dice_rolls.py:36(main)
21 0.000 0.000 0.000 0.000 rpc.py:153(debug)
3 0.000 0.000 0.017 0.006 rpc.py:216(remotecall)
3 0.000 0.000 0.000 0.000 rpc.py:226(asynccall)
3 0.000 0.000 0.016 0.005 rpc.py:246(asyncreturn)
3 0.000 0.000 0.000 0.000 rpc.py:252(decoderesponse)
3 0.000 0.000 0.016 0.005 rpc.py:290(getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:298(_proxify)
3 0.000 0.000 0.016 0.005 rpc.py:306(_getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:328(newseq)
3 0.000 0.000 0.000 0.000 rpc.py:332(putmessage)
2 0.000 0.000 0.001 0.000 rpc.py:559(__getattr__)
3 0.000 0.000 0.000 0.000 rpc.py:57(dumps)
1 0.000 0.000 0.001 0.001 rpc.py:577(__getmethods)
2 0.000 0.000 0.000 0.000 rpc.py:601(__init__)
2 0.000 0.000 0.016 0.008 rpc.py:606(__call__)
4 0.000 0.000 0.000 0.000 run.py:412(encoding)
4 0.000 0.000 0.000 0.000 run.py:416(errors)
2 0.000 0.000 0.017 0.008 run.py:433(write)
6 0.000 0.000 0.000 0.000 threading.py:1306(current_thread)
3 0.000 0.000 0.000 0.000 threading.py:222(__init__)
3 0.000 0.000 0.016 0.005 threading.py:270(wait)
3 0.000 0.000 0.000 0.000 threading.py:81(RLock)
3 0.000 0.000 0.000 0.000 {built-in method _struct.pack}
3 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
6 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
1 0.000 0.000 0.018 0.018 {built-in method builtins.exec}
6 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
9 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1 0.000 0.000 0.017 0.017 {built-in method builtins.print}
179/1 0.000 0.000 0.001 0.001 {built-in method builtins.sum}
3 0.000 0.000 0.000 0.000 {built-in method select.select}
3 0.000 0.000 0.000 0.000 {method '_acquire_restore' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_is_owned' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_release_save' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
6 0.016 0.003 0.016 0.003 {method 'acquire' of '_thread.lock' objects}
3 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects}
2 0.000 0.000 0.000 0.000 {method 'decode' of 'bytes' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
3 0.000 0.000 0.000 0.000 {method 'dump' of '_pickle.Pickler' objects}
2 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
201 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {method 'getvalue' of '_io.BytesIO' objects}
3 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'send' of '_socket.socket' objects}
Then I tried to run modified/simplified version, and compare the results.
35
387 function calls (193 primitive calls) in 0.006 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.006 0.006 <string>:1(<module>)
1 0.000 0.000 0.006 0.006 dice_rolls.py:36(main)
1 0.000 0.000 0.000 0.000 dice_rolls.py:5(numRollsToTarget)
195/1 0.000 0.000 0.000 0.000 dice_rolls.py:8(dp)
21 0.000 0.000 0.000 0.000 rpc.py:153(debug)
3 0.000 0.000 0.006 0.002 rpc.py:216(remotecall)
3 0.000 0.000 0.000 0.000 rpc.py:226(asynccall)
3 0.000 0.000 0.006 0.002 rpc.py:246(asyncreturn)
3 0.000 0.000 0.000 0.000 rpc.py:252(decoderesponse)
3 0.000 0.000 0.006 0.002 rpc.py:290(getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:298(_proxify)
3 0.000 0.000 0.006 0.002 rpc.py:306(_getresponse)
3 0.000 0.000 0.000 0.000 rpc.py:328(newseq)
3 0.000 0.000 0.000 0.000 rpc.py:332(putmessage)
2 0.000 0.000 0.001 0.000 rpc.py:559(__getattr__)
3 0.000 0.000 0.000 0.000 rpc.py:57(dumps)
1 0.000 0.000 0.001 0.001 rpc.py:577(__getmethods)
2 0.000 0.000 0.000 0.000 rpc.py:601(__init__)
2 0.000 0.000 0.005 0.003 rpc.py:606(__call__)
4 0.000 0.000 0.000 0.000 run.py:412(encoding)
4 0.000 0.000 0.000 0.000 run.py:416(errors)
2 0.000 0.000 0.006 0.003 run.py:433(write)
6 0.000 0.000 0.000 0.000 threading.py:1306(current_thread)
3 0.000 0.000 0.000 0.000 threading.py:222(__init__)
3 0.000 0.000 0.006 0.002 threading.py:270(wait)
3 0.000 0.000 0.000 0.000 threading.py:81(RLock)
3 0.000 0.000 0.000 0.000 {built-in method _struct.pack}
3 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
6 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
1 0.000 0.000 0.006 0.006 {built-in method builtins.exec}
6 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
9 0.000 0.000 0.000 0.000 {built-in method builtins.len}
34 0.000 0.000 0.000 0.000 {built-in method builtins.max}
1 0.000 0.000 0.006 0.006 {built-in method builtins.print}
3 0.000 0.000 0.000 0.000 {built-in method select.select}
3 0.000 0.000 0.000 0.000 {method '_acquire_restore' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_is_owned' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method '_release_save' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
6 0.006 0.001 0.006 0.001 {method 'acquire' of '_thread.lock' objects}
3 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects}
2 0.000 0.000 0.000 0.000 {method 'decode' of 'bytes' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
3 0.000 0.000 0.000 0.000 {method 'dump' of '_pickle.Pickler' objects}
2 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {method 'getvalue' of '_io.BytesIO' objects}
3 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
3 0.000 0.000 0.000 0.000 {method 'send' of '_socket.socket' objects}
The profiling codes are here:
import cProfile
from typing import List
def numRollsToTarget(d, f, target):
memo = {}
def dp(d, target):
if d == 0:
return 0 if target > 0 else 1
if (d, target) in memo:
return memo[(d, target)]
result = 0
for k in range(max(0, target-f), target):
result += dp(d-1, k)
memo[(d, target)] = result
return result
return dp(d, target) % (10**9 + 7)
def numRollsToTarget2(dices: int, faces: int, target: int) -> int:
dp = {}
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
if dp.get((t,rd)): return dp[(t,rd)]
dp[(t,rd)] = sum(ways(t-i, rd-1) for i in range(1,faces+1))
return dp[(t,rd)]
return ways(target, dices)
def numRollsToTarget3(dices: int, faces: int, target: int) -> int:
from functools import lru_cache
#lru_cache(None)
def ways(t, rd):
if t == 0 and rd == 0: return 1
if t <= 0 or rd <= 0: return 0
return sum(ways(t-i, rd-1) for i in range(1,faces+1))
return ways(target, dices)
def main():
print(numRollsToTarget(4, 6, 20))
#print(numRollsToTarget2(4, 6, 20))
#print(numRollsToTarget3(4, 6, 20)) # not faster than first
if __name__ == '__main__':
cProfile.run('main()')

Performance problems w/ many widgets respectively stacked QTableWidgets

I need to display a complex Table and decided to use stacked QTableWidgets. With an increasing number of rows the program needs a lot of time for creating all the widgets and almost the same time for displaying.
The maintable looks like this: MainTable
The stacked TableWidget in the table:
StackedTables
if the cell contains data, there is at least one TableWidget in one cell of the MainTable and in the worst case there are 2 more TableWidgets in that one. That means I could have 3 TableWidgets in one cell.
Time measurement with cProfile and time.time for 80rows (with 48 of the complex cells for each row):
complete update time: 15s (manually stopped)
time to create the table: 7.548534870147705s (time.time over complete function)
display time: 7.5s (complete update time - function time)
rows: 80
63600 function calls in 7.462 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1896 2.455 0.001 3.882 0.002 DigitalePlanungstafel.py:6054(grundWidgetErstellen) -- (create table in cell)
3936 2.027 0.001 2.027 0.001 {built-in method setCellWidget}
2535 1.306 0.001 1.306 0.001 {built-in method setColumnCount}
630 0.770 0.001 1.183 0.002 DigitalePlanungstafel.py:6035(obenWidgetErstellen) -- (create table in table in cell)
2607 0.674 0.000 0.674 0.000 {built-in method setRowCount}
2528 0.059 0.000 0.059 0.000 {built-in method horizontalHeader}
2526 0.021 0.000 0.021 0.000 {built-in method verticalHeader}
163 0.019 0.000 0.019 0.000 {method 'execute' of 'sqlite3.Cursor' objects}
2526 0.016 0.000 0.016 0.000 {built-in method setFrameShape}
1410 0.014 0.000 0.014 0.000 {built-in method setStyleSheet}
4502 0.013 0.000 0.013 0.000 {built-in method setRowHeight}
2526 0.009 0.000 0.009 0.000 {built-in method setFixedSize}
2546 0.009 0.000 0.009 0.000 {built-in method setColumnWidth}
5052 0.009 0.000 0.009 0.000 {built-in method setVisible}
1329 0.007 0.000 0.007 0.000 {built-in method setItem}
2181 0.006 0.000 0.006 0.000 {built-in method cellWidget}
80 0.005 0.000 0.005 0.000 {built-in method addWidget}
2526 0.004 0.000 0.004 0.000 {built-in method setEditTriggers}
929 0.004 0.000 0.004 0.000 {built-in method setBackground}
1330 0.003 0.000 0.003 0.000 {method 'format' of 'str' objects}
414 0.003 0.000 0.003 0.000 {built-in method _pickle.loads}
336 0.003 0.000 0.003 0.000 {method 'strftime' of 'datetime.date' objects}
2526 0.003 0.000 0.003 0.000 {built-in method setHorizontalScrollBarPolicy}
1410 0.002 0.000 0.002 0.000 {built-in method setFixedHeight}
1377 0.002 0.000 0.002 0.000 {built-in method setTextAlignment}
83 0.002 0.000 0.004 0.000 _strptime.py:321(_strptime)
2 0.002 0.001 0.002 0.001 {built-in method setSortingEnabled}
2526 0.001 0.000 0.001 0.000 {built-in method setShowGrid}
1570 0.001 0.000 0.001 0.000 {built-in method rowHeight}
2526 0.001 0.000 0.001 0.000 {built-in method setSelectionMode}
163 0.001 0.000 0.001 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
240 0.001 0.000 0.001 0.000 DigitalePlanungstafel.py:7494(__init__)
2526 0.001 0.000 0.001 0.000 {built-in method setVerticalScrollBarPolicy}
1 0.001 0.001 0.001 0.001 {built-in method sortByColumn}
80 0.001 0.000 0.001 0.000 {built-in method setLayout}
83 0.001 0.000 0.001 0.000 {built-in method _locale.setlocale}
1 0.001 0.001 0.001 0.001 {built-in method _sqlite3.connect}
89 0.000 0.000 0.000 0.000 {built-in method setForeground}
83 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
83 0.000 0.000 0.004 0.000 _strptime.py:562(_strptime_datetime)
48 0.000 0.000 0.000 0.000 {built-in method setHorizontalHeaderItem}
80 0.000 0.000 0.000 0.000 {built-in method setContentsMargins}
83 0.000 0.000 0.005 0.000 {built-in method strptime}
83 0.000 0.000 0.000 0.000 locale.py:379(normalize)
88 0.000 0.000 0.000 0.000 {built-in method setFont}
160 0.000 0.000 0.000 0.000 {built-in method setData}
83 0.000 0.000 0.000 0.000 {method 'groupdict' of '_sre.SRE_Match' objects}
80 0.000 0.000 0.000 0.000 {built-in method setAlignment}
83 0.000 0.000 0.001 0.000 _strptime.py:29(_getlang)
83 0.000 0.000 0.001 0.000 locale.py:565(getlocale)
83 0.000 0.000 0.001 0.000 locale.py:462(_parse_localename)
80 0.000 0.000 0.000 0.000 {built-in method setUnderline}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
249 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of 'sqlite3.Connection' objects}
160 0.000 0.000 0.000 0.000 {built-in method item}
742 0.000 0.000 0.000 0.000 DigitalePlanungstafel.py:7499(__lt__)
48 0.000 0.000 0.000 0.000 {built-in method today}
165 0.000 0.000 0.000 0.000 {method 'toordinal' of 'datetime.date' objects}
475 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
167 0.000 0.000 0.000 0.000 {built-in method builtins.len}
166 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
83 0.000 0.000 0.000 0.000 {method 'end' of '_sre.SRE_Match' objects}
84 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.TextIOWrapper' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.print}
83 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
47 0.000 0.000 0.000 0.000 {built-in method columnCount}
83 0.000 0.000 0.000 0.000 {method 'weekday' of 'datetime.date' objects}
96 0.000 0.000 0.000 0.000 {method 'date' of 'datetime.datetime' objects}
20 0.000 0.000 0.000 0.000 {built-in method columnWidth}
1 0.000 0.000 0.000 0.000 {built-in method _locale._getdefaultlocale}
3 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
1 0.000 0.000 0.000 0.000 _strptime.py:284(_calc_julian_from_U_or_W)
1 0.000 0.000 0.000 0.000 _bootlocale.py:11(getpreferredencoding)
1 0.000 0.000 0.000 0.000 {built-in method sortIndicatorOrder}
2 0.000 0.000 0.000 0.000 {built-in method time.time}
1 0.000 0.000 0.000 0.000 {method 'cursor' of 'sqlite3.Connection' objects}
2 0.000 0.000 0.000 0.000 {method 'index' of 'list' objects}
1 0.000 0.000 0.000 0.000 {built-in method fromordinal}
1 0.000 0.000 0.000 0.000 {built-in method sortIndicatorSection}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 codecs.py:259(__init__)
The time for calling the function and creating the Table is okay but the program needs almost the same time again until it reacts.
The Goal would be to reduce the update time by approx. 50%. I need to display around 200 rows.
Is the stacked QTableWidget the right approach - If so what do I have to do to optimize the update times?
I already thought about changing the presenatation from QTableWidget to a QGraphicsView and simply drawing the rectangles.
Or maybe a combnation, for example: Using the QTableWidget for the Header and the first columns and then merging all the complex cells and inserting a QGraphicsView but I am not sure if I am able to get the right size for the drawed cells and I do not know if the displaying time will be shorter.
What do you guys think is the right approch for a table like this?
If you need I can append the function, which is updating the Table.
EDIT:
I am using now 2 rows in the MainTable for one Block which reduced the number of stacked widgets from 2.526 to 7!
The time measurement for the same rows now looks like this:
complete update time: ~2s (manually stopped)
time to create the table: 0.572490930557251 (time.time over complete function)
display time: ~1.5s (complete update time - function time)
rows: 160
20912 function calls in 0.534 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1417 0.258 0.000 0.258 0.000 {built-in method setCellWidget}
88 0.209 0.002 0.209 0.002 {built-in method setRowCount}
163 0.017 0.000 0.017 0.000 {method 'execute' of 'sqlite3.Cursor' objects}
1410 0.010 0.000 0.010 0.000 {built-in method setStyleSheet}
80 0.005 0.000 0.005 0.000 {built-in method addWidget}
1336 0.004 0.000 0.004 0.000 {built-in method setItem}
7 0.003 0.000 0.004 0.001 DigitalePlanungstafel.py:6037(obenWidgetErstellen)
501 0.003 0.000 0.003 0.000 {built-in method _pickle.loads}
1990 0.003 0.000 0.003 0.000 {built-in method cellWidget}
336 0.002 0.000 0.002 0.000 {method 'strftime' of 'datetime.date' objects}
83 0.002 0.000 0.004 0.000 _strptime.py:321(_strptime)
1330 0.002 0.000 0.002 0.000 {method 'format' of 'str' objects}
1410 0.002 0.000 0.002 0.000 {built-in method setFixedHeight}
929 0.001 0.000 0.001 0.000 {built-in method setBackground}
1377 0.001 0.000 0.001 0.000 {built-in method setTextAlignment}
240 0.001 0.000 0.001 0.000 DigitalePlanungstafel.py:7452(__init__)
163 0.001 0.000 0.001 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
16 0.001 0.000 0.001 0.000 {built-in method setColumnCount}
770 0.001 0.000 0.001 0.000 {built-in method setSpan}
2127 0.001 0.000 0.001 0.000 {built-in method item}
1570 0.001 0.000 0.001 0.000 {built-in method rowHeight}
80 0.001 0.000 0.001 0.000 {built-in method setLayout}
83 0.000 0.000 0.000 0.000 {built-in method _locale.setlocale}
1 0.000 0.000 0.000 0.000 {built-in method _sqlite3.connect}
167 0.000 0.000 0.000 0.000 {built-in method setRowHeight}
89 0.000 0.000 0.000 0.000 {built-in method setForeground}
83 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
48 0.000 0.000 0.000 0.000 {built-in method setHorizontalHeaderItem}
83 0.000 0.000 0.004 0.000 _strptime.py:562(_strptime_datetime)
7 0.000 0.000 0.000 0.000 {built-in method takeItem}
80 0.000 0.000 0.000 0.000 {built-in method setContentsMargins}
83 0.000 0.000 0.004 0.000 {built-in method strptime}
88 0.000 0.000 0.000 0.000 {built-in method setFont}
160 0.000 0.000 0.000 0.000 {built-in method setData}
83 0.000 0.000 0.000 0.000 locale.py:379(normalize)
80 0.000 0.000 0.000 0.000 {built-in method setAlignment}
83 0.000 0.000 0.001 0.000 _strptime.py:29(_getlang)
83 0.000 0.000 0.000 0.000 {method 'groupdict' of '_sre.SRE_Match' objects}
83 0.000 0.000 0.001 0.000 locale.py:565(getlocale)
1 0.000 0.000 0.000 0.000 {method 'close' of 'sqlite3.Connection' objects}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
7 0.000 0.000 0.000 0.000 {built-in method horizontalHeader}
80 0.000 0.000 0.000 0.000 {built-in method setUnderline}
249 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
83 0.000 0.000 0.000 0.000 locale.py:462(_parse_localename)
48 0.000 0.000 0.000 0.000 {built-in method today}
7 0.000 0.000 0.000 0.000 {built-in method verticalHeader}
475 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
167 0.000 0.000 0.000 0.000 {built-in method builtins.len}
165 0.000 0.000 0.000 0.000 {method 'toordinal' of 'datetime.date' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.TextIOWrapper' objects}
166 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
7 0.000 0.000 0.000 0.000 {built-in method setFrameShape}
27 0.000 0.000 0.000 0.000 {built-in method setColumnWidth}
84 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'sort' of 'list' objects}
83 0.000 0.000 0.000 0.000 {method 'end' of '_sre.SRE_Match' objects}
87 0.000 0.000 0.000 0.000 DigitalePlanungstafel.py:6059(<lambda>)
83 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.print}
7 0.000 0.000 0.000 0.000 {built-in method setFixedSize}
14 0.000 0.000 0.000 0.000 {built-in method setVisible}
83 0.000 0.000 0.000 0.000 {method 'weekday' of 'datetime.date' objects}
96 0.000 0.000 0.000 0.000 {method 'date' of 'datetime.datetime' objects}
7 0.000 0.000 0.000 0.000 {built-in method setEditTriggers}
20 0.000 0.000 0.000 0.000 {built-in method columnWidth}
7 0.000 0.000 0.000 0.000 {built-in method setHorizontalScrollBarPolicy}
1 0.000 0.000 0.000 0.000 {built-in method _locale._getdefaultlocale}
The updateing time is more than good but I have 2 problems because of the splitting.
Sorting (over the header) is not working anymore. This is caused by the merging, it messes up everything. Is there a way to freeze two associated rows before sorting?
I only want one row to be selected at a time. Cause of the splitting I need to select the two associated rows no matter which one gets selected. Not really a big deal but doesen´t really look good (see pictures).
MainTable selected row
MainTable selected row

You do not want to do any of this - instead:
Use a model for your data.
Use a viewmodel that adapts the model to the view you wish to have.
Use a QTableView to display the viewmodel.

CPU bound code twice as fast on laptop as desktop

I have a single-threaded Python 3 program that's CPU bound, the only IO is printing a couple of lines to output (no reading/writing of files).
On my desktop machine (AMD Ryzen 1700x 3.8 GHz, 16GB 3000 MHz DDR4) it performs (consistently) at 3400 episodes/second where a run takes around 60 seconds.
On my laptop (Intel i7-6600U 2.8 GHz, 16GB 2000 MHz DDR3) the performance is doubled at 7000 episodes/second, and a run coming in at just under 30 seconds.
Both machines run the same operating systems (Fedora 26) and the same python version (not built from source).
What's more, when profiling, there's a line showing
10.999 tottime, 28.814 cumtime for arrayprint.py:557(fillFormat)
but only when the code is run on the desktop. On the laptop, the particular function does not appear at all (and none of the arrayprint functions use more than 1 second tottime).
Not only is it strange that the performance differs between the machines, but no arrays or lists are ever printed to the screen, converted to strings, or saved to files during the execution of the program.
Here's the full profile for the desktop:
54499635 function calls (53787999 primitive calls) in 58.746 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
533727 0.359 0.000 0.514 0.000 <frozen importlib._bootstrap>:402(parent)
533727 0.469 0.000 0.697 0.000 <frozen importlib._bootstrap>:989(_handle_fromlist)
1 0.000 0.000 58.746 58.746 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 __init__.py:120(getLevelName)
567524 0.237 0.000 0.727 0.000 __init__.py:1284(debug)
2 0.000 0.000 0.000 0.000 __init__.py:1296(info)
2 0.000 0.000 0.000 0.000 __init__.py:1308(warning)
2 0.000 0.000 0.000 0.000 __init__.py:1320(warn)
4 0.000 0.000 0.000 0.000 __init__.py:1374(findCaller)
4 0.000 0.000 0.000 0.000 __init__.py:1404(makeRecord)
4 0.000 0.000 0.000 0.000 __init__.py:1419(_log)
4 0.000 0.000 0.000 0.000 __init__.py:1444(handle)
4 0.000 0.000 0.000 0.000 __init__.py:1498(callHandlers)
567528 0.175 0.000 0.175 0.000 __init__.py:1528(getEffectiveLevel)
567528 0.315 0.000 0.490 0.000 __init__.py:1542(isEnabledFor)
4 0.000 0.000 0.000 0.000 __init__.py:157(<lambda>)
4 0.000 0.000 0.000 0.000 __init__.py:251(__init__)
4 0.000 0.000 0.000 0.000 __init__.py:329(getMessage)
4 0.000 0.000 0.000 0.000 __init__.py:387(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:390(format)
4 0.000 0.000 0.000 0.000 __init__.py:540(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:546(formatMessage)
4 0.000 0.000 0.000 0.000 __init__.py:562(format)
8 0.000 0.000 0.000 0.000 __init__.py:703(filter)
8 0.000 0.000 0.000 0.000 __init__.py:807(acquire)
8 0.000 0.000 0.000 0.000 __init__.py:814(release)
4 0.000 0.000 0.000 0.000 __init__.py:827(format)
4 0.000 0.000 0.000 0.000 __init__.py:850(handle)
4 0.000 0.000 0.000 0.000 __init__.py:969(flush)
4 0.000 0.000 0.000 0.000 __init__.py:980(emit)
289159 0.101 0.000 1.491 0.000 _methods.py:31(_sum)
533727 0.175 0.000 1.613 0.000 _methods.py:37(_any)
177909 0.862 0.000 33.737 0.000 arrayprint.py:237(_get_formatdict)
177909 0.370 0.000 34.214 0.000 arrayprint.py:273(_get_format_function)
177909 0.686 0.000 39.971 0.000 arrayprint.py:315(_array2string)
533727/177909 0.674 0.000 40.351 0.000 arrayprint.py:340(array2string)
1224652 0.960 0.000 1.554 0.000 arrayprint.py:467(_extendLine)
177909 1.671 0.000 4.320 0.000 arrayprint.py:475(_formatArray)
533727 0.682 0.000 29.496 0.000 arrayprint.py:543(__init__)
533727 10.999 0.000 28.814 0.000 arrayprint.py:557(fillFormat)
355336 1.600 0.000 5.432 0.000 arrayprint.py:589(<listcomp>)
2416068 2.677 0.000 3.832 0.000 arrayprint.py:642(_digits)
177909 0.720 0.000 2.378 0.000 arrayprint.py:652(__init__)
1224652 1.057 0.000 1.057 0.000 arrayprint.py:665(__call__)
533727 0.147 0.000 0.147 0.000 arrayprint.py:674(__init__)
177909 0.227 0.000 0.319 0.000 arrayprint.py:702(__init__)
177909 0.415 0.000 17.986 0.000 arrayprint.py:713(__init__)
177909 0.166 0.000 0.166 0.000 arrayprint.py:730(__init__)
177909 0.046 0.000 0.046 0.000 arrayprint.py:751(__init__)
1 0.000 0.000 0.000 0.000 enum.py:265(__call__)
1 0.000 0.000 0.000 0.000 enum.py:515(__new__)
1 0.000 0.000 0.000 0.000 enum.py:544(_missing_)
177909 0.206 0.000 0.206 0.000 enum.py:552(__str__)
177909 0.269 0.000 0.475 0.000 enum.py:564(__format__)
755408 0.248 0.000 0.366 0.000 enum.py:579(__hash__)
200000 0.037 0.000 0.037 0.000 enum.py:592(name)
27976 0.005 0.000 0.005 0.000 enum.py:597(value)
200524 0.443 0.000 0.641 0.000 eventgen.py:115(_push)
200001 0.492 0.000 0.885 0.000 eventgen.py:122(pop)
200000 0.892 0.000 1.036 0.000 eventgen.py:137(ce_str)
13988 0.017 0.000 0.034 0.000 eventgen.py:15(__lt__)
99676 0.168 0.000 0.911 0.000 eventgen.py:44(event_new)
79335 0.096 0.000 0.520 0.000 eventgen.py:52(event_end)
11689 0.078 0.000 0.261 0.000 eventgen.py:61(event_new_handoff)
9824 0.014 0.000 0.098 0.000 eventgen.py:90(event_end_handoff)
77441 0.295 0.000 0.380 0.000 eventgen.py:94(reassign)
177909 0.177 0.000 0.555 0.000 fromnumeric.py:1364(ravel)
200001 0.093 0.000 0.464 0.000 fromnumeric.py:1471(nonzero)
289159 0.542 0.000 2.148 0.000 fromnumeric.py:1710(sum)
533727 0.637 0.000 2.956 0.000 fromnumeric.py:1866(any)
200001 0.120 0.000 0.372 0.000 fromnumeric.py:55(_wrapfunc)
4 0.000 0.000 0.000 0.000 genericpath.py:117(_splitext)
49 0.000 0.000 0.001 0.000 grid.py:172(neighbors1)
49 0.001 0.000 0.001 0.000 grid.py:195(neighbors2)
533727/177909 0.311 0.000 40.472 0.000 numeric.py:1927(array_str)
1067454 1.724 0.000 4.091 0.000 numeric.py:2692(seterr)
1067454 1.466 0.000 1.603 0.000 numeric.py:2792(geterr)
533727 0.299 0.000 0.422 0.000 numeric.py:3085(__init__)
533727 0.411 0.000 2.588 0.000 numeric.py:3089(__enter__)
533727 0.461 0.000 2.374 0.000 numeric.py:3094(__exit__)
177909 0.064 0.000 0.151 0.000 numeric.py:463(asarray)
711636 0.223 0.000 0.503 0.000 numeric.py:534(asanyarray)
4 0.000 0.000 0.000 0.000 posixpath.py:119(splitext)
4 0.000 0.000 0.000 0.000 posixpath.py:142(basename)
4 0.000 0.000 0.000 0.000 posixpath.py:39(_get_sep)
6 0.000 0.000 0.000 0.000 posixpath.py:50(normcase)
4 0.000 0.000 0.000 0.000 process.py:137(name)
4 0.000 0.000 0.000 0.000 process.py:35(current_process)
1 0.000 0.000 0.000 0.000 signal.py:25(_int_to_enum)
2 0.000 0.000 0.000 0.000 signal.py:35(_enum_to_int)
1 0.000 0.000 0.000 0.000 signal.py:45(signal)
99627 0.062 0.000 0.062 0.000 stats.py:38(new)
20292 0.028 0.000 0.039 0.000 stats.py:42(new_rej)
88750 0.047 0.000 0.047 0.000 stats.py:48(end)
11623 0.005 0.000 0.005 0.000 stats.py:51(hoff_new)
1799 0.001 0.000 0.002 0.000 stats.py:54(hoff_rej)
22091 0.012 0.000 0.012 0.000 stats.py:58(rej)
200000 0.234 0.000 1.513 0.000 stats.py:64(iter)
1 0.000 0.000 0.000 0.000 stats.py:69(n_iter)
1 0.000 0.000 0.000 0.000 stats.py:86(endsim)
1 0.000 0.000 0.001 0.001 strats.py:189(get_init_action)
200000 1.070 0.000 49.964 0.000 strats.py:193(get_action)
177909 1.348 0.000 1.937 0.000 strats.py:220(execute_action)
200001 4.572 0.000 47.626 0.000 strats.py:243(optimal_ch)
89158 0.071 0.000 0.958 0.000 strats.py:299(reward)
89158 0.018 0.000 0.018 0.000 strats.py:308(discount)
1242355 0.944 0.000 0.944 0.000 strats.py:333(get_qval)
89158 0.160 0.000 0.160 0.000 strats.py:336(update_qval)
1 0.000 0.000 58.746 58.746 strats.py:40(init_sim)
1 1.271 1.271 58.745 58.745 strats.py:49(_simulate)
4 0.000 0.000 0.000 0.000 threading.py:1076(name)
4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread)
227976 0.120 0.000 0.162 0.000 types.py:135(__get__)
177909 0.079 0.000 0.079 0.000 {built-in method _functools.reduce}
200001 0.192 0.000 0.222 0.000 {built-in method _heapq.heappop}
200524 0.084 0.000 0.088 0.000 {built-in method _heapq.heappush}
310143 0.064 0.000 0.064 0.000 {built-in method _operator.gt}
843054 0.152 0.000 0.152 0.000 {built-in method _operator.lt}
1 0.000 0.000 0.000 0.000 {built-in method _signal.signal}
8 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
2 0.000 0.000 0.000 0.000 {built-in method _warnings.warn}
1 0.000 0.000 58.746 58.746 {built-in method builtins.exec}
200001 0.056 0.000 0.056 0.000 {built-in method builtins.getattr}
1067468 0.228 0.000 0.228 0.000 {built-in method builtins.hasattr}
755408 0.118 0.000 0.118 0.000 {built-in method builtins.hash}
467082 0.164 0.000 0.164 0.000 {built-in method builtins.isinstance}
533727 0.107 0.000 0.107 0.000 {built-in method builtins.issubclass}
10361766 1.076 0.000 1.076 0.000 {built-in method builtins.len}
533441 0.304 0.000 0.304 0.000 {built-in method builtins.max}
533923 0.198 0.000 0.198 0.000 {built-in method builtins.min}
889545 0.368 0.000 0.368 0.000 {built-in method numpy.core.multiarray.array}
111251 0.101 0.000 0.101 0.000 {built-in method numpy.core.multiarray.where}
2134908 0.377 0.000 0.377 0.000 {built-in method numpy.core.umath.geterrobj}
1067454 0.524 0.000 0.524 0.000 {built-in method numpy.core.umath.seterrobj}
14 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getpid}
4 0.000 0.000 0.000 0.000 {built-in method sys._getframe}
5 0.000 0.000 0.000 0.000 {built-in method time.time}
8 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
533727 0.299 0.000 1.912 0.000 {method 'any' of 'numpy.ndarray' objects}
875 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
16544 0.119 0.000 0.119 0.000 {method 'choice' of 'mtrand.RandomState' objects}
533727 0.872 0.000 0.872 0.000 {method 'compress' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
188835 0.641 0.000 0.641 0.000 {method 'exponential' of 'mtrand.RandomState' objects}
4 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'flush' of '_io.TextIOWrapper' objects}
8 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
355818 0.069 0.000 0.069 0.000 {method 'item' of 'numpy.ndarray' objects}
200001 0.196 0.000 0.196 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
533727 0.123 0.000 0.123 0.000 {method 'pop' of 'dict' objects}
11689 0.053 0.000 0.053 0.000 {method 'randint' of 'mtrand.RandomState' objects}
168494 0.128 0.000 0.128 0.000 {method 'random_sample' of 'mtrand.RandomState' objects}
177909 0.232 0.000 0.232 0.000 {method 'ravel' of 'numpy.ndarray' objects}
1889376 5.023 0.000 5.023 0.000 {method 'reduce' of 'numpy.ufunc' objects}
8 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
533727 0.155 0.000 0.155 0.000 {method 'rpartition' of 'str' objects}
4865786 1.100 0.000 1.100 0.000 {method 'rstrip' of 'str' objects}
8 0.000 0.000 0.000 0.000 {method 'write' of '_io.TextIOWrapper' objects}
And here's for the laptop:
27738517 function calls (26673571 primitive calls) in 28.612 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 28.612 28.612 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 __init__.py:120(getLevelName)
566894 0.244 0.000 0.720 0.000 __init__.py:1284(debug)
2 0.000 0.000 0.000 0.000 __init__.py:1296(info)
2 0.000 0.000 0.000 0.000 __init__.py:1308(warning)
2 0.000 0.000 0.000 0.000 __init__.py:1320(warn)
4 0.000 0.000 0.000 0.000 __init__.py:1374(findCaller)
4 0.000 0.000 0.000 0.000 __init__.py:1404(makeRecord)
4 0.000 0.000 0.000 0.000 __init__.py:1419(_log)
4 0.000 0.000 0.000 0.000 __init__.py:1444(handle)
4 0.000 0.000 0.000 0.000 __init__.py:1498(callHandlers)
566898 0.166 0.000 0.166 0.000 __init__.py:1528(getEffectiveLevel)
566898 0.309 0.000 0.476 0.000 __init__.py:1542(isEnabledFor)
4 0.000 0.000 0.000 0.000 __init__.py:157(<lambda>)
4 0.000 0.000 0.000 0.000 __init__.py:251(__init__)
4 0.000 0.000 0.000 0.000 __init__.py:329(getMessage)
4 0.000 0.000 0.000 0.000 __init__.py:387(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:390(format)
4 0.000 0.000 0.000 0.000 __init__.py:540(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:546(formatMessage)
4 0.000 0.000 0.000 0.000 __init__.py:562(format)
8 0.000 0.000 0.000 0.000 __init__.py:703(filter)
8 0.000 0.000 0.000 0.000 __init__.py:807(acquire)
8 0.000 0.000 0.000 0.000 __init__.py:814(release)
4 0.000 0.000 0.000 0.000 __init__.py:827(format)
4 0.000 0.000 0.000 0.000 __init__.py:850(handle)
4 0.000 0.000 0.000 0.000 __init__.py:969(flush)
4 0.000 0.000 0.000 0.000 __init__.py:980(emit)
288946 0.112 0.000 1.643 0.000 _methods.py:31(_sum)
177491 0.330 0.000 0.330 0.000 arrayprint.py:256(_get_formatdict)
177491 0.169 0.000 3.542 0.000 arrayprint.py:259(<lambda>)
177491 0.465 0.000 4.419 0.000 arrayprint.py:299(_get_format_function)
177491 0.623 0.000 9.729 0.000 arrayprint.py:343(_array2string)
532473/177491 0.987 0.000 10.679 0.000 arrayprint.py:381(wrapper)
532473/177491 0.721 0.000 10.150 0.000 arrayprint.py:399(array2string)
1225350 0.971 0.000 1.470 0.000 arrayprint.py:527(_extendLine)
177491 1.458 0.000 3.920 0.000 arrayprint.py:535(_formatArray)
177491 0.768 0.000 3.373 0.000 arrayprint.py:712(__init__)
1225350 0.960 0.000 0.960 0.000 arrayprint.py:725(__call__)
1 0.000 0.000 0.000 0.000 enum.py:265(__call__)
1 0.000 0.000 0.000 0.000 enum.py:515(__new__)
1 0.000 0.000 0.000 0.000 enum.py:544(_missing_)
177491 0.209 0.000 0.209 0.000 enum.py:552(__str__)
177491 0.316 0.000 0.525 0.000 enum.py:564(__format__)
755255 0.238 0.000 0.352 0.000 enum.py:579(__hash__)
200000 0.039 0.000 0.039 0.000 enum.py:592(name)
28626 0.005 0.000 0.005 0.000 enum.py:597(value)
200505 0.443 0.000 0.643 0.000 eventgen.py:115(_push)
200001 0.474 0.000 0.863 0.000 eventgen.py:122(pop)
200000 0.834 0.000 0.983 0.000 eventgen.py:137(ce_str)
14313 0.017 0.000 0.035 0.000 eventgen.py:15(__lt__)
99673 0.186 0.000 0.939 0.000 eventgen.py:44(event_new)
78949 0.094 0.000 0.500 0.000 eventgen.py:52(event_end)
11887 0.078 0.000 0.261 0.000 eventgen.py:61(event_new_handoff)
9996 0.017 0.000 0.103 0.000 eventgen.py:90(event_end_handoff)
77374 0.284 0.000 0.364 0.000 eventgen.py:94(reassign)
177491 0.195 0.000 0.595 0.000 fromnumeric.py:1380(ravel)
200001 0.098 0.000 0.490 0.000 fromnumeric.py:1487(nonzero)
288946 0.590 0.000 2.352 0.000 fromnumeric.py:1730(sum)
200001 0.130 0.000 0.392 0.000 fromnumeric.py:55(_wrapfunc)
4 0.000 0.000 0.000 0.000 genericpath.py:117(_splitext)
49 0.000 0.000 0.001 0.000 grid.py:172(neighbors1)
49 0.001 0.000 0.001 0.000 grid.py:195(neighbors2)
532473/177491 0.365 0.000 10.826 0.000 numeric.py:1905(array_str)
177491 0.062 0.000 0.151 0.000 numeric.py:463(asarray)
177491 0.051 0.000 0.104 0.000 numeric.py:534(asanyarray)
4 0.000 0.000 0.000 0.000 posixpath.py:119(splitext)
4 0.000 0.000 0.000 0.000 posixpath.py:142(basename)
4 0.000 0.000 0.000 0.000 posixpath.py:39(_get_sep)
6 0.000 0.000 0.000 0.000 posixpath.py:50(normcase)
4 0.000 0.000 0.000 0.000 process.py:137(name)
4 0.000 0.000 0.000 0.000 process.py:35(current_process)
1 0.000 0.000 0.000 0.000 signal.py:25(_int_to_enum)
2 0.000 0.000 0.000 0.000 signal.py:35(_enum_to_int)
1 0.000 0.000 0.000 0.000 signal.py:45(signal)
99624 0.066 0.000 0.066 0.000 stats.py:38(new)
20675 0.028 0.000 0.040 0.000 stats.py:42(new_rej)
88545 0.045 0.000 0.045 0.000 stats.py:48(end)
11831 0.006 0.000 0.006 0.000 stats.py:51(hoff_new)
1835 0.001 0.000 0.002 0.000 stats.py:54(hoff_rej)
22510 0.013 0.000 0.013 0.000 stats.py:58(rej)
200000 0.261 0.000 1.490 0.000 stats.py:64(iter)
1 0.000 0.000 0.000 0.000 stats.py:69(n_iter)
1 0.000 0.000 0.000 0.000 stats.py:86(endsim)
1 0.000 0.000 0.000 0.000 strats.py:189(get_init_action)
200000 1.234 0.000 19.760 0.000 strats.py:193(get_action)
177490 1.294 0.000 1.860 0.000 strats.py:220(execute_action)
200001 3.897 0.000 17.128 0.000 strats.py:243(optimal_ch)
88945 0.074 0.000 1.112 0.000 strats.py:299(reward)
88945 0.017 0.000 0.017 0.000 strats.py:308(discount)
1241938 0.681 0.000 0.681 0.000 strats.py:333(get_qval)
88945 0.167 0.000 0.167 0.000 strats.py:336(update_qval)
1 0.000 0.000 28.612 28.612 strats.py:40(init_sim)
1 1.383 1.383 28.611 28.611 strats.py:49(_simulate)
4 0.000 0.000 0.000 0.000 threading.py:1076(name)
4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread)
228626 0.122 0.000 0.166 0.000 types.py:135(__get__)
177491 0.075 0.000 0.075 0.000 {built-in method _functools.reduce}
200001 0.203 0.000 0.234 0.000 {built-in method _heapq.heappop}
200505 0.079 0.000 0.083 0.000 {built-in method _heapq.heappush}
320262 0.068 0.000 0.068 0.000 {built-in method _operator.gt}
832731 0.136 0.000 0.136 0.000 {built-in method _operator.lt}
1 0.000 0.000 0.000 0.000 {built-in method _signal.signal}
532481 0.090 0.000 0.090 0.000 {built-in method _thread.get_ident}
2 0.000 0.000 0.000 0.000 {built-in method _warnings.warn}
1 0.000 0.000 28.612 28.612 {built-in method builtins.exec}
200001 0.066 0.000 0.066 0.000 {built-in method builtins.getattr}
14 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
755255 0.113 0.000 0.113 0.000 {built-in method builtins.hash}
532473 0.092 0.000 0.092 0.000 {built-in method builtins.id}
466451 0.166 0.000 0.166 0.000 {built-in method builtins.isinstance}
532473 0.083 0.000 0.083 0.000 {built-in method builtins.issubclass}
3750044 0.325 0.000 0.325 0.000 {built-in method builtins.len}
177687 0.091 0.000 0.091 0.000 {built-in method builtins.max}
196 0.000 0.000 0.000 0.000 {built-in method builtins.min}
354982 0.142 0.000 0.142 0.000 {built-in method numpy.core.multiarray.array}
111456 0.095 0.000 0.095 0.000 {built-in method numpy.core.multiarray.where}
14 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getpid}
4 0.000 0.000 0.000 0.000 {built-in method sys._getframe}
5 0.000 0.000 0.000 0.000 {built-in method time.time}
8 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
532473 0.089 0.000 0.089 0.000 {method 'add' of 'set' objects}
875 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
16345 0.110 0.000 0.110 0.000 {method 'choice' of 'mtrand.RandomState' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
532473 0.097 0.000 0.097 0.000 {method 'discard' of 'set' objects}
188618 0.633 0.000 0.633 0.000 {method 'exponential' of 'mtrand.RandomState' objects}
4 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'flush' of '_io.TextIOWrapper' objects}
8 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
354982 0.066 0.000 0.066 0.000 {method 'item' of 'numpy.ndarray' objects}
200001 0.196 0.000 0.196 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
11887 0.052 0.000 0.052 0.000 {method 'randint' of 'mtrand.RandomState' objects}
167895 0.157 0.000 0.157 0.000 {method 'random_sample' of 'mtrand.RandomState' objects}
177491 0.251 0.000 0.251 0.000 {method 'ravel' of 'numpy.ndarray' objects}
643928 2.511 0.000 2.511 0.000 {method 'reduce' of 'numpy.ufunc' objects}
8 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
2451118 0.328 0.000 0.328 0.000 {method 'rstrip' of 'str' objects}
8 0.000 0.000 0.000 0.000 {method 'write' of '_io.TextIOWrapper' objects}

numpy was installed through pip on the laptop, and through the Fedora repositories on the desktop. Removing the package and installing it through pip removed arrayprint (fillFormat) from the profiling results and the runtime is now very much the same (which is still a bit weird). It's also strange that the other arrayprint functions are still being called, with 10 seconds of cumulative time.

Python Profiling - discrepancy between inlining and calling function

I have this code:
def getNeighbors(cfg, cand, adj):
c_nocfg = np.setdiff1d(cand, cfg)
deg = np.sum(adj[np.ix_(cfg, c_nocfg)], axis=0)
degidx = np.where(deg) > 0
nbs = c_nocfg[degidx]
deg = deg[degidx]
return nbs, deg
which retrieves neighbors (and their degree in a subgraph spanned by nodes in cfg) from an adjacency matrix.
Inlining the code gives reasonable performance (~10kx10k adjacency matrix as boolean array, 10k candidates in cand, subgraph cfg spanning 500 nodes): 0.02s
However, calling the function getNeighbors results in an overhead of roughly 0.5s.
Mocking
deg = np.sum(adj[np.ix_(cfg, c_nocfg)], axis=0)
with
deg = np.random.randint(500, size=c_nocfg.shape[0])
drives down the runtime of the function call to 0.002s.
Could someone explain me what causes the enormous overhead in my case - after all, the sum-operation itself is not all too costly.
Profiling output
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.466 0.466 0.488 0.488 /home/x/x/x/utils/benchmarks.py:15(getNeighbors)
1 0.000 0.000 0.019 0.019 /usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py:1621(sum)
1 0.000 0.000 0.019 0.019 /usr/local/lib/python2.7/dist-packages/numpy/core/_methods.py:23(_sum)
1 0.019 0.019 0.019 0.019 {method 'reduce' of 'numpy.ufunc' objects}
1 0.000 0.000 0.002 0.002 /usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py:410(setdiff1d)
1 0.000 0.000 0.001 0.001 /usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py:284(in1d)
2 0.001 0.000 0.001 0.000 {method 'argsort' of 'numpy.ndarray' objects}
2 0.000 0.000 0.001 0.000 /usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py:93(unique)
2 0.000 0.000 0.000 0.000 {method 'sort' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.where}
4 0.000 0.000 0.000 0.000 {numpy.core.multiarray.concatenate}
2 0.000 0.000 0.000 0.000 {method 'flatten' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 /usr/local/lib/python2.7/dist-packages/numpy/lib/index_tricks.py:26(ix_)
5 0.000 0.000 0.000 0.000 /usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py:392(asarray)
5 0.000 0.000 0.000 0.000 {numpy.core.multiarray.array}
1 0.000 0.000 0.000 0.000 {isinstance}
2 0.000 0.000 0.000 0.000 {method 'reshape' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {range}
2 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
2 0.000 0.000 0.000 0.000 {issubclass}
2 0.000 0.000 0.000 0.000 {method 'ravel' of 'numpy.ndarray' objects}
6 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
inline version:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.019 0.019 /usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py:1621(sum)
1 0.000 0.000 0.019 0.019 /usr/local/lib/python2.7/dist-packages/numpy/core/_methods.py:23(_sum)
1 0.019 0.019 0.019 0.019 {method 'reduce' of 'numpy.ufunc' objects}
1 0.000 0.000 0.002 0.002 /usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py:410(setdiff1d)
1 0.000 0.000 0.001 0.001 /usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py:284(in1d)
2 0.001 0.000 0.001 0.000 {method 'argsort' of 'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 /usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py:93(unique)
2 0.000 0.000 0.000 0.000 {method 'sort' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.where}
4 0.000 0.000 0.000 0.000 {numpy.core.multiarray.concatenate}
2 0.000 0.000 0.000 0.000 {method 'flatten' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 /usr/local/lib/python2.7/dist-packages/numpy/lib/index_tricks.py:26(ix_)
5 0.000 0.000 0.000 0.000 /usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py:392(asarray)
5 0.000 0.000 0.000 0.000 {numpy.core.multiarray.array}
1 0.000 0.000 0.000 0.000 {isinstance}
2 0.000 0.000 0.000 0.000 {method 'reshape' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {range}
2 0.000 0.000 0.000 0.000 {method 'ravel' of 'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
2 0.000 0.000 0.000 0.000 {issubclass}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
6 0.000 0.000 0.000 0.000 {len}
sample data for testing:
np.random.seed(0)
adj = np.zeros(10000*10000, dtype=np.bool)
adj[np.random.randint(low=0, high=10000*10000+1, size=100000)] = True
adj = adj.reshape((10000, 10000))
cand = np.arange(adj.shape[0])
cfgs = np.random.choice(cand, size=500)

Profile Python cProfile vs unix time

I am profiling a python code ; why does it spend more time in the user space ?
user#terminal$ time python main.py
1964 function calls in 0.003 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.003 0.003 :1()
1 0.000 0.000 0.000 0.000 ConfigParser.py:218(init)
1 0.000 0.000 0.001 0.001 ConfigParser.py:266(read)
30 0.000 0.000 0.000 0.000 ConfigParser.py:354(optionxform)
1 0.000 0.000 0.000 0.000 ConfigParser.py:434(_read)
15 0.000 0.000 0.000 0.000 ConfigParser.py:515(get)
15 0.000 0.000 0.000 0.000 ConfigParser.py:611(_interpolate)
15 0.000 0.000 0.000 0.000 ConfigParser.py:619(_interpolate_some)
1 0.000 0.000 0.000 0.000 config.py:32(read_config_data)
1 0.000 0.000 0.001 0.001 config.py:9(init)
6 0.000 0.000 0.000 0.000 entity.py:108(add_to_filter)
1 0.000 0.000 0.002 0.002 entity.py:24(init)
1 0.001 0.001 0.002 0.002 entity.py:39(create_inverted_index)
493 0.000 0.000 0.001 0.000 entity.py:80(beautify)
1 0.000 0.000 0.000 0.000 entity.py:84(create_bucket_lookup)
1 0.000 0.000 0.000 0.000 main.py:15()
2 0.000 0.000 0.000 0.000 main.py:18()
1 0.000 0.000 0.003 0.003 main.py:23(main)
1 0.000 0.000 0.000 0.000 main.py:9(get_bag_of_words)
19 0.000 0.000 0.000 0.000 {built-in method group}
34 0.000 0.000 0.000 0.000 {built-in method match}
1 0.000 0.000 0.000 0.000 {isinstance}
2 0.000 0.000 0.000 0.000 {len}
28 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
15 0.000 0.000 0.000 0.000 {method 'copy' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
15 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
19 0.000 0.000 0.000 0.000 {method 'isspace' of 'str' objects}
24 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects}
49 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
20 0.000 0.000 0.000 0.000 {method 'readline' of 'file' objects}
6 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects}
24 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
47 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
9 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
1030 0.000 0.000 0.000 0.000 {method 'strip' of 'str' objects}
15 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'write' of 'file' objects}
10 0.000 0.000 0.000 0.000 {open}
2 0.000 0.000 0.000 0.000 {range}
3 0.000 0.000 0.000 0.000 {reduce}
Done
real 0m0.063s
user 0m0.050s
sys 0m0.010s
While the cProfile says it took only 0.003 seconds, why is unix (sys) time saying it runs in 0.01 seconds?

time(1) is measuring the execution time of the whole process, whereas the profiler excludes Python interpreter startup time, bytecode compilation time, etc.