I am not a computer scientist and I have been having a hard time trying to make my agent-based model reasonably fast (Implementing Pool on a for loop with a lot of inputs, Multiprocessing pool example (parallel) is slower than sequential. Trying to understand pool in python, Returning lists of instances makes no difference? And what about performance?)
Everybody keeps telling me to profile. I gave it another shot, but I have no idea how to interpret the results (using timeit I know which function is slower, but I could not use Pool or #jit).
So given the results below, how should I proceed?
Thanks a bunch.
98008290 function calls (96393650 primitive calls) in 568.193 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <decorator-gen-0>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-10>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-11>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-1>:1(<module>)
6 0.000 0.000 0.006 0.001 <decorator-gen-1>:1(non_reentrant)
1 0.000 0.000 0.000 0.000 <decorator-gen-2>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-3>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-4>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-5>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-6>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-7>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-8>:1(<module>)
3 0.000 0.000 0.002 0.001 <decorator-gen-8>:1(_held_figure)
1 0.000 0.000 0.000 0.000 <decorator-gen-9>:1(<module>)
1416 0.003 0.000 0.003 0.000 <frozen importlib._bootstrap>:1000(__init__)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1003(module_repr)
1245 0.034 0.000 0.166 0.000 <frozen importlib._bootstrap>:1019(init_module_attrs)
1245 0.011 0.000 0.209 0.000 <frozen importlib._bootstrap>:1099(create)
1245/32 0.008 0.000 23.613 0.738 <frozen importlib._bootstrap>:1122(_exec)
169/105 0.003 0.000 4.104 0.039 <frozen importlib._bootstrap>:1156(_load_backward_compatible)
1414/32 0.034 0.000 23.620 0.738 <frozen importlib._bootstrap>:1186(_load_unlocked)
15 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap>:1223(_fix_up_module)
1650 0.002 0.000 0.006 0.000 <frozen importlib._bootstrap>:1266(find_spec)
19 0.000 0.000 0.006 0.000 <frozen importlib._bootstrap>:1287(load_module)
1245 0.019 0.000 0.019 0.000 <frozen importlib._bootstrap>:129(_new_module)
19 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1311(is_package)
1631 0.004 0.000 0.009 0.000 <frozen importlib._bootstrap>:1336(find_spec)
1631 0.022 0.000 0.254 0.000 <frozen importlib._bootstrap>:1401(_open_registry)
1631 0.034 0.000 0.307 0.000 <frozen importlib._bootstrap>:1408(_search_registry)
139 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:141(__init__)
1631 0.006 0.000 0.314 0.000 <frozen importlib._bootstrap>:1423(find_spec)
139 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:144(__enter__)
1245/32 0.022 0.000 23.612 0.738 <frozen importlib._bootstrap>:1465(exec_module)
139 0.001 0.000 0.003 0.000 <frozen importlib._bootstrap>:147(__exit__)
556 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:148(<genexpr>)
1245 0.055 0.000 2.437 0.002 <frozen importlib._bootstrap>:1534(get_code)
1245 0.008 0.000 0.008 0.000 <frozen importlib._bootstrap>:1591(__init__)
1245 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1616(get_filename)
1245 0.941 0.001 1.037 0.001 <frozen importlib._bootstrap>:1621(get_data)
1245 0.011 0.000 0.384 0.000 <frozen importlib._bootstrap>:1631(path_stats)
120 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1697(__init__)
120/92 0.004 0.000 4.087 0.044 <frozen importlib._bootstrap>:1708(load_module)
1888 0.017 0.000 0.066 0.000 <frozen importlib._bootstrap>:172(__init__)
120 0.001 0.000 0.006 0.000 <frozen importlib._bootstrap>:1726(is_package)
240 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1729(<genexpr>)
181 0.058 0.000 0.128 0.001 <frozen importlib._bootstrap>:1853(_path_hooks)
2614 0.012 0.000 0.142 0.000 <frozen importlib._bootstrap>:1870(_path_importer_cache)
135 0.002 0.000 0.461 0.003 <frozen importlib._bootstrap>:1887(_legacy_get_spec)
1631 0.035 0.000 2.252 0.001 <frozen importlib._bootstrap>:1902(_get_spec)
2857 0.033 0.000 0.035 0.000 <frozen importlib._bootstrap>:192(acquire)
1631 0.009 0.000 2.261 0.001 <frozen importlib._bootstrap>:1934(find_spec)
179 0.006 0.000 0.012 0.000 <frozen importlib._bootstrap>:1979(__init__)
1253 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1985(<genexpr>)
1365 0.016 0.000 0.055 0.000 <frozen importlib._bootstrap>:2011(_get_spec)
2335 0.118 0.000 1.603 0.001 <frozen importlib._bootstrap>:2016(find_spec)
178 0.016 0.000 0.100 0.001 <frozen importlib._bootstrap>:2061(_fill_cache)
178 0.003 0.000 0.004 0.000 <frozen importlib._bootstrap>:2090(<setcomp>)
179 0.002 0.000 0.071 0.000 <frozen importlib._bootstrap>:2102(path_hook_for_FileFinder)
7719 0.012 0.000 0.017 0.000 <frozen importlib._bootstrap>:2120(__enter__)
7719 0.013 0.000 0.020 0.000 <frozen importlib._bootstrap>:2124(__exit__)
1176 0.007 0.000 0.019 0.000 <frozen importlib._bootstrap>:2138(_find_spec_legacy)
1650/1648 0.097 0.000 2.739 0.002 <frozen importlib._bootstrap>:2147(_find_spec)
2857 0.031 0.000 0.033 0.000 <frozen importlib._bootstrap>:217(release)
216 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:2187(_sanity_check)
1730/92 0.045 0.000 23.736 0.258 <frozen importlib._bootstrap>:2207(_find_and_load_unlocked)
1730/92 0.057 0.000 23.753 0.258 <frozen importlib._bootstrap>:2234(_find_and_load)
216/214 0.004 0.000 0.051 0.000 <frozen importlib._bootstrap>:2240(_gcd_import)
338220/334774 1.206 0.000 10.385 0.000 <frozen importlib._bootstrap>:2264(_handle_fromlist)
108/107 0.001 0.000 0.052 0.000 <frozen importlib._bootstrap>:2322(__import__)
1730 0.006 0.000 0.006 0.000 <frozen importlib._bootstrap>:257(__init__)
1730 0.017 0.000 0.134 0.000 <frozen importlib._bootstrap>:261(__enter__)
1730 0.007 0.000 0.033 0.000 <frozen importlib._bootstrap>:268(__exit__)
2857 0.040 0.000 0.106 0.000 <frozen importlib._bootstrap>:274(_get_module_lock)
1888 0.003 0.000 0.003 0.000 <frozen importlib._bootstrap>:288(cb)
1127 0.011 0.000 0.044 0.000 <frozen importlib._bootstrap>:293(_lock_unlock_module)
2335 0.007 0.000 0.007 0.000 <frozen importlib._bootstrap>:30(_relax_case)
1804/32 0.007 0.000 23.649 0.739 <frozen importlib._bootstrap>:313(_call_with_frames_removed)
2490 0.035 0.000 0.140 0.000 <frozen importlib._bootstrap>:437(cache_from_source)
2490 0.008 0.000 0.014 0.000 <frozen importlib._bootstrap>:45(_r_long)
9163 0.047 0.000 0.132 0.000 <frozen importlib._bootstrap>:50(_path_join)
9990 0.024 0.000 0.024 0.000 <frozen importlib._bootstrap>:518(_verbose_message)
9163 0.050 0.000 0.071 0.000 <frozen importlib._bootstrap>:52(<listcomp>)
1365/1155 0.007 0.000 4.094 0.004 <frozen importlib._bootstrap>:534(_check_name_wrapper)
38 0.000 0.000 0.006 0.000 <frozen importlib._bootstrap>:546(_requires_builtin_wrapper)
2618 0.042 0.000 0.063 0.000 <frozen importlib._bootstrap>:56(_path_split)
1245 0.055 0.000 0.070 0.000 <frozen importlib._bootstrap>:599(_validate_bytecode_header)
1245 0.017 0.000 0.823 0.001 <frozen importlib._bootstrap>:654(_compile_bytecode)
5307 0.020 0.000 1.598 0.000 <frozen importlib._bootstrap>:68(_path_stat)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:690(_module_repr)
1245 0.003 0.000 0.003 0.000 <frozen importlib._bootstrap>:728(__init__)
1245 0.004 0.000 0.004 0.000 <frozen importlib._bootstrap>:732(__enter__)
1245 0.017 0.000 0.032 0.000 <frozen importlib._bootstrap>:739(__exit__)
4980 0.005 0.000 0.005 0.000 <frozen importlib._bootstrap>:742(<genexpr>)
1727 0.014 0.000 0.597 0.000 <frozen importlib._bootstrap>:78(_path_is_mode_type)
1801 0.016 0.000 0.016 0.000 <frozen importlib._bootstrap>:790(__init__)
2490 0.017 0.000 0.104 0.000 <frozen importlib._bootstrap>:824(cached)
1245 0.008 0.000 0.010 0.000 <frozen importlib._bootstrap>:842(parent)
1247 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:850(has_location)
49/48 0.001 0.000 0.464 0.010 <frozen importlib._bootstrap>:859(spec_from_loader)
1547 0.006 0.000 0.547 0.000 <frozen importlib._bootstrap>:87(_path_isfile)
1395 0.020 0.000 0.487 0.000 <frozen importlib._bootstrap>:884(spec_from_file_location)
180 0.001 0.000 0.057 0.000 <frozen importlib._bootstrap>:92(_path_isdir)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 <string>:1(error)
1 0.000 0.000 0.000 0.000 <string>:11(com_error)
313 0.001 0.000 0.001 0.000 <string>:12(__new__)
1 0.000 0.000 0.000 0.000 <string>:2(<module>)
1 0.000 0.000 0.000 0.000 <string>:2(__init__)
4 0.000 0.000 0.000 0.000 <string>:2(_parse_args)
1 0.000 0.000 0.000 0.000 <string>:5(AndersonResult)
1 0.000 0.000 0.000 0.000 <string>:5(Anderson_ksampResult)
1 0.000 0.000 0.000 0.000 <string>:5(AnsariResult)
1 0.000 0.000 0.000 0.000 <string>:5(ArgInfo)
1 0.000 0.000 0.000 0.000 <string>:5(ArgSpec)
1 0.000 0.000 0.000 0.000 <string>:5(Arguments)
1 0.000 0.000 0.000 0.000 <string>:5(ArrayFormula)
1 0.000 0.000 0.000 0.000 <string>:5(Attribute)
1 0.000 0.000 0.000 0.000 <string>:5(BartlettResult)
1 0.000 0.000 0.000 0.000 <string>:5(BinnedStatistic2dResult)
1 0.000 0.000 0.000 0.000 <string>:5(FlignerResult)
1 0.000 0.000 0.000 0.000 <string>:5(Formula)
1 0.002 0.002 0.002 0.002 <string>:5(FriedmanchisquareResult)
1 0.000 0.000 0.000 0.000 <string>:5(FullArgSpec)
1 0.000 0.000 0.000 0.000 <string>:5(FunctionAttributes)
1 0.000 0.000 0.000 0.000 <string>:5(HistogramResult)
19424 2.346 0.000 2.346 0.000 {pandas.algos.take_2d_axis1_object_object}
82904 0.318 0.000 0.318 0.000 {pandas.index.get_value_at}
199 0.051 0.000 0.051 0.000 {pandas.lib.clean_index_list}
57 0.002 0.000 0.002 0.000 {pandas.lib.get_blkno_indexers}
2918 0.095 0.000 0.095 0.000 {pandas.lib.infer_dtype}
38 0.001 0.000 0.001 0.000 {pandas.lib.is_bool_array}
76 0.000 0.000 0.000 0.000 {pandas.lib.is_bool}
166001 0.169 0.000 0.169 0.000 {pandas.lib.is_float}
249112 0.215 0.000 0.215 0.000 {pandas.lib.is_integer}
20 0.000 0.000 0.000 0.000 {pandas.lib.is_timedelta_array}
38 0.002 0.000 0.002 0.000 {pandas.lib.isnullobj}
414822 1.023 0.000 1.023 0.000 {pandas.lib.isscalar}
11 0.054 0.005 0.054 0.005 {pandas.lib.maybe_convert_numeric}
56 0.006 0.000 0.006 0.000 {pandas.lib.maybe_convert_objects}
3 0.000 0.000 0.018 0.006 {pandas.lib.reduce}
65 0.009 0.000 0.105 0.002 {pandas.lib.scalar_compare}
28 0.001 0.000 0.001 0.000 {pandas.lib.to_object_array_tuples}
415734 2.533 0.000 4.116 0.000 {pandas.lib.values_from_object}
20 0.000 0.000 0.000 0.000 {pandas.tslib.is_timestamp_array}
48 0.001 0.000 0.001 0.000 {reduce.nansum}
I would also recommend flamegraphs for interpreting profiling results. It works especially well with PyFlame, which is a PTracing profiler (it has minimal overhead).
A flamegraph looks like this:
You might want to use some of the tools available to analyze/visualize the cProfile output like pstats or RunShakeRun
Related
Currently I have this result:
That's not bad (I guess), but I'm wondering if I can speed things up a little bit.
I've looked at penultimate query and don't really know how to speed it up, I guess I should get rid off join, but don't know how:
I'm already using prefetch_related in my viewset, my viewset is:
class GameViewSet(viewsets.ModelViewSet):
queryset = Game.objects.prefetch_related(
"timestamp",
"fighters",
"score",
"coefs",
"rounds",
"rounds_view",
"rounds_view_f",
"finishes",
"rounds_time",
"round_time",
"time_coef",
"totals",
).all()
serializer_class = GameSerializer
permission_classes = [AllowAny]
pagination_class = StandardResultsSetPagination
#silk_profile(name="Get Games")
def list(self, request):
qs = self.get_queryset().order_by("-timestamp__ts")
page = self.paginate_queryset(qs)
if page is not None:
serializer = GameSerializer(page, many=True)
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(qs, many=True)
return Response(serializer.data)
Join is happening because I'm ordering on a related field?
My models looks like:
class Game(models.Model):
id = models.AutoField(primary_key=True)
...
class Timestamp(models.Model):
id = models.AutoField(primary_key=True)
game = models.ForeignKey(Game, related_name="timestamp", on_delete=models.CASCADE)
ts = models.DateTimeField(db_index=True)
time_of_day = models.TimeField()
And my serializers:
class TimestampSerializer(serializers.Serializer):
ts = serializers.DateTimeField(read_only=True)
time_of_day = serializers.TimeField(read_only=True)
class GameSerializer(serializers.Serializer):
id = serializers.IntegerField(read_only=True)
timestamp = TimestampSerializer(many=True)
fighters = FighterSerializer(many=True)
score = ScoreSerializer(many=True)
coefs = CoefsSerializer(many=True)
rounds = RoundsSerializer(many=True)
rounds_view = RoundsViewSerializer(many=True)
rounds_view_f = RoundsViewFinishSerializer(many=True)
finishes = FinishesSerializer(many=True)
rounds_time = RoundTimesSerializer(many=True)
round_time = RoundTimeSerializer(many=True)
time_coef = TimeCoefsSerializer(many=True)
totals = TotalsSerializer(many=True)
Also results of profling:
166039 function calls (159016 primitive calls) in 3.226 seconds
Ordered by: internal time
List reduced from 677 to 100 due to restriction <100>
ncalls tottime percall cumtime percall filename:lineno(function)
20959/20958 0.206 0.000 0.283 0.000 {built-in method builtins.isinstance}
2700 0.123 0.000 0.359 0.000 fields.py:62(is_simple_callable)
390/30 0.113 0.000 1.211 0.040 serializers.py:493(to_representation)
8943/8473 0.098 0.000 0.307 0.000 {built-in method builtins.getattr}
2700 0.096 0.000 0.650 0.000 fields.py:85(get_attribute)
14 0.068 0.005 0.130 0.009 traceback.py:388(format)
7653 0.065 0.000 0.065 0.000 {method 'append' of 'list' objects}
28 0.064 0.002 0.072 0.003 {method 'execute' of 'psycopg2.extensions.cursor' objects}
390 0.062 0.000 0.153 0.000 base.py:406(__init__)
3090 0.060 0.000 0.257 0.000 serializers.py:359(_readable_fields)
3090 0.059 0.000 0.093 0.000 _collections_abc.py:760(__iter__)
6390 0.055 0.000 0.055 0.000 {built-in method builtins.hasattr}
1440 0.054 0.000 0.112 0.000 related.py:652(get_instance_value_for_fields)
2749 0.052 0.000 0.078 0.000 abc.py:96(__instancecheck__)
388 0.052 0.000 0.107 0.000 query.py:303(clone)
2700 0.049 0.000 0.072 0.000 inspect.py:158(isfunction)
2700 0.048 0.000 0.699 0.000 fields.py:451(get_attribute)
2702 0.048 0.000 0.071 0.000 inspect.py:80(ismethod)
2701 0.047 0.000 0.070 0.000 inspect.py:285(isbuiltin)
14 0.047 0.003 0.189 0.014 traceback.py:321(extract)
3786/3426 0.043 0.000 0.123 0.000 {built-in method builtins.setattr}
4445/247 0.038 0.000 1.936 0.008 {built-in method builtins.len}
360 0.035 0.000 0.374 0.001 related_descriptors.py:575(_apply_rel_filters)
2247 0.034 0.000 0.088 0.000 traceback.py:285(line)
3203 0.029 0.000 0.029 0.000 {method 'copy' of 'dict' objects}
12 0.028 0.002 1.836 0.153 query.py:1831(prefetch_one_level)
2749 0.026 0.000 0.026 0.000 {built-in method _abc._abc_instancecheck}
2700 0.024 0.000 0.024 0.000 serializer_helpers.py:154(__getitem__)
2780 0.024 0.000 0.024 0.000 {method 'get' of 'dict' objects}
360 0.023 0.000 0.069 0.000 related_lookups.py:26(get_normalized_value)
720 0.022 0.000 0.458 0.001 related_descriptors.py:615(get_queryset)
744 0.022 0.000 0.087 0.000 related_descriptors.py:523(__get__)
360 0.022 0.000 0.057 0.000 related_descriptors.py:203(__set__)
470 0.022 0.000 0.081 0.000 local.py:46(_get_context_id)
749 0.020 0.000 0.048 0.000 linecache.py:15(getline)
720 0.019 0.000 0.031 0.000 lookups.py:252(resolve_expression_parameter)
361/1 0.018 0.000 1.211 1.211 serializers.py:655(to_representation)
296/14 0.018 0.000 0.084 0.006 copy.py:128(deepcopy)
470 0.018 0.000 0.106 0.000 local.py:82(_get_storage)
732 0.017 0.000 0.043 0.000 related_descriptors.py:560(__init__)
720 0.017 0.000 0.040 0.000 related_descriptors.py:76(__set__)
1185/1151 0.017 0.000 0.032 0.000 {method 'join' of 'str' objects}
1501 0.017 0.000 0.017 0.000 {method 'format' of 'str' objects}
732 0.016 0.000 0.026 0.000 manager.py:26(__init__)
372 0.016 0.000 0.414 0.001 query.py:951(_filter_or_exclude)
14 0.016 0.001 0.028 0.002 traceback.py:369(from_list)
759 0.015 0.000 0.040 0.000 query.py:178(__init__)
387 0.015 0.000 0.143 0.000 query.py:1308(_clone)
732 0.015 0.000 0.022 0.000 manager.py:20(__new__)
1710 0.015 0.000 0.015 0.000 {built-in method __new__ of type object at 0x7fa87d9ad940}
720 0.015 0.000 0.034 0.000 __init__.py:1818(get_prep_value)
470 0.014 0.000 0.033 0.000 sync.py:469(get_current_task)
390 0.014 0.000 0.174 0.000 base.py:507(from_db)
1365 0.014 0.000 0.014 0.000 {method 'update' of 'dict' objects}
749 0.013 0.000 0.022 0.000 linecache.py:37(getlines)
744 0.013 0.000 0.044 0.000 lookups.py:266()
12 0.013 0.001 0.054 0.004 lookups.py:230(get_prep_lookup)
749 0.013 0.000 0.020 0.000 linecache.py:147(lazycache)
720 0.013 0.000 0.066 0.000 related.py:646(get_local_related_value)
720 0.013 0.000 0.071 0.000 related.py:649(get_foreign_related_value)
720 0.013 0.000 0.019 0.000 __init__.py:824(get_prep_value)
1506 0.013 0.000 0.013 0.000 {method 'strip' of 'str' objects}
372/12 0.013 0.000 0.026 0.002 query.py:1088(resolve_lookup_value)
638 0.013 0.000 0.018 0.000 threading.py:1306(current_thread)
732 0.013 0.000 0.021 0.000 reverse_related.py:200(get_cache_name)
720 0.013 0.000 0.018 0.000 base.py:573(_get_pk_val)
360 0.012 0.000 0.021 0.000 __init__.py:543(__hash__)
470 0.011 0.000 0.117 0.000 local.py:101(__getattr__)
28 0.011 0.000 0.089 0.003 compiler.py:199(get_select)
385 0.011 0.000 0.857 0.002 query.py:265(__iter__)
320 0.011 0.000 0.019 0.000 __init__.py:515(__eq__)
387 0.011 0.000 0.120 0.000 query.py:354(chain)
780 0.011 0.000 0.021 0.000 dispatcher.py:159(send)
360 0.010 0.000 0.031 0.000 related.py:976(get_prep_value)
387 0.010 0.000 0.157 0.000 query.py:1296(_chain)
372 0.010 0.000 0.014 0.000 query.py:151(__init__)
403 0.010 0.000 0.914 0.002 query.py:45(__iter__)
1204 0.010 0.000 0.010 0.000 {built-in method builtins.iter}
360 0.010 0.000 0.016 0.000 :1017(_handle_fromlist)
372 0.010 0.000 0.434 0.001 query.py:935(filter)
360 0.010 0.000 0.023 0.000 query.py:1124(check_query_object_type)
360 0.009 0.000 0.017 0.000 mixins.py:21(is_cached)
1152 0.009 0.000 0.009 0.000 query.py:194(query)
104 0.009 0.000 0.017 0.000 fields.py:323(__init__)
732 0.009 0.000 0.009 0.000 manager.py:120(_set_creation_counter)
470 0.009 0.000 0.013 0.000 :389(parent)
470 0.009 0.000 0.014 0.000 tasks.py:34(current_task)
459 0.009 0.000 0.013 0.000 deconstruct.py:14(__new__)
870 0.009 0.000 0.009 0.000 fields.py:810(to_representation)
322 0.009 0.000 0.019 0.000 linecache.py:53(checkcache)
318/266 0.009 0.000 0.145 0.001 compiler.py:434(compile)
763 0.008 0.000 0.008 0.000 traceback.py:292(walk_stack)
494 0.008 0.000 0.014 0.000 compiler.py:417(quote_name_unless_alias)
396 0.008 0.000 0.019 0.000 related.py:710(get_path_info)
374 0.008 0.000 0.011 0.000 utils.py:237(_route_db)
796 0.008 0.000 0.008 0.000 tree.py:21(__init__)
322 0.008 0.000 0.008 0.000 {built-in method posix.stat}
413/365 0.008 0.000 1.938 0.005 query.py:1322(_fetch_all)
12 0.008 0.001 1.244 0.104 related_descriptors.py:622(get_prefetch_queryset)
732 0.008 0.000 0.008 0.000 reverse_related.py:180(get_accessor_name)
And visual representation:
So, my questions is, how can I speed it up?
On the database side, you could set up a Materialized View for your query and trigger an update anytime a new timestamp is added (or whatever is happening in your application that requires a refresh). That way the results are pre-calculated for your lookup. However, I suppose there could be edge cases where when you update and lookup at the same time, you end up with a pre-updated result? It'd be a trade off and only you know whether that'd be worth it...
In any case, I did not come up with that, check out e.g. https://hashrocket.com/blog/posts/materialized-view-strategies-using-postgresql
I need to display a complex Table and decided to use stacked QTableWidgets. With an increasing number of rows the program needs a lot of time for creating all the widgets and almost the same time for displaying.
The maintable looks like this: MainTable
The stacked TableWidget in the table:
StackedTables
if the cell contains data, there is at least one TableWidget in one cell of the MainTable and in the worst case there are 2 more TableWidgets in that one. That means I could have 3 TableWidgets in one cell.
Time measurement with cProfile and time.time for 80rows (with 48 of the complex cells for each row):
complete update time: 15s (manually stopped)
time to create the table: 7.548534870147705s (time.time over complete function)
display time: 7.5s (complete update time - function time)
rows: 80
63600 function calls in 7.462 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1896 2.455 0.001 3.882 0.002 DigitalePlanungstafel.py:6054(grundWidgetErstellen) -- (create table in cell)
3936 2.027 0.001 2.027 0.001 {built-in method setCellWidget}
2535 1.306 0.001 1.306 0.001 {built-in method setColumnCount}
630 0.770 0.001 1.183 0.002 DigitalePlanungstafel.py:6035(obenWidgetErstellen) -- (create table in table in cell)
2607 0.674 0.000 0.674 0.000 {built-in method setRowCount}
2528 0.059 0.000 0.059 0.000 {built-in method horizontalHeader}
2526 0.021 0.000 0.021 0.000 {built-in method verticalHeader}
163 0.019 0.000 0.019 0.000 {method 'execute' of 'sqlite3.Cursor' objects}
2526 0.016 0.000 0.016 0.000 {built-in method setFrameShape}
1410 0.014 0.000 0.014 0.000 {built-in method setStyleSheet}
4502 0.013 0.000 0.013 0.000 {built-in method setRowHeight}
2526 0.009 0.000 0.009 0.000 {built-in method setFixedSize}
2546 0.009 0.000 0.009 0.000 {built-in method setColumnWidth}
5052 0.009 0.000 0.009 0.000 {built-in method setVisible}
1329 0.007 0.000 0.007 0.000 {built-in method setItem}
2181 0.006 0.000 0.006 0.000 {built-in method cellWidget}
80 0.005 0.000 0.005 0.000 {built-in method addWidget}
2526 0.004 0.000 0.004 0.000 {built-in method setEditTriggers}
929 0.004 0.000 0.004 0.000 {built-in method setBackground}
1330 0.003 0.000 0.003 0.000 {method 'format' of 'str' objects}
414 0.003 0.000 0.003 0.000 {built-in method _pickle.loads}
336 0.003 0.000 0.003 0.000 {method 'strftime' of 'datetime.date' objects}
2526 0.003 0.000 0.003 0.000 {built-in method setHorizontalScrollBarPolicy}
1410 0.002 0.000 0.002 0.000 {built-in method setFixedHeight}
1377 0.002 0.000 0.002 0.000 {built-in method setTextAlignment}
83 0.002 0.000 0.004 0.000 _strptime.py:321(_strptime)
2 0.002 0.001 0.002 0.001 {built-in method setSortingEnabled}
2526 0.001 0.000 0.001 0.000 {built-in method setShowGrid}
1570 0.001 0.000 0.001 0.000 {built-in method rowHeight}
2526 0.001 0.000 0.001 0.000 {built-in method setSelectionMode}
163 0.001 0.000 0.001 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
240 0.001 0.000 0.001 0.000 DigitalePlanungstafel.py:7494(__init__)
2526 0.001 0.000 0.001 0.000 {built-in method setVerticalScrollBarPolicy}
1 0.001 0.001 0.001 0.001 {built-in method sortByColumn}
80 0.001 0.000 0.001 0.000 {built-in method setLayout}
83 0.001 0.000 0.001 0.000 {built-in method _locale.setlocale}
1 0.001 0.001 0.001 0.001 {built-in method _sqlite3.connect}
89 0.000 0.000 0.000 0.000 {built-in method setForeground}
83 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
83 0.000 0.000 0.004 0.000 _strptime.py:562(_strptime_datetime)
48 0.000 0.000 0.000 0.000 {built-in method setHorizontalHeaderItem}
80 0.000 0.000 0.000 0.000 {built-in method setContentsMargins}
83 0.000 0.000 0.005 0.000 {built-in method strptime}
83 0.000 0.000 0.000 0.000 locale.py:379(normalize)
88 0.000 0.000 0.000 0.000 {built-in method setFont}
160 0.000 0.000 0.000 0.000 {built-in method setData}
83 0.000 0.000 0.000 0.000 {method 'groupdict' of '_sre.SRE_Match' objects}
80 0.000 0.000 0.000 0.000 {built-in method setAlignment}
83 0.000 0.000 0.001 0.000 _strptime.py:29(_getlang)
83 0.000 0.000 0.001 0.000 locale.py:565(getlocale)
83 0.000 0.000 0.001 0.000 locale.py:462(_parse_localename)
80 0.000 0.000 0.000 0.000 {built-in method setUnderline}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
249 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of 'sqlite3.Connection' objects}
160 0.000 0.000 0.000 0.000 {built-in method item}
742 0.000 0.000 0.000 0.000 DigitalePlanungstafel.py:7499(__lt__)
48 0.000 0.000 0.000 0.000 {built-in method today}
165 0.000 0.000 0.000 0.000 {method 'toordinal' of 'datetime.date' objects}
475 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
167 0.000 0.000 0.000 0.000 {built-in method builtins.len}
166 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
83 0.000 0.000 0.000 0.000 {method 'end' of '_sre.SRE_Match' objects}
84 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.TextIOWrapper' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.print}
83 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
47 0.000 0.000 0.000 0.000 {built-in method columnCount}
83 0.000 0.000 0.000 0.000 {method 'weekday' of 'datetime.date' objects}
96 0.000 0.000 0.000 0.000 {method 'date' of 'datetime.datetime' objects}
20 0.000 0.000 0.000 0.000 {built-in method columnWidth}
1 0.000 0.000 0.000 0.000 {built-in method _locale._getdefaultlocale}
3 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
1 0.000 0.000 0.000 0.000 _strptime.py:284(_calc_julian_from_U_or_W)
1 0.000 0.000 0.000 0.000 _bootlocale.py:11(getpreferredencoding)
1 0.000 0.000 0.000 0.000 {built-in method sortIndicatorOrder}
2 0.000 0.000 0.000 0.000 {built-in method time.time}
1 0.000 0.000 0.000 0.000 {method 'cursor' of 'sqlite3.Connection' objects}
2 0.000 0.000 0.000 0.000 {method 'index' of 'list' objects}
1 0.000 0.000 0.000 0.000 {built-in method fromordinal}
1 0.000 0.000 0.000 0.000 {built-in method sortIndicatorSection}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 codecs.py:259(__init__)
The time for calling the function and creating the Table is okay but the program needs almost the same time again until it reacts.
The Goal would be to reduce the update time by approx. 50%. I need to display around 200 rows.
Is the stacked QTableWidget the right approach - If so what do I have to do to optimize the update times?
I already thought about changing the presenatation from QTableWidget to a QGraphicsView and simply drawing the rectangles.
Or maybe a combnation, for example: Using the QTableWidget for the Header and the first columns and then merging all the complex cells and inserting a QGraphicsView but I am not sure if I am able to get the right size for the drawed cells and I do not know if the displaying time will be shorter.
What do you guys think is the right approch for a table like this?
If you need I can append the function, which is updating the Table.
EDIT:
I am using now 2 rows in the MainTable for one Block which reduced the number of stacked widgets from 2.526 to 7!
The time measurement for the same rows now looks like this:
complete update time: ~2s (manually stopped)
time to create the table: 0.572490930557251 (time.time over complete function)
display time: ~1.5s (complete update time - function time)
rows: 160
20912 function calls in 0.534 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1417 0.258 0.000 0.258 0.000 {built-in method setCellWidget}
88 0.209 0.002 0.209 0.002 {built-in method setRowCount}
163 0.017 0.000 0.017 0.000 {method 'execute' of 'sqlite3.Cursor' objects}
1410 0.010 0.000 0.010 0.000 {built-in method setStyleSheet}
80 0.005 0.000 0.005 0.000 {built-in method addWidget}
1336 0.004 0.000 0.004 0.000 {built-in method setItem}
7 0.003 0.000 0.004 0.001 DigitalePlanungstafel.py:6037(obenWidgetErstellen)
501 0.003 0.000 0.003 0.000 {built-in method _pickle.loads}
1990 0.003 0.000 0.003 0.000 {built-in method cellWidget}
336 0.002 0.000 0.002 0.000 {method 'strftime' of 'datetime.date' objects}
83 0.002 0.000 0.004 0.000 _strptime.py:321(_strptime)
1330 0.002 0.000 0.002 0.000 {method 'format' of 'str' objects}
1410 0.002 0.000 0.002 0.000 {built-in method setFixedHeight}
929 0.001 0.000 0.001 0.000 {built-in method setBackground}
1377 0.001 0.000 0.001 0.000 {built-in method setTextAlignment}
240 0.001 0.000 0.001 0.000 DigitalePlanungstafel.py:7452(__init__)
163 0.001 0.000 0.001 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
16 0.001 0.000 0.001 0.000 {built-in method setColumnCount}
770 0.001 0.000 0.001 0.000 {built-in method setSpan}
2127 0.001 0.000 0.001 0.000 {built-in method item}
1570 0.001 0.000 0.001 0.000 {built-in method rowHeight}
80 0.001 0.000 0.001 0.000 {built-in method setLayout}
83 0.000 0.000 0.000 0.000 {built-in method _locale.setlocale}
1 0.000 0.000 0.000 0.000 {built-in method _sqlite3.connect}
167 0.000 0.000 0.000 0.000 {built-in method setRowHeight}
89 0.000 0.000 0.000 0.000 {built-in method setForeground}
83 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
48 0.000 0.000 0.000 0.000 {built-in method setHorizontalHeaderItem}
83 0.000 0.000 0.004 0.000 _strptime.py:562(_strptime_datetime)
7 0.000 0.000 0.000 0.000 {built-in method takeItem}
80 0.000 0.000 0.000 0.000 {built-in method setContentsMargins}
83 0.000 0.000 0.004 0.000 {built-in method strptime}
88 0.000 0.000 0.000 0.000 {built-in method setFont}
160 0.000 0.000 0.000 0.000 {built-in method setData}
83 0.000 0.000 0.000 0.000 locale.py:379(normalize)
80 0.000 0.000 0.000 0.000 {built-in method setAlignment}
83 0.000 0.000 0.001 0.000 _strptime.py:29(_getlang)
83 0.000 0.000 0.000 0.000 {method 'groupdict' of '_sre.SRE_Match' objects}
83 0.000 0.000 0.001 0.000 locale.py:565(getlocale)
1 0.000 0.000 0.000 0.000 {method 'close' of 'sqlite3.Connection' objects}
1 0.000 0.000 0.000 0.000 {built-in method io.open}
7 0.000 0.000 0.000 0.000 {built-in method horizontalHeader}
80 0.000 0.000 0.000 0.000 {built-in method setUnderline}
249 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
83 0.000 0.000 0.000 0.000 locale.py:462(_parse_localename)
48 0.000 0.000 0.000 0.000 {built-in method today}
7 0.000 0.000 0.000 0.000 {built-in method verticalHeader}
475 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
167 0.000 0.000 0.000 0.000 {built-in method builtins.len}
165 0.000 0.000 0.000 0.000 {method 'toordinal' of 'datetime.date' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.TextIOWrapper' objects}
166 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
7 0.000 0.000 0.000 0.000 {built-in method setFrameShape}
27 0.000 0.000 0.000 0.000 {built-in method setColumnWidth}
84 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'sort' of 'list' objects}
83 0.000 0.000 0.000 0.000 {method 'end' of '_sre.SRE_Match' objects}
87 0.000 0.000 0.000 0.000 DigitalePlanungstafel.py:6059(<lambda>)
83 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.print}
7 0.000 0.000 0.000 0.000 {built-in method setFixedSize}
14 0.000 0.000 0.000 0.000 {built-in method setVisible}
83 0.000 0.000 0.000 0.000 {method 'weekday' of 'datetime.date' objects}
96 0.000 0.000 0.000 0.000 {method 'date' of 'datetime.datetime' objects}
7 0.000 0.000 0.000 0.000 {built-in method setEditTriggers}
20 0.000 0.000 0.000 0.000 {built-in method columnWidth}
7 0.000 0.000 0.000 0.000 {built-in method setHorizontalScrollBarPolicy}
1 0.000 0.000 0.000 0.000 {built-in method _locale._getdefaultlocale}
The updateing time is more than good but I have 2 problems because of the splitting.
Sorting (over the header) is not working anymore. This is caused by the merging, it messes up everything. Is there a way to freeze two associated rows before sorting?
I only want one row to be selected at a time. Cause of the splitting I need to select the two associated rows no matter which one gets selected. Not really a big deal but doesen´t really look good (see pictures).
MainTable selected row
MainTable selected row
You do not want to do any of this - instead:
Use a model for your data.
Use a viewmodel that adapts the model to the view you wish to have.
Use a QTableView to display the viewmodel.
I have a single-threaded Python 3 program that's CPU bound, the only IO is printing a couple of lines to output (no reading/writing of files).
On my desktop machine (AMD Ryzen 1700x 3.8 GHz, 16GB 3000 MHz DDR4) it performs (consistently) at 3400 episodes/second where a run takes around 60 seconds.
On my laptop (Intel i7-6600U 2.8 GHz, 16GB 2000 MHz DDR3) the performance is doubled at 7000 episodes/second, and a run coming in at just under 30 seconds.
Both machines run the same operating systems (Fedora 26) and the same python version (not built from source).
What's more, when profiling, there's a line showing
10.999 tottime, 28.814 cumtime for arrayprint.py:557(fillFormat)
but only when the code is run on the desktop. On the laptop, the particular function does not appear at all (and none of the arrayprint functions use more than 1 second tottime).
Not only is it strange that the performance differs between the machines, but no arrays or lists are ever printed to the screen, converted to strings, or saved to files during the execution of the program.
Here's the full profile for the desktop:
54499635 function calls (53787999 primitive calls) in 58.746 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
533727 0.359 0.000 0.514 0.000 <frozen importlib._bootstrap>:402(parent)
533727 0.469 0.000 0.697 0.000 <frozen importlib._bootstrap>:989(_handle_fromlist)
1 0.000 0.000 58.746 58.746 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 __init__.py:120(getLevelName)
567524 0.237 0.000 0.727 0.000 __init__.py:1284(debug)
2 0.000 0.000 0.000 0.000 __init__.py:1296(info)
2 0.000 0.000 0.000 0.000 __init__.py:1308(warning)
2 0.000 0.000 0.000 0.000 __init__.py:1320(warn)
4 0.000 0.000 0.000 0.000 __init__.py:1374(findCaller)
4 0.000 0.000 0.000 0.000 __init__.py:1404(makeRecord)
4 0.000 0.000 0.000 0.000 __init__.py:1419(_log)
4 0.000 0.000 0.000 0.000 __init__.py:1444(handle)
4 0.000 0.000 0.000 0.000 __init__.py:1498(callHandlers)
567528 0.175 0.000 0.175 0.000 __init__.py:1528(getEffectiveLevel)
567528 0.315 0.000 0.490 0.000 __init__.py:1542(isEnabledFor)
4 0.000 0.000 0.000 0.000 __init__.py:157(<lambda>)
4 0.000 0.000 0.000 0.000 __init__.py:251(__init__)
4 0.000 0.000 0.000 0.000 __init__.py:329(getMessage)
4 0.000 0.000 0.000 0.000 __init__.py:387(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:390(format)
4 0.000 0.000 0.000 0.000 __init__.py:540(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:546(formatMessage)
4 0.000 0.000 0.000 0.000 __init__.py:562(format)
8 0.000 0.000 0.000 0.000 __init__.py:703(filter)
8 0.000 0.000 0.000 0.000 __init__.py:807(acquire)
8 0.000 0.000 0.000 0.000 __init__.py:814(release)
4 0.000 0.000 0.000 0.000 __init__.py:827(format)
4 0.000 0.000 0.000 0.000 __init__.py:850(handle)
4 0.000 0.000 0.000 0.000 __init__.py:969(flush)
4 0.000 0.000 0.000 0.000 __init__.py:980(emit)
289159 0.101 0.000 1.491 0.000 _methods.py:31(_sum)
533727 0.175 0.000 1.613 0.000 _methods.py:37(_any)
177909 0.862 0.000 33.737 0.000 arrayprint.py:237(_get_formatdict)
177909 0.370 0.000 34.214 0.000 arrayprint.py:273(_get_format_function)
177909 0.686 0.000 39.971 0.000 arrayprint.py:315(_array2string)
533727/177909 0.674 0.000 40.351 0.000 arrayprint.py:340(array2string)
1224652 0.960 0.000 1.554 0.000 arrayprint.py:467(_extendLine)
177909 1.671 0.000 4.320 0.000 arrayprint.py:475(_formatArray)
533727 0.682 0.000 29.496 0.000 arrayprint.py:543(__init__)
533727 10.999 0.000 28.814 0.000 arrayprint.py:557(fillFormat)
355336 1.600 0.000 5.432 0.000 arrayprint.py:589(<listcomp>)
2416068 2.677 0.000 3.832 0.000 arrayprint.py:642(_digits)
177909 0.720 0.000 2.378 0.000 arrayprint.py:652(__init__)
1224652 1.057 0.000 1.057 0.000 arrayprint.py:665(__call__)
533727 0.147 0.000 0.147 0.000 arrayprint.py:674(__init__)
177909 0.227 0.000 0.319 0.000 arrayprint.py:702(__init__)
177909 0.415 0.000 17.986 0.000 arrayprint.py:713(__init__)
177909 0.166 0.000 0.166 0.000 arrayprint.py:730(__init__)
177909 0.046 0.000 0.046 0.000 arrayprint.py:751(__init__)
1 0.000 0.000 0.000 0.000 enum.py:265(__call__)
1 0.000 0.000 0.000 0.000 enum.py:515(__new__)
1 0.000 0.000 0.000 0.000 enum.py:544(_missing_)
177909 0.206 0.000 0.206 0.000 enum.py:552(__str__)
177909 0.269 0.000 0.475 0.000 enum.py:564(__format__)
755408 0.248 0.000 0.366 0.000 enum.py:579(__hash__)
200000 0.037 0.000 0.037 0.000 enum.py:592(name)
27976 0.005 0.000 0.005 0.000 enum.py:597(value)
200524 0.443 0.000 0.641 0.000 eventgen.py:115(_push)
200001 0.492 0.000 0.885 0.000 eventgen.py:122(pop)
200000 0.892 0.000 1.036 0.000 eventgen.py:137(ce_str)
13988 0.017 0.000 0.034 0.000 eventgen.py:15(__lt__)
99676 0.168 0.000 0.911 0.000 eventgen.py:44(event_new)
79335 0.096 0.000 0.520 0.000 eventgen.py:52(event_end)
11689 0.078 0.000 0.261 0.000 eventgen.py:61(event_new_handoff)
9824 0.014 0.000 0.098 0.000 eventgen.py:90(event_end_handoff)
77441 0.295 0.000 0.380 0.000 eventgen.py:94(reassign)
177909 0.177 0.000 0.555 0.000 fromnumeric.py:1364(ravel)
200001 0.093 0.000 0.464 0.000 fromnumeric.py:1471(nonzero)
289159 0.542 0.000 2.148 0.000 fromnumeric.py:1710(sum)
533727 0.637 0.000 2.956 0.000 fromnumeric.py:1866(any)
200001 0.120 0.000 0.372 0.000 fromnumeric.py:55(_wrapfunc)
4 0.000 0.000 0.000 0.000 genericpath.py:117(_splitext)
49 0.000 0.000 0.001 0.000 grid.py:172(neighbors1)
49 0.001 0.000 0.001 0.000 grid.py:195(neighbors2)
533727/177909 0.311 0.000 40.472 0.000 numeric.py:1927(array_str)
1067454 1.724 0.000 4.091 0.000 numeric.py:2692(seterr)
1067454 1.466 0.000 1.603 0.000 numeric.py:2792(geterr)
533727 0.299 0.000 0.422 0.000 numeric.py:3085(__init__)
533727 0.411 0.000 2.588 0.000 numeric.py:3089(__enter__)
533727 0.461 0.000 2.374 0.000 numeric.py:3094(__exit__)
177909 0.064 0.000 0.151 0.000 numeric.py:463(asarray)
711636 0.223 0.000 0.503 0.000 numeric.py:534(asanyarray)
4 0.000 0.000 0.000 0.000 posixpath.py:119(splitext)
4 0.000 0.000 0.000 0.000 posixpath.py:142(basename)
4 0.000 0.000 0.000 0.000 posixpath.py:39(_get_sep)
6 0.000 0.000 0.000 0.000 posixpath.py:50(normcase)
4 0.000 0.000 0.000 0.000 process.py:137(name)
4 0.000 0.000 0.000 0.000 process.py:35(current_process)
1 0.000 0.000 0.000 0.000 signal.py:25(_int_to_enum)
2 0.000 0.000 0.000 0.000 signal.py:35(_enum_to_int)
1 0.000 0.000 0.000 0.000 signal.py:45(signal)
99627 0.062 0.000 0.062 0.000 stats.py:38(new)
20292 0.028 0.000 0.039 0.000 stats.py:42(new_rej)
88750 0.047 0.000 0.047 0.000 stats.py:48(end)
11623 0.005 0.000 0.005 0.000 stats.py:51(hoff_new)
1799 0.001 0.000 0.002 0.000 stats.py:54(hoff_rej)
22091 0.012 0.000 0.012 0.000 stats.py:58(rej)
200000 0.234 0.000 1.513 0.000 stats.py:64(iter)
1 0.000 0.000 0.000 0.000 stats.py:69(n_iter)
1 0.000 0.000 0.000 0.000 stats.py:86(endsim)
1 0.000 0.000 0.001 0.001 strats.py:189(get_init_action)
200000 1.070 0.000 49.964 0.000 strats.py:193(get_action)
177909 1.348 0.000 1.937 0.000 strats.py:220(execute_action)
200001 4.572 0.000 47.626 0.000 strats.py:243(optimal_ch)
89158 0.071 0.000 0.958 0.000 strats.py:299(reward)
89158 0.018 0.000 0.018 0.000 strats.py:308(discount)
1242355 0.944 0.000 0.944 0.000 strats.py:333(get_qval)
89158 0.160 0.000 0.160 0.000 strats.py:336(update_qval)
1 0.000 0.000 58.746 58.746 strats.py:40(init_sim)
1 1.271 1.271 58.745 58.745 strats.py:49(_simulate)
4 0.000 0.000 0.000 0.000 threading.py:1076(name)
4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread)
227976 0.120 0.000 0.162 0.000 types.py:135(__get__)
177909 0.079 0.000 0.079 0.000 {built-in method _functools.reduce}
200001 0.192 0.000 0.222 0.000 {built-in method _heapq.heappop}
200524 0.084 0.000 0.088 0.000 {built-in method _heapq.heappush}
310143 0.064 0.000 0.064 0.000 {built-in method _operator.gt}
843054 0.152 0.000 0.152 0.000 {built-in method _operator.lt}
1 0.000 0.000 0.000 0.000 {built-in method _signal.signal}
8 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
2 0.000 0.000 0.000 0.000 {built-in method _warnings.warn}
1 0.000 0.000 58.746 58.746 {built-in method builtins.exec}
200001 0.056 0.000 0.056 0.000 {built-in method builtins.getattr}
1067468 0.228 0.000 0.228 0.000 {built-in method builtins.hasattr}
755408 0.118 0.000 0.118 0.000 {built-in method builtins.hash}
467082 0.164 0.000 0.164 0.000 {built-in method builtins.isinstance}
533727 0.107 0.000 0.107 0.000 {built-in method builtins.issubclass}
10361766 1.076 0.000 1.076 0.000 {built-in method builtins.len}
533441 0.304 0.000 0.304 0.000 {built-in method builtins.max}
533923 0.198 0.000 0.198 0.000 {built-in method builtins.min}
889545 0.368 0.000 0.368 0.000 {built-in method numpy.core.multiarray.array}
111251 0.101 0.000 0.101 0.000 {built-in method numpy.core.multiarray.where}
2134908 0.377 0.000 0.377 0.000 {built-in method numpy.core.umath.geterrobj}
1067454 0.524 0.000 0.524 0.000 {built-in method numpy.core.umath.seterrobj}
14 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getpid}
4 0.000 0.000 0.000 0.000 {built-in method sys._getframe}
5 0.000 0.000 0.000 0.000 {built-in method time.time}
8 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
533727 0.299 0.000 1.912 0.000 {method 'any' of 'numpy.ndarray' objects}
875 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
16544 0.119 0.000 0.119 0.000 {method 'choice' of 'mtrand.RandomState' objects}
533727 0.872 0.000 0.872 0.000 {method 'compress' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
188835 0.641 0.000 0.641 0.000 {method 'exponential' of 'mtrand.RandomState' objects}
4 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'flush' of '_io.TextIOWrapper' objects}
8 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
355818 0.069 0.000 0.069 0.000 {method 'item' of 'numpy.ndarray' objects}
200001 0.196 0.000 0.196 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
533727 0.123 0.000 0.123 0.000 {method 'pop' of 'dict' objects}
11689 0.053 0.000 0.053 0.000 {method 'randint' of 'mtrand.RandomState' objects}
168494 0.128 0.000 0.128 0.000 {method 'random_sample' of 'mtrand.RandomState' objects}
177909 0.232 0.000 0.232 0.000 {method 'ravel' of 'numpy.ndarray' objects}
1889376 5.023 0.000 5.023 0.000 {method 'reduce' of 'numpy.ufunc' objects}
8 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
533727 0.155 0.000 0.155 0.000 {method 'rpartition' of 'str' objects}
4865786 1.100 0.000 1.100 0.000 {method 'rstrip' of 'str' objects}
8 0.000 0.000 0.000 0.000 {method 'write' of '_io.TextIOWrapper' objects}
And here's for the laptop:
27738517 function calls (26673571 primitive calls) in 28.612 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 28.612 28.612 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 __init__.py:120(getLevelName)
566894 0.244 0.000 0.720 0.000 __init__.py:1284(debug)
2 0.000 0.000 0.000 0.000 __init__.py:1296(info)
2 0.000 0.000 0.000 0.000 __init__.py:1308(warning)
2 0.000 0.000 0.000 0.000 __init__.py:1320(warn)
4 0.000 0.000 0.000 0.000 __init__.py:1374(findCaller)
4 0.000 0.000 0.000 0.000 __init__.py:1404(makeRecord)
4 0.000 0.000 0.000 0.000 __init__.py:1419(_log)
4 0.000 0.000 0.000 0.000 __init__.py:1444(handle)
4 0.000 0.000 0.000 0.000 __init__.py:1498(callHandlers)
566898 0.166 0.000 0.166 0.000 __init__.py:1528(getEffectiveLevel)
566898 0.309 0.000 0.476 0.000 __init__.py:1542(isEnabledFor)
4 0.000 0.000 0.000 0.000 __init__.py:157(<lambda>)
4 0.000 0.000 0.000 0.000 __init__.py:251(__init__)
4 0.000 0.000 0.000 0.000 __init__.py:329(getMessage)
4 0.000 0.000 0.000 0.000 __init__.py:387(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:390(format)
4 0.000 0.000 0.000 0.000 __init__.py:540(usesTime)
4 0.000 0.000 0.000 0.000 __init__.py:546(formatMessage)
4 0.000 0.000 0.000 0.000 __init__.py:562(format)
8 0.000 0.000 0.000 0.000 __init__.py:703(filter)
8 0.000 0.000 0.000 0.000 __init__.py:807(acquire)
8 0.000 0.000 0.000 0.000 __init__.py:814(release)
4 0.000 0.000 0.000 0.000 __init__.py:827(format)
4 0.000 0.000 0.000 0.000 __init__.py:850(handle)
4 0.000 0.000 0.000 0.000 __init__.py:969(flush)
4 0.000 0.000 0.000 0.000 __init__.py:980(emit)
288946 0.112 0.000 1.643 0.000 _methods.py:31(_sum)
177491 0.330 0.000 0.330 0.000 arrayprint.py:256(_get_formatdict)
177491 0.169 0.000 3.542 0.000 arrayprint.py:259(<lambda>)
177491 0.465 0.000 4.419 0.000 arrayprint.py:299(_get_format_function)
177491 0.623 0.000 9.729 0.000 arrayprint.py:343(_array2string)
532473/177491 0.987 0.000 10.679 0.000 arrayprint.py:381(wrapper)
532473/177491 0.721 0.000 10.150 0.000 arrayprint.py:399(array2string)
1225350 0.971 0.000 1.470 0.000 arrayprint.py:527(_extendLine)
177491 1.458 0.000 3.920 0.000 arrayprint.py:535(_formatArray)
177491 0.768 0.000 3.373 0.000 arrayprint.py:712(__init__)
1225350 0.960 0.000 0.960 0.000 arrayprint.py:725(__call__)
1 0.000 0.000 0.000 0.000 enum.py:265(__call__)
1 0.000 0.000 0.000 0.000 enum.py:515(__new__)
1 0.000 0.000 0.000 0.000 enum.py:544(_missing_)
177491 0.209 0.000 0.209 0.000 enum.py:552(__str__)
177491 0.316 0.000 0.525 0.000 enum.py:564(__format__)
755255 0.238 0.000 0.352 0.000 enum.py:579(__hash__)
200000 0.039 0.000 0.039 0.000 enum.py:592(name)
28626 0.005 0.000 0.005 0.000 enum.py:597(value)
200505 0.443 0.000 0.643 0.000 eventgen.py:115(_push)
200001 0.474 0.000 0.863 0.000 eventgen.py:122(pop)
200000 0.834 0.000 0.983 0.000 eventgen.py:137(ce_str)
14313 0.017 0.000 0.035 0.000 eventgen.py:15(__lt__)
99673 0.186 0.000 0.939 0.000 eventgen.py:44(event_new)
78949 0.094 0.000 0.500 0.000 eventgen.py:52(event_end)
11887 0.078 0.000 0.261 0.000 eventgen.py:61(event_new_handoff)
9996 0.017 0.000 0.103 0.000 eventgen.py:90(event_end_handoff)
77374 0.284 0.000 0.364 0.000 eventgen.py:94(reassign)
177491 0.195 0.000 0.595 0.000 fromnumeric.py:1380(ravel)
200001 0.098 0.000 0.490 0.000 fromnumeric.py:1487(nonzero)
288946 0.590 0.000 2.352 0.000 fromnumeric.py:1730(sum)
200001 0.130 0.000 0.392 0.000 fromnumeric.py:55(_wrapfunc)
4 0.000 0.000 0.000 0.000 genericpath.py:117(_splitext)
49 0.000 0.000 0.001 0.000 grid.py:172(neighbors1)
49 0.001 0.000 0.001 0.000 grid.py:195(neighbors2)
532473/177491 0.365 0.000 10.826 0.000 numeric.py:1905(array_str)
177491 0.062 0.000 0.151 0.000 numeric.py:463(asarray)
177491 0.051 0.000 0.104 0.000 numeric.py:534(asanyarray)
4 0.000 0.000 0.000 0.000 posixpath.py:119(splitext)
4 0.000 0.000 0.000 0.000 posixpath.py:142(basename)
4 0.000 0.000 0.000 0.000 posixpath.py:39(_get_sep)
6 0.000 0.000 0.000 0.000 posixpath.py:50(normcase)
4 0.000 0.000 0.000 0.000 process.py:137(name)
4 0.000 0.000 0.000 0.000 process.py:35(current_process)
1 0.000 0.000 0.000 0.000 signal.py:25(_int_to_enum)
2 0.000 0.000 0.000 0.000 signal.py:35(_enum_to_int)
1 0.000 0.000 0.000 0.000 signal.py:45(signal)
99624 0.066 0.000 0.066 0.000 stats.py:38(new)
20675 0.028 0.000 0.040 0.000 stats.py:42(new_rej)
88545 0.045 0.000 0.045 0.000 stats.py:48(end)
11831 0.006 0.000 0.006 0.000 stats.py:51(hoff_new)
1835 0.001 0.000 0.002 0.000 stats.py:54(hoff_rej)
22510 0.013 0.000 0.013 0.000 stats.py:58(rej)
200000 0.261 0.000 1.490 0.000 stats.py:64(iter)
1 0.000 0.000 0.000 0.000 stats.py:69(n_iter)
1 0.000 0.000 0.000 0.000 stats.py:86(endsim)
1 0.000 0.000 0.000 0.000 strats.py:189(get_init_action)
200000 1.234 0.000 19.760 0.000 strats.py:193(get_action)
177490 1.294 0.000 1.860 0.000 strats.py:220(execute_action)
200001 3.897 0.000 17.128 0.000 strats.py:243(optimal_ch)
88945 0.074 0.000 1.112 0.000 strats.py:299(reward)
88945 0.017 0.000 0.017 0.000 strats.py:308(discount)
1241938 0.681 0.000 0.681 0.000 strats.py:333(get_qval)
88945 0.167 0.000 0.167 0.000 strats.py:336(update_qval)
1 0.000 0.000 28.612 28.612 strats.py:40(init_sim)
1 1.383 1.383 28.611 28.611 strats.py:49(_simulate)
4 0.000 0.000 0.000 0.000 threading.py:1076(name)
4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread)
228626 0.122 0.000 0.166 0.000 types.py:135(__get__)
177491 0.075 0.000 0.075 0.000 {built-in method _functools.reduce}
200001 0.203 0.000 0.234 0.000 {built-in method _heapq.heappop}
200505 0.079 0.000 0.083 0.000 {built-in method _heapq.heappush}
320262 0.068 0.000 0.068 0.000 {built-in method _operator.gt}
832731 0.136 0.000 0.136 0.000 {built-in method _operator.lt}
1 0.000 0.000 0.000 0.000 {built-in method _signal.signal}
532481 0.090 0.000 0.090 0.000 {built-in method _thread.get_ident}
2 0.000 0.000 0.000 0.000 {built-in method _warnings.warn}
1 0.000 0.000 28.612 28.612 {built-in method builtins.exec}
200001 0.066 0.000 0.066 0.000 {built-in method builtins.getattr}
14 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
755255 0.113 0.000 0.113 0.000 {built-in method builtins.hash}
532473 0.092 0.000 0.092 0.000 {built-in method builtins.id}
466451 0.166 0.000 0.166 0.000 {built-in method builtins.isinstance}
532473 0.083 0.000 0.083 0.000 {built-in method builtins.issubclass}
3750044 0.325 0.000 0.325 0.000 {built-in method builtins.len}
177687 0.091 0.000 0.091 0.000 {built-in method builtins.max}
196 0.000 0.000 0.000 0.000 {built-in method builtins.min}
354982 0.142 0.000 0.142 0.000 {built-in method numpy.core.multiarray.array}
111456 0.095 0.000 0.095 0.000 {built-in method numpy.core.multiarray.where}
14 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getpid}
4 0.000 0.000 0.000 0.000 {built-in method sys._getframe}
5 0.000 0.000 0.000 0.000 {built-in method time.time}
8 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.RLock' objects}
532473 0.089 0.000 0.089 0.000 {method 'add' of 'set' objects}
875 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
16345 0.110 0.000 0.110 0.000 {method 'choice' of 'mtrand.RandomState' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
532473 0.097 0.000 0.097 0.000 {method 'discard' of 'set' objects}
188618 0.633 0.000 0.633 0.000 {method 'exponential' of 'mtrand.RandomState' objects}
4 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'flush' of '_io.TextIOWrapper' objects}
8 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
354982 0.066 0.000 0.066 0.000 {method 'item' of 'numpy.ndarray' objects}
200001 0.196 0.000 0.196 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
11887 0.052 0.000 0.052 0.000 {method 'randint' of 'mtrand.RandomState' objects}
167895 0.157 0.000 0.157 0.000 {method 'random_sample' of 'mtrand.RandomState' objects}
177491 0.251 0.000 0.251 0.000 {method 'ravel' of 'numpy.ndarray' objects}
643928 2.511 0.000 2.511 0.000 {method 'reduce' of 'numpy.ufunc' objects}
8 0.000 0.000 0.000 0.000 {method 'release' of '_thread.RLock' objects}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
2451118 0.328 0.000 0.328 0.000 {method 'rstrip' of 'str' objects}
8 0.000 0.000 0.000 0.000 {method 'write' of '_io.TextIOWrapper' objects}
numpy was installed through pip on the laptop, and through the Fedora repositories on the desktop. Removing the package and installing it through pip removed arrayprint (fillFormat) from the profiling results and the runtime is now very much the same (which is still a bit weird). It's also strange that the other arrayprint functions are still being called, with 10 seconds of cumulative time.
I am running tests for my project using nose2:
#!/bin/sh
nose2 --config=tests/nose2.cfg "$#"
Under CPython tests run 4 times faster than under PyPy:
Python 2.7.8 (default, Oct 20 2014, 15:05:19)
[GCC 4.9.1] on linux2
...
Ran 58 tests in 25.369s
Python 2.7.9 (2.5.1+dfsg-1~ppa1+ubuntu14.04, Mar 27 2015, 19:19:42)
[PyPy 2.5.1 with GCC 4.8.2] on linux2
...
Ran 58 tests in 100.854s
What could be the cause?
Is there a way to tweak PyPy configuration using environment variables or or a configuration file on some standard path? Because in my case I am running nose bootstrap script and I cannot control command line options for PyPy.
Here is one specific test:
1272695 function calls (1261234 primitive calls) in 1.165 seconds
Ordered by: cumulative time, internal time, call count
List reduced from 1224 to 40 due to restriction <40>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.171 1.171 test_progress.py:37(test_progress)
15 0.000 0.000 1.169 0.078 __init__.py:52(api_request)
15 0.000 0.000 1.160 0.077 __init__.py:46(request)
15 0.000 0.000 1.152 0.077 test.py:695(open)
15 0.000 0.000 1.150 0.077 test.py:655(run_wsgi_app)
15 0.000 0.000 1.144 0.076 test.py:828(run_wsgi_app)
15 0.000 0.000 1.144 0.076 application.py:101(__call__)
15 0.000 0.000 1.138 0.076 sessions.py:329(__call__)
15 0.000 0.000 1.071 0.071 course_object.py:14(__call__)
15 0.000 0.000 1.005 0.067 user_auth.py:7(__call__)
15 0.000 0.000 0.938 0.063 application.py:27(application)
15 0.000 0.000 0.938 0.063 application.py:81(wsgi_app)
15 0.000 0.000 0.876 0.058 ember_backend.py:188(__call__)
15 0.000 0.000 0.875 0.058 ember_backend.py:233(handle_request)
176 0.002 0.000 0.738 0.004 __init__.py:42(instrumented_method)
176 0.003 0.000 0.623 0.004 __init__.py:58(get_stack)
176 0.000 0.000 0.619 0.004 inspect.py:1053(stack)
176 0.010 0.000 0.619 0.004 inspect.py:1026(getouterframes)
294 0.001 0.000 0.614 0.002 cursor.py:1072(next)
248 0.001 0.000 0.612 0.002 cursor.py:998(_refresh)
144 0.002 0.000 0.608 0.004 cursor.py:912(__send_message)
7248 0.041 0.000 0.607 0.000 inspect.py:988(getframeinfo)
8 0.000 0.000 0.544 0.068 test_progress.py:31(get_progress_data)
4 0.000 0.000 0.529 0.132 test_progress.py:25(finish_item)
249 0.001 0.000 0.511 0.002 base.py:1131(next)
7 0.000 0.000 0.449 0.064 ember_backend.py:240(_handle_request)
8 0.000 0.000 0.420 0.053 ember_backend.py:307(_handle_request)
8 0.000 0.000 0.420 0.053 user_state.py:13(list)
8 0.001 0.000 0.407 0.051 user_progress.py:28(update_progress)
4 0.000 0.000 0.397 0.099 entity.py:253(post)
7248 0.051 0.000 0.362 0.000 inspect.py:518(findsource)
14532 0.083 0.000 0.332 0.000 inspect.py:440(getsourcefile)
92/63 0.000 0.000 0.308 0.005 objects.py:22(__get__)
61 0.001 0.000 0.304 0.005 base.py:168(get)
139 0.000 0.000 0.250 0.002 queryset.py:65(_iter_results)
51 0.001 0.000 0.249 0.005 queryset.py:83(_populate_cache)
29 0.000 0.000 0.220 0.008 __init__.py:81(save)
29 0.001 0.000 0.219 0.008 document.py:181(save)
21780 0.051 0.000 0.140 0.000 inspect.py:398(getfile)
32 0.002 0.000 0.139 0.004 {pymongo._cmessage._do_batched_write_command}
And the with PyPy:
6037905 function calls (6012014 primitive calls) in 7.475 seconds
Ordered by: cumulative time, internal time, call count
List reduced from 1354 to 40 due to restriction <40>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 7.960 7.960 test_progress.py:37(test_progress)
15 0.000 0.000 7.948 0.530 __init__.py:52(api_request)
15 0.000 0.000 7.873 0.525 __init__.py:46(request)
15 0.000 0.000 7.860 0.524 test.py:695(open)
15 0.000 0.000 7.856 0.524 test.py:655(run_wsgi_app)
15 0.000 0.000 7.845 0.523 test.py:828(run_wsgi_app)
15 0.000 0.000 7.844 0.523 application.py:101(__call__)
15 0.000 0.000 7.827 0.522 sessions.py:329(__call__)
15 0.000 0.000 7.205 0.480 course_object.py:14(__call__)
176 0.004 0.000 6.605 0.038 __init__.py:42(instrumented_method)
15 0.000 0.000 6.591 0.439 user_auth.py:7(__call__)
176 0.008 0.000 6.314 0.036 __init__.py:58(get_stack)
176 0.001 0.000 6.305 0.036 inspect.py:1063(stack)
176 0.027 0.000 6.304 0.036 inspect.py:1036(getouterframes)
7839 0.081 0.000 6.274 0.001 inspect.py:998(getframeinfo)
15 0.000 0.000 5.983 0.399 application.py:27(application)
15 0.001 0.000 5.983 0.399 application.py:81(wsgi_app)
15 0.000 0.000 5.901 0.393 ember_backend.py:188(__call__)
15 0.000 0.000 5.899 0.393 ember_backend.py:233(handle_request)
15714/15713 0.189 0.000 5.828 0.000 inspect.py:441(getsourcefile)
294 0.002 0.000 5.473 0.019 cursor.py:1072(next)
248 0.002 0.000 5.470 0.022 cursor.py:998(_refresh)
144 0.004 0.000 5.445 0.038 cursor.py:912(__send_message)
8367 2.133 0.000 5.342 0.001 inspect.py:473(getmodule)
249 0.002 0.000 4.316 0.017 base.py:1131(next)
8 0.000 0.000 3.966 0.496 test_progress.py:31(get_progress_data)
4 0.000 0.000 3.209 0.802 test_progress.py:25(finish_item)
7839 0.098 0.000 3.185 0.000 inspect.py:519(findsource)
7 0.000 0.000 2.944 0.421 ember_backend.py:240(_handle_request)
8 0.000 0.000 2.898 0.362 ember_backend.py:307(_handle_request)
8 0.000 0.000 2.898 0.362 user_state.py:13(list)
8 0.001 0.000 2.820 0.352 user_progress.py:28(update_progress)
61 0.001 0.000 2.546 0.042 base.py:168(get)
4 0.000 0.000 2.534 0.633 entity.py:253(post)
850362/849305 2.315 0.000 2.344 0.000 {hasattr}
92/63 0.001 0.000 2.004 0.032 objects.py:22(__get__)
127 0.001 0.000 1.915 0.015 queryset.py:65(_iter_results)
51 0.001 0.000 1.914 0.038 queryset.py:83(_populate_cache)
29 0.000 0.000 1.607 0.055 __init__.py:81(save)
29 0.001 0.000 1.605 0.055 document.py:181(save)
Do you have any idea of how can I make this function more time-efficient?
def c(n):
word = 32
#l = []
c = 0
for i in range(0, 2**word):
#print(str(bin(i)))#.count('1')
if str(bin(i)).count('1') == n:
c = c + 1
print(c)
if i == 2**28:
print('6 %')
if i == 2**29:
print('12 %')
if i == 2**30:
print('25 %')
if i == 2**31:
print('50 %')
if i == 2**32:
print('100 %')
return c
135274023 function calls in 742.161 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 391.662 391.662 742.161 742.161 <pyshell#3>:1(c)
1 0.000 0.000 742.161 742.161 <string>:1(<module>)
4816 0.014 0.000 0.014 0.000 rpc.py:149(debug)
688 0.010 0.000 3.162 0.005 rpc.py:208(remotecall)
688 0.017 0.000 0.107 0.000 rpc.py:218(asynccall)
688 0.019 0.000 3.043 0.004 rpc.py:238(asyncreturn)
688 0.002 0.000 0.002 0.000 rpc.py:244(decoderesponse)
688 0.007 0.000 3.018 0.004 rpc.py:279(getresponse)
688 0.006 0.000 0.010 0.000 rpc.py:287(_proxify)
688 0.025 0.000 3.000 0.004 rpc.py:295(_getresponse)
688 0.002 0.000 0.002 0.000 rpc.py:317(newseq)
688 0.023 0.000 0.062 0.000 rpc.py:321(putmessage)
688 0.007 0.000 0.011 0.000 rpc.py:546(__getattr__)
688 0.002 0.000 0.002 0.000 rpc.py:587(__init__)
688 0.004 0.000 3.166 0.005 rpc.py:592(__call__)
1376 0.008 0.000 0.011 0.000 threading.py:1012(current_thread)
688 0.004 0.000 0.019 0.000 threading.py:172(Condition)
688 0.009 0.000 0.015 0.000 threading.py:177(__init__)
688 0.019 0.000 2.962 0.004 threading.py:226(wait)
688 0.002 0.000 0.002 0.000 threading.py:45(__init__)
688 0.002 0.000 0.002 0.000 threading.py:50(_note)
688 0.004 0.000 0.004 0.000 threading.py:88(RLock)
688 0.004 0.000 0.004 0.000 {built-in method allocate_lock}
67620326 162.442 0.000 162.442 0.000 {built-in method bin}
688 0.007 0.000 0.007 0.000 {built-in method dumps}
1 0.000 0.000 742.161 742.161 {built-in method exec}
1376 0.003 0.000 0.003 0.000 {built-in method get_ident}
1376 0.004 0.000 0.004 0.000 {built-in method isinstance}
2064 0.005 0.000 0.005 0.000 {built-in method len}
688 0.002 0.000 0.002 0.000 {built-in method pack}
344 0.009 0.000 3.187 0.009 {built-in method print}
688 0.008 0.000 0.008 0.000 {built-in method select}
688 0.003 0.000 0.003 0.000 {method '_acquire_restore' of '_thread.RLock' objects}
688 0.002 0.000 0.002 0.000 {method '_is_owned' of '_thread.RLock' objects}
688 0.002 0.000 0.002 0.000 {method '_release_save' of '_thread.RLock' objects}
688 0.003 0.000 0.003 0.000 {method 'acquire' of '_thread.RLock' objects}
1376 2.929 0.002 2.929 0.002 {method 'acquire' of '_thread.lock' objects}
688 0.002 0.000 0.002 0.000 {method 'append' of 'list' objects}
67620325 184.869 0.000 184.869 0.000 {method 'count' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
688 0.002 0.000 0.002 0.000 {method 'get' of 'dict' objects}
688 0.002 0.000 0.002 0.000 {method 'release' of '_thread.RLock' objects}
688 0.015 0.000 0.015 0.000 {method 'send' of '_socket.socket' objects}
What I try to achieve is to calculate how many of numbers from 0 to 2**32 have n number of 1 in their binary representation.
You are counting how many 32-bit numbers have a given number of 1s. This number is the binomial coefficient 32 choose bits, and can be calculated with:
from math import factorial
print factorial(32) // (factorial(bits) * factorial(32-bits))