Develop Reference

How to get 8 maximum / highest values in each row of 2D array with their indices

I'm working on a Python script. I'm trying to achieve a list of 8 highest values and their respective indices of every single row of my 2D array in Python. The shape of my array has 4148 rows and 167 columns. What I essentially want is that for every row it should give me the 8 highest values (in descending order) present in that row with their indices.
I'm relatively new to Python, and I've tried to implement this below, however it gives me the overall 8 maximum values and their indices in the whole array.
a = predicting[:]
indices = np.argpartition(a.flatten(), -8)[-8:]
np.vstack(np.unravel_index(indices, a.shape)).T

You can refer to my exampl. You will get a 2D sequence containing the index of the 8 largest numbers and a 2D sequence containing the 8 largest numbers.:
a = np.random.randint(0,12,size=(12,12))
indice = np.argsort(-a)
indice = indice[:,:8]
b = np.sort(-a.copy())*-1
maximum_8 = b[:,:8]
One output:
a
array([[ 1, 10, 11, 8, 8, 2, 4, 11, 2, 5, 3, 6],
[11, 2, 2, 7, 3, 3, 9, 0, 0, 0, 10, 4],
[ 8, 10, 8, 10, 5, 9, 6, 7, 3, 5, 2, 8],
[ 4, 8, 8, 2, 6, 2, 0, 7, 1, 10, 10, 6],
[ 9, 1, 5, 0, 6, 4, 3, 6, 7, 0, 7, 7],
[ 3, 7, 8, 0, 11, 10, 10, 8, 2, 7, 2, 7],
[ 6, 7, 5, 11, 6, 5, 4, 3, 0, 0, 8, 2],
[ 7, 11, 7, 9, 11, 11, 8, 11, 4, 11, 6, 11],
[11, 3, 9, 7, 11, 8, 11, 3, 8, 9, 0, 3],
[ 4, 7, 6, 9, 11, 3, 8, 0, 5, 11, 6, 5],
[ 9, 11, 8, 2, 5, 4, 4, 4, 9, 4, 7, 9],
[ 5, 5, 3, 6, 4, 8, 4, 9, 4, 1, 8, 9]])
indice
array([[ 2, 7, 1, 3, 4, 11, 9, 6],
[ 0, 10, 6, 3, 11, 4, 5, 1],
[ 1, 3, 5, 0, 2, 11, 7, 6],
[ 9, 10, 1, 2, 7, 4, 11, 0],
[ 0, 8, 10, 11, 4, 7, 2, 5],
[ 4, 5, 6, 2, 7, 1, 9, 11],
[ 3, 10, 1, 0, 4, 2, 5, 6],
[ 1, 4, 5, 7, 9, 11, 3, 6],
[ 0, 4, 6, 2, 9, 5, 8, 3],
[ 4, 9, 3, 6, 1, 2, 10, 8],
[ 1, 0, 8, 11, 2, 10, 4, 5],
[ 7, 11, 5, 10, 3, 0, 1, 4]], dtype=int64)
maximum_8
array([[11, 11, 10, 8, 8, 6, 5, 4],
[11, 10, 9, 7, 4, 3, 3, 2],
[10, 10, 9, 8, 8, 8, 7, 6],
[10, 10, 8, 8, 7, 6, 6, 4],
[ 9, 7, 7, 7, 6, 6, 5, 4],
[11, 10, 10, 8, 8, 7, 7, 7],
[11, 8, 7, 6, 6, 5, 5, 4],
[11, 11, 11, 11, 11, 11, 9, 8],
[11, 11, 11, 9, 9, 8, 8, 7],
[11, 11, 9, 8, 7, 6, 6, 5],
[11, 9, 9, 9, 8, 7, 5, 4],
[ 9, 9, 8, 8, 6, 5, 5, 4]])

Another possible solution:
b = np.flip(np.argsort(a, axis=1)[:,-8:], axis=1)
v = np.take_along_axis(a, b, axis=1)
v, b # v = values; b = indices

If you can put the row data in a dictionary, swap the key and value and put it in to a list, then sort it in reverse. Then take the first 8 values of the list for each row.
# put the row data in a dictionary called dictionaryRow, swap key and value and put into a list, sort in reverse)
x = sorted( [ (value, key) for key, value in dictionaryRow.items() ] ), reverse=True)
# Take the top 8 from the list
x[:8]

Python for loops does not take every element in list [duplicate]

This question already has answers here:
Why does "a == x or y or z" always evaluate to True? How can I compare "a" to all of those?
(8 answers)
Closed 2 years ago.
Hello I am trying to add numbers to a new list in a for loop if condintions match. However I do not get the result I was hopping for. For me it seems like the for loop stops after the first encounter of a match ever though i is probably analized.
print(months_list)
for i in months_list:
n_days_list = []
if i == 1 or 3 or 5 or 7 or 8 or 10 or 12:
n_days_list.append(31)
elif i == 2:
n_days_list.append(28)
else:
n_days_list.append(30)
print(n_days_list)
Output:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
[31]
Thanks for the help.

The reason why the length of ur code's output is 1 is because you are initializing the n_days_list to [] at each iteration which means u are removing the effect of the previous iteration.
However, if u remove "n_days_list = []", you will end up with a list of length of n_days_list containing only 31 that's bcz the first condition is always true because if i == 1 or 3 or 5 or.... is equivalent to if i == 1 or True or True or.... since the numbers 3, 5.... are different than 0 hence they are True. So just change to if i == 1 or i == 3 or i == 5 or.... and remove that n_days_list = []
and u will be set to go.

combinations of a list [duplicate]

This question already has answers here:
How do I clone a list so that it doesn't change unexpectedly after assignment?
(24 answers)
Closed 4 years ago.
I'm trying to get all the combinations of my list, but everytime two elements need to be removed, how do i remove those elements?
I tried to make a for loop two times and every time its removes two elements, but at the end it does not restore the list
indexes = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for een in indexes:
for twee in indexes:
temp = indexes
if een == twee:
pass
else:
if een in temp:
temp.remove(een)
temp.remove(twee)
print(temp)
temp = indexes
i expected to output every time a list of length of 9 but the list keeps getting shorter.
the output i got was:
[2, 3, 4, 5, 6, 7, 8, 9, 10]
[2, 3, 5, 6, 7, 8, 9, 10]
[2, 3, 5, 7, 8, 9, 10]
[2, 3, 5, 7, 9, 10]
[2, 3, 5, 7, 9]
[5, 7, 9]
[5, 9]
the first list is correct, but on the next one, the 1 does not return to the list. what am i doing wrong? Also after this one is done een should equal to 1 and do it all over again, after that een should equal to 2.....
this should be the output
[2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[2, 3, 4, 5, 6, 7, 8, 9, 10]
[0, 3, 4, 5, 6, 7, 8, 9, 10]
[0, 2, 4, 5, 6, 7, 8, 9, 10]
[0, 2, 3, 5, 6, 7, 8, 9, 10]
[0, 2, 3, 4, 6, 7, 8, 9, 10]
[0, 2, 3, 4, 5, 7, 8, 9, 10]
[0, 2, 3, 4, 5, 6, 8, 9, 10]
[0, 2, 3, 4, 5, 6, 7, 9, 10]
[0, 2, 3, 4, 5, 6, 7, 8, 10]
[0, 2, 3, 4, 5, 6, 7, 8, 9]
[1, 3, 4, 5, 6, 7, 8, 9, 10]
[0, 3, 4, 5, 6, 7, 8, 9, 10]
[0, 1, 4, 5, 6, 7, 8, 9, 10]
and should go one untill every combination is reached

Replace temp = indexes by temp = indexes[:]

Applying rectangular step function using scipy

Is there a rectangular filter in scipy? I want to replicate #Mike T's solution https://gis.stackexchange.com/questions/9431/what-raster-smoothing-generalization-tools-are-available/162852 but nstead of doing gaussian blur, I want to apply a rectangular filter (i.e. essentially a step function)

You can simply use scipy.ndimage.convolve and define the weights (filter) yourself. There is also a specialized and optimized rectangular filter function: uniform_filter
For example to apply a width 3 mean-filter (multiply by ndim*size if you want a sum-filter):
>>> from scipy.ndimage import uniform_filter
>>> uniform_filter([1,4,2,56,2,3,6,1,3,1,3], size=3)
array([ 2, 2, 20, 19, 20, 3, 3, 3, 1, 2, 2])
This can also be applied to multidimensional arrays:
>>> uniform_filter(np.random.randint(0, 20, (10, 10)), size=3) # 3x3 filter
array([[ 6, 7, 10, 9, 9, 7, 5, 7, 9, 12],
[ 6, 7, 9, 9, 7, 6, 5, 5, 7, 9],
[ 5, 8, 8, 9, 7, 6, 4, 4, 6, 8],
[ 9, 10, 9, 10, 8, 6, 4, 4, 8, 11],
[10, 12, 9, 10, 9, 10, 8, 8, 9, 10],
[12, 12, 9, 10, 10, 10, 9, 9, 9, 9],
[12, 11, 9, 8, 7, 8, 7, 8, 6, 5],
[11, 10, 9, 9, 9, 9, 6, 8, 8, 9],
[12, 9, 7, 7, 9, 8, 6, 6, 6, 8],
[12, 9, 8, 9, 12, 10, 7, 5, 6, 9]])
>>> uniform_filter(np.random.randint(0, 20, (10, 10)), size=(5, 3)) # 5x3 filter
array([[ 7, 7, 8, 10, 11, 10, 11, 11, 12, 12],
[ 7, 7, 7, 9, 10, 11, 11, 10, 9, 9],
[ 7, 6, 6, 8, 8, 9, 9, 9, 8, 8],
[ 6, 6, 6, 8, 8, 9, 8, 7, 7, 8],
[ 8, 8, 7, 9, 8, 11, 8, 8, 6, 6],
[ 7, 8, 7, 9, 8, 11, 7, 7, 5, 7],
[ 9, 8, 8, 9, 9, 9, 7, 7, 7, 8],
[ 8, 8, 8, 9, 10, 10, 8, 7, 6, 6],
[ 9, 7, 6, 8, 9, 10, 8, 7, 6, 6],
[ 9, 6, 6, 7, 10, 8, 8, 6, 6, 6]])

Performance issues in Python for small loops over simple array look-ups

Why are simple loops and/or simple array look-ups so slow in Python?
Specifically, Python (using pypy) is approximately 9 times slower than C++ (with -O2) in the following example. What is the technical reason that explains the performance penalty? Is it the implementation of Python loops in machine code? differences in the optimizations used by the compilers? the memory management? or something else?
The Python code:
# File: timing.py
import sys
T = [ \
[ 0, 6, 7, 7, 7, 8, 6, 7, 8,19,20, 7, 7, 7, 7, 7, 7,21,22,19,20,21,22,29, 7, 7, 7, 7, 7,29,35,36, 7, 7, 7,35,36, 7, 7,], \
[ 9, 7,10, 7, 7, 7, 1, 7,23, 7, 7, 1,23, 7, 7, 7, 7, 7, 7, 9,10,30,31, 7, 9,10,30,31, 7,23, 7, 7,23, 7, 7,30,31,30,31,], \
[ 7, 7, 2,11,12, 7, 7, 7, 7, 7, 7,11,12,24,25,26,27, 7, 7, 7, 7, 7, 7, 7,24,25,26,27,32, 7, 7, 7,32,37,38, 7, 7,37,38,], \
[13, 7,14, 7, 7, 7, 3, 7,28, 7, 7, 3,28, 7, 7, 7, 7, 7, 7,13,14,33,34, 7,13,14,33,34, 7,28, 7, 7,28, 7, 7,33,34,33,34,], \
[15, 7,16, 7, 7, 7, 7, 7, 4, 7, 7, 7, 4, 7, 7, 7, 7, 7, 7, 7, 7,15,16, 7, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[17, 7,18, 7, 7, 7, 7, 7, 5, 7, 7, 7, 5, 7, 7, 7, 7, 7, 7, 7, 7,17,18, 7, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[19, 7,20, 7, 7, 7, 6, 7,29, 7, 7, 6,29, 7, 7, 7, 7, 7, 7,19,20,35,36, 7,19,20,35,36, 7,29, 7, 7,29, 7, 7,35,36,35,36,], \
[ 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[21, 7,22, 7, 7, 7, 7, 7, 8, 7, 7, 7, 8, 7, 7, 7, 7, 7, 7, 7, 7,21,22, 7, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 9, 1, 7, 7, 7,23, 1, 7,23, 9,10, 7, 7, 7, 7, 7, 7,30,31, 9,10,30,31,23, 7, 7, 7, 7, 7,23,30,31, 7, 7, 7,30,31, 7, 7,], \
[ 7, 7,10, 1,23, 7, 7, 7, 7, 7, 7, 1,23, 9,10,30,31, 7, 7, 7, 7, 7, 7, 7, 9,10,30,31,23, 7, 7, 7,23,30,31, 7, 7,30,31,], \
[24, 7,25, 7, 7, 7,11, 7,32, 7, 7,11,32, 7, 7, 7, 7, 7, 7,24,25,37,38, 7,24,25,37,38, 7,32, 7, 7,32, 7, 7,37,38,37,38,], \
[26, 7,27, 7, 7, 7, 7, 7,12, 7, 7, 7,12, 7, 7, 7, 7, 7, 7, 7, 7,26,27, 7, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[13, 3, 7, 7, 7,28, 3, 7,28,13,14, 7, 7, 7, 7, 7, 7,33,34,13,14,33,34,28, 7, 7, 7, 7, 7,28,33,34, 7, 7, 7,33,34, 7, 7,], \
[ 7, 7,14, 3,28, 7, 7, 7, 7, 7, 7, 3,28,13,14,33,34, 7, 7, 7, 7, 7, 7, 7,13,14,33,34,28, 7, 7, 7,28,33,34, 7, 7,33,34,], \
[15, 7, 7, 7, 7, 4, 7, 7, 4, 7, 7, 7, 7, 7, 7, 7, 7,15,16, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,16, 7, 4, 7, 7, 7, 7, 7, 7, 7, 4, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[17, 7, 7, 7, 7, 5, 7, 7, 5, 7, 7, 7, 7, 7, 7, 7, 7,17,18, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,18, 7, 5, 7, 7, 7, 7, 7, 7, 7, 5, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[19, 6, 7, 7, 7,29, 6, 7,29,19,20, 7, 7, 7, 7, 7, 7,35,36,19,20,35,36,29, 7, 7, 7, 7, 7,29,35,36, 7, 7, 7,35,36, 7, 7,], \
[ 7, 7,20, 6,29, 7, 7, 7, 7, 7, 7, 6,29,19,20,35,36, 7, 7, 7, 7, 7, 7, 7,19,20,35,36,29, 7, 7, 7,29,35,36, 7, 7,35,36,], \
[21, 7, 7, 7, 7, 8, 7, 7, 8, 7, 7, 7, 7, 7, 7, 7, 7,21,22, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,22, 7, 8, 7, 7, 7, 7, 7, 7, 7, 8, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[30, 7,31, 7, 7, 7, 7, 7,23, 7, 7, 7,23, 7, 7, 7, 7, 7, 7, 7, 7,30,31, 7, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[24,11, 7, 7, 7,32,11, 7,32,24,25, 7, 7, 7, 7, 7, 7,37,38,24,25,37,38,32, 7, 7, 7, 7, 7,32,37,38, 7, 7, 7,37,38, 7, 7,], \
[ 7, 7,25,11,32, 7, 7, 7, 7, 7, 7,11,32,24,25,37,38, 7, 7, 7, 7, 7, 7, 7,24,25,37,38,32, 7, 7, 7,32,37,38, 7, 7,37,38,], \
[26, 7, 7, 7, 7,12, 7, 7,12, 7, 7, 7, 7, 7, 7, 7, 7,26,27, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,27, 7,12, 7, 7, 7, 7, 7, 7, 7,12, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[33, 7,34, 7, 7, 7, 7, 7,28, 7, 7, 7,28, 7, 7, 7, 7, 7, 7, 7, 7,33,34, 7, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[35, 7,36, 7, 7, 7, 7, 7,29, 7, 7, 7,29, 7, 7, 7, 7, 7, 7, 7, 7,35,36, 7, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[30, 7, 7, 7, 7,23, 7, 7,23, 7, 7, 7, 7, 7, 7, 7, 7,30,31, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,31, 7,23, 7, 7, 7, 7, 7, 7, 7,23, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[37, 7,38, 7, 7, 7, 7, 7,32, 7, 7, 7,32, 7, 7, 7, 7, 7, 7, 7, 7,37,38, 7, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[33, 7, 7, 7, 7,28, 7, 7,28, 7, 7, 7, 7, 7, 7, 7, 7,33,34, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,34, 7,28, 7, 7, 7, 7, 7, 7, 7,28, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[35, 7, 7, 7, 7,29, 7, 7,29, 7, 7, 7, 7, 7, 7, 7, 7,35,36, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,36, 7,29, 7, 7, 7, 7, 7, 7, 7,29, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[37, 7, 7, 7, 7,32, 7, 7,32, 7, 7, 7, 7, 7, 7, 7, 7,37,38, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
[ 7, 7,38, 7,32, 7, 7, 7, 7, 7, 7, 7,32, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,], \
]
M = range(39)
idempotents = [0,2,6,7,8,9,11,12,14,16,17,19,21,25,27]
omega = [0,7,2,7,7,7,6,7,8,9,7,11,12,7,14,7,16,17,7,19,7,21,7,7,7,25,7,27,7,7,7,7,7,7,7,7,7,7,7]
def check():
for e in idempotents:
for x in M:
ex = T[e][x]
for s in M:
es = T[e][s]
for f in idempotents:
exf = T[ex][f]
esf = T[es][f]
for y in M:
exfy = omega[T[exf][y]]
for t in M:
tesf = omega[T[t][esf]]
if T[T[exfy][exf]][tesf] != T[T[exfy][esf]][tesf]:
return 0
return 1
sys.exit(check())
The C++ code (requires C++11 because of the new array initialization and iteration syntax):
// File: timing.cc
// Compile via 'g++ -std=c++11 -O2 timing.cc'
// Run via 'time ./a.out'
#include <vector>
#include <cstddef>
int main(int, char **) {
const size_t N = 39;
typedef unsigned element_t;
const std::vector<std::vector<element_t>> T{{
{{ 0, 6, 7, 7, 7, 8, 6, 7, 8,19,20, 7, 7, 7, 7, 7, 7,21,22,19,20,21,22,29, 7, 7, 7, 7, 7,29,35,36, 7, 7, 7,35,36, 7, 7,}},
{{ 9, 7,10, 7, 7, 7, 1, 7,23, 7, 7, 1,23, 7, 7, 7, 7, 7, 7, 9,10,30,31, 7, 9,10,30,31, 7,23, 7, 7,23, 7, 7,30,31,30,31,}},
{{ 7, 7, 2,11,12, 7, 7, 7, 7, 7, 7,11,12,24,25,26,27, 7, 7, 7, 7, 7, 7, 7,24,25,26,27,32, 7, 7, 7,32,37,38, 7, 7,37,38,}},
{{13, 7,14, 7, 7, 7, 3, 7,28, 7, 7, 3,28, 7, 7, 7, 7, 7, 7,13,14,33,34, 7,13,14,33,34, 7,28, 7, 7,28, 7, 7,33,34,33,34,}},
{{15, 7,16, 7, 7, 7, 7, 7, 4, 7, 7, 7, 4, 7, 7, 7, 7, 7, 7, 7, 7,15,16, 7, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{17, 7,18, 7, 7, 7, 7, 7, 5, 7, 7, 7, 5, 7, 7, 7, 7, 7, 7, 7, 7,17,18, 7, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{19, 7,20, 7, 7, 7, 6, 7,29, 7, 7, 6,29, 7, 7, 7, 7, 7, 7,19,20,35,36, 7,19,20,35,36, 7,29, 7, 7,29, 7, 7,35,36,35,36,}},
{{ 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{21, 7,22, 7, 7, 7, 7, 7, 8, 7, 7, 7, 8, 7, 7, 7, 7, 7, 7, 7, 7,21,22, 7, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 9, 1, 7, 7, 7,23, 1, 7,23, 9,10, 7, 7, 7, 7, 7, 7,30,31, 9,10,30,31,23, 7, 7, 7, 7, 7,23,30,31, 7, 7, 7,30,31, 7, 7,}},
{{ 7, 7,10, 1,23, 7, 7, 7, 7, 7, 7, 1,23, 9,10,30,31, 7, 7, 7, 7, 7, 7, 7, 9,10,30,31,23, 7, 7, 7,23,30,31, 7, 7,30,31,}},
{{24, 7,25, 7, 7, 7,11, 7,32, 7, 7,11,32, 7, 7, 7, 7, 7, 7,24,25,37,38, 7,24,25,37,38, 7,32, 7, 7,32, 7, 7,37,38,37,38,}},
{{26, 7,27, 7, 7, 7, 7, 7,12, 7, 7, 7,12, 7, 7, 7, 7, 7, 7, 7, 7,26,27, 7, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{13, 3, 7, 7, 7,28, 3, 7,28,13,14, 7, 7, 7, 7, 7, 7,33,34,13,14,33,34,28, 7, 7, 7, 7, 7,28,33,34, 7, 7, 7,33,34, 7, 7,}},
{{ 7, 7,14, 3,28, 7, 7, 7, 7, 7, 7, 3,28,13,14,33,34, 7, 7, 7, 7, 7, 7, 7,13,14,33,34,28, 7, 7, 7,28,33,34, 7, 7,33,34,}},
{{15, 7, 7, 7, 7, 4, 7, 7, 4, 7, 7, 7, 7, 7, 7, 7, 7,15,16, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,16, 7, 4, 7, 7, 7, 7, 7, 7, 7, 4, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7,15,16, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{17, 7, 7, 7, 7, 5, 7, 7, 5, 7, 7, 7, 7, 7, 7, 7, 7,17,18, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,18, 7, 5, 7, 7, 7, 7, 7, 7, 7, 5, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7,17,18, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{19, 6, 7, 7, 7,29, 6, 7,29,19,20, 7, 7, 7, 7, 7, 7,35,36,19,20,35,36,29, 7, 7, 7, 7, 7,29,35,36, 7, 7, 7,35,36, 7, 7,}},
{{ 7, 7,20, 6,29, 7, 7, 7, 7, 7, 7, 6,29,19,20,35,36, 7, 7, 7, 7, 7, 7, 7,19,20,35,36,29, 7, 7, 7,29,35,36, 7, 7,35,36,}},
{{21, 7, 7, 7, 7, 8, 7, 7, 8, 7, 7, 7, 7, 7, 7, 7, 7,21,22, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,22, 7, 8, 7, 7, 7, 7, 7, 7, 7, 8, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7,21,22, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{30, 7,31, 7, 7, 7, 7, 7,23, 7, 7, 7,23, 7, 7, 7, 7, 7, 7, 7, 7,30,31, 7, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{24,11, 7, 7, 7,32,11, 7,32,24,25, 7, 7, 7, 7, 7, 7,37,38,24,25,37,38,32, 7, 7, 7, 7, 7,32,37,38, 7, 7, 7,37,38, 7, 7,}},
{{ 7, 7,25,11,32, 7, 7, 7, 7, 7, 7,11,32,24,25,37,38, 7, 7, 7, 7, 7, 7, 7,24,25,37,38,32, 7, 7, 7,32,37,38, 7, 7,37,38,}},
{{26, 7, 7, 7, 7,12, 7, 7,12, 7, 7, 7, 7, 7, 7, 7, 7,26,27, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,27, 7,12, 7, 7, 7, 7, 7, 7, 7,12, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7,26,27, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{33, 7,34, 7, 7, 7, 7, 7,28, 7, 7, 7,28, 7, 7, 7, 7, 7, 7, 7, 7,33,34, 7, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{35, 7,36, 7, 7, 7, 7, 7,29, 7, 7, 7,29, 7, 7, 7, 7, 7, 7, 7, 7,35,36, 7, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{30, 7, 7, 7, 7,23, 7, 7,23, 7, 7, 7, 7, 7, 7, 7, 7,30,31, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,31, 7,23, 7, 7, 7, 7, 7, 7, 7,23, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7,30,31, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{37, 7,38, 7, 7, 7, 7, 7,32, 7, 7, 7,32, 7, 7, 7, 7, 7, 7, 7, 7,37,38, 7, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{33, 7, 7, 7, 7,28, 7, 7,28, 7, 7, 7, 7, 7, 7, 7, 7,33,34, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,34, 7,28, 7, 7, 7, 7, 7, 7, 7,28, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7,33,34, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{35, 7, 7, 7, 7,29, 7, 7,29, 7, 7, 7, 7, 7, 7, 7, 7,35,36, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,36, 7,29, 7, 7, 7, 7, 7, 7, 7,29, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7,35,36, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{37, 7, 7, 7, 7,32, 7, 7,32, 7, 7, 7, 7, 7, 7, 7, 7,37,38, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
{{ 7, 7,38, 7,32, 7, 7, 7, 7, 7, 7, 7,32, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7,37,38, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,}},
}};
const std::vector<element_t> idempotents{{0,2,6,7,8,9,11,12,14,16,17,19,21,25,27}};
const std::vector<element_t> omega{{0,7,2,7,7,7,6,7,8,9,7,11,12,7,14,7,16,17,7,19,7,21,7,7,7,25,7,27,7,7,7,7,7,7,7,7,7,7,7}};
element_t ex, es, exf, esf, exfy, tesf;
for(auto e: idempotents) {
for(size_t x = 0; x < N; ++x) {
ex = T[e][x];
for(size_t s = 0; s < N; ++s) {
es = T[e][s];
for(auto f: idempotents) {
exf = T[ex][f];
esf = T[es][f];
for(size_t y = 0; y < N; ++y) {
exfy = omega[T[exf][y]];
for(size_t t = 0; t < N; ++t) {
tesf = omega[T[t][esf]];
if(T[T[exfy][exf]][tesf] != T[T[exfy][esf]][tesf])
return 0;
}}}}}}
return 1;
}
(Don't ask for details about what the code does. Roughly speaking, a decision procedure in the context of algebraic formal language theory is implemented; the code verifies an identity on the monoid given by the multiplication table T. In particular, the code is not a contrived example but a real-world application. Of course, one can argue about "applications" in the context of formal language theory.)
Timing results
With the code as above the user CPU running times on my machine are as follows:
time pypy timing.py outputs 0m9.329s
time python timing.py outputs 2m18.389s
g++ -std=c++11 -O2 timing.cc && time ./a.out outputs 0m1.064s
Edit
For a fairer comparison I did some optimizations that g++ seems to incorporate automatically. I re-ordered the loops and moved the variable assignments as far outward as possible (as suggested by the comments). This yields a speedup factor for pypy of about 2.5 and for python of about 2.
Also for fairness, I used the dynamic-sized std::vector instead of the constant-sized std::array.
I deleted the asides as to why numpy is slower (the comments indicate that I did not use it correctly) and why executing the loops in the main part of the script is slower (it is known that Python is faster with local variables than with global variables; the comments give references for this).
I am aware of the fact that C++ and Python have different scopes. I also know that C++ is compiled, whereas Python is normally interpreted (which is why I used pypy). I want to know the ultimate technical reason as to why pypy is so much slower on this specific piece of code. (Using numpy and numba one may be able to attain near-native performance here, but this is not in the spirit of my question, because it shifts virtually all the computation back to C code.) I clarified my question accordingly.

Short answer to numpy and python loops
if you are using numpy correctly you push everything back to C-level again.
Long answer
How can one do that?
I'll show here a little bit what I meant with my comment and how one avoids unnecessary loops with numpy. Let's have a look at the original code assuming you have put all lists into np.arrays with np.asarray(list).
for e in idempotents:
for x in M:
ex = T[e][x]
This translates directly to:
T[idempotents]
Why is that?
Numpy arrays can use arrays of indices for indexing. E.g.
T[0] returns all columns (actually all following dimensions) of matrix T. So T[0]==T[0,:] for 2d arrays. Since you are looping over all idempotents as indices and than over all elements in the columns T[e][x], T[idempotents] is identical to these two loops.
For details about it see here.
ex, es
Next is
for e in idempotents:
for x in M:
ex = T[e][x]
for s in M:
es = T[e][s]
Since there is no point in redoing the entire loop again, this translates to
es=ex
because we are using python, the matrix es is not even copied, just referenced.
exf, esf
I am skipping now some of the for loops in the snippets.
for f in idempotents:
exf = T[ex][f]
esf = T[es][f]
Now you are accessing the outer most index again with the vector of idempotents. So we can do this exactly with numpy in the same way:
T[T[idempotents]]
print T[T[idempotents]].shape
>> (15, 39, 39)
Now, we have an array of dimension (15, 39, 39), because for each element of the 2d array T[idempotents] you return the element of T. This is basically the third loop.
exf = T[T[idempotents]]
esf = exf
from here on it gets more complicated and I will skip the rest. It will be along following lines:
Ti = T[idempotents] # T[e][x] == T[e][s] by loop definition
TTi = T[Ti] # T[T[e][x]]
TTi.shape = -1 , 39 # bring first index back into shape
exf = TTi[:, idempotents] # T[T[e][x]][f]
esf = exf # T[T[e][s]][f] == T[T[e][x]][f] by loop definition
Texf = T[exf].ravel()
exfy = omega[Texf]
TTexf = T.T[exf].ravel() # tesf = omega[T[t][esf]] # since I cannot index fast along t I use the transpose of T
tesf = omega[TTexf]
and so on...

Because Python interprets its input at runtime, whereas the C++ compiler is able to perform major optimizations, particularly on for-loops with fixed sized arrays.
Some compilers might even compute the entire output of the nested for-loops.