Develop Reference

Can multiple table be formed from a condition in dataframe in python?

I have a dataset:
list1 list2
0 [1,3,4] [4,3,2]
1 [1,3,2] [0,4,6]
2 [4,5,8] NA
3 [6,3,7] [8,2,3]
Is there a process where i can find the count of the common term for- each of the index,
Expected output:
intersection_0, it will compare 0 of list1 with each of list2 and give output, intersection_1 which will compare 1 of list1 with each of list2
Expected_output:
Intersection_0 intersection_1 intersection_2 intersection_3
1 2 1 1
1 0 1 1
0 0 0 0
1 2 0 1
For intersection i was trying:
df['intersection'] = [len(set(a).intersection(b)) for a, b in zip(df1.list1, df1.list2)]
Is there a better way or faster way to achieve this? Thank you in advance

The double loop would go like this:
intersections = []
for l2 in df['list2']:
intersection = []
for l1 in df['list1']:
try:
i = len(np.intersect1d(l1,l2))
except:
i = 0
intersection.append(i)
intersections.append(intersection)
out = (pd.DataFrame(intersections))
Output:
0 1 2 3
0 2 2 1 1
1 1 0 1 1
2 0 0 0 0
3 1 2 1 1

How do a update a variable used to generate the value inside of a list comprehension?

I currently have a loop that looks like this:
payoffs =[[0 1 1 0 1 0 0 1 0 1 0 0 1 1 1 0 0 0 0 0 1 0 1 0 1 0 1 1 1 0]
[1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1]]
n= 30
actions = [0 for _ in range(n)]
for i in range(n):
actions[i] = some_function(n, cumulative_payouts)
cumulative_payoffs += payoffs[:, i]
Is there a more pythonic way to do this with list comprehensions, that allows you to update cumulative_payoffs alongside the loop?

Here's one way:
actions = []
for payoff in payoffs.T:
actions.append(self.generate_action(n=n, cumulative_payoffs=cumulative_payoffs))
cumulative_payoffs += payoff
This can be condensed in Python 3.8 using an assignment expression, but readability suffers and I'm not personally a fan:
actions = [
self.generate_action(
n=n,
cumulative_payoffs=(cumulative_payoffs := cumulative_payoffs + payoff)
)
for payoff in payoffs.T
]
There's probably a good numpy way to create an array of cumulative_payoffs without a loop, and then a list comprehension becomes simpler.

is there a 2D dictionary in python?

I was about to create a matrix like :
33 12 23 42 11 32 43 22
33 − 1 1 1 0 0 1 1
12 1 − 1 1 0 0 1 1
23 1 1 − 1 1 1 0 0
42 1 1 1 − 1 1 0 0
11 0 0 1 1 − 1 1 1
32 0 0 1 1 1 − 1 1
43 1 1 0 0 1 1 − 1
22 1 1 0 0 1 1 1 −
I want to query by horizontal or vertical titles, so I created the matrix by：
a = np.matrix('99 33 12 23 42 11 32 43 22;33 99 1 1 1 0 0 1 1;12 1 99 1 1 0 0 1 1;23 1 1 99 1 1 1 0 0;42 1 1 1 99 1 1 0 0;11 0 0 1 1 99 1 1 1;32 0 0 1 1 1 99 1 1;43 1 1 0 0 1 1 99 1;22 1 1 0 0 1 1 1 99')
I want to have the certain data if I query a[23][11] = 1
so is there a way we can create a 2D dictionary, so that a[23][11] = 1?
Thanks

You're clearly asking for something outside of numpy.
A defauldict with the default_factory as dict gives a sense of the 2D dictionary you want:
>>> from collections import defaultdict
>>> a = defaultdict(dict)
>>> a[23][11] = 1
>>> a[23]
{11: 1}
>>> a[23][11]
1

Another possibility is to use tuples as the dictionary keys
dict((33,12):1, (23,12):1, ...]
scipy.sparse has a sparse matrix format that stores it's values in such a dictionary. With your values such a matrix would represent a 50x50 matrix with mostly 0 values, and just 1's at these selected coordinates.
Keep in mind that the keys of a dictionary (ordinary at least) are not ordered
What are going to be doing with this data? A dictionary, whether type or nested, is good for one kind of usage, but bad for others. A matrix such as you sample is better for other things, like operations along rows or columns. The dictionary format largely obscures that kind of ordered layout.

Are you looking for a dictionary with pairs as keys?
d = {}
d[33, 12] = 1
d[33, 23] = 1
# etc
Note that in python d[a, b] is just syntactic sugar for d[(a, b)]

If I understand correctly you just want to label your row/columns. To stay within the numpy array framework, a simple solution would be to create a mapping between the labels and the array order. I am also going to assume that it is OK to convert the labels into strings as they can be anything (though integers would also be fine).
l = {str(x) : ind for ind , x in enumerate((33,12,23,42,11,32,43,22))}
a = sp.linalg.circulant([99,1,1,1,0,0,1,1])
a[l['32'],l['23']]

Dynamic For Loops in Python

i understand that to create dynamic for loops, recursive or itertools module in python is the way to go. Lets say I am doing it in recursive.
What I want is
for var1 in range(var1_lowerlimit, var1_upperlimit, var1_stepsize):
for var2 in range(var2_lowerlimit, var2_upperlimit, var2_stepsize):
:
:
# do_whatever()
repeat for n loops where n is the number of variables
What I have now is I have 2 lists
variable_list = [ var1, var2, var3, ... ]
boundaries_list = [ [var1_lowerlimit, var1_upperlimit, var1_stepsize],
[var2_lowerlimit, var2_upperlimit, var2_stepsize], ...]
def dynamic_for_loop(variable_list , boundaries_list, no_of_loops, list_index = 0):
if no_of_loops <= 0:
# do_whatever()
else:
lower_bound = boundaries_list[list_index][0]
upper_bound = boundaries_list[list_index][1]
step_size = boundaries_list[list_index][2]
for index in range(lower_bound, upper_bound, step_size):
list_index += 1
try:
dynamic_for_loop(variable_list , boundaries_list, no_of_loops - 1, list_index)
except:
list_index = 0
dynamic_for_loop(variable_list , boundaries_list, no_of_loops - 1, list_index)
I did a reset on list_index as it gets out of range, but i couldn't get the result I want. Can someone enlighten me what went wrong?

Use the itertools.product() function to generate the values over a variable number of ranges:
for values in product(*(range(*b) for b in boundaries_list)):
# do things with the values tuple, do_whatever(*values) perhaps
Don't try to set a variable number of variables; just iterate over the values tuple or use indexing as needed.
Using * in a call tells Python to take all elements of an iterable and apply them as separate arguments. So each b in your boundaries_list is applied to range() as separate arguments, as if you called range(b[0], b[1], b[2]).
The same applies to the product() call; each range() object the generator expression produces is passed to product() as a separate argument. This way you can pass a dynamic number of range() objects to that call.

Just for fun, I thought that I would implement this using recursion to perhaps demonstrate the pythonic style. Of course, the most pythonic way would be to use the itertools package as demonstrated by Martijn Pieters.
def dynamic_for_loop(boundaries, *vargs):
if not boundaries:
print(*vargs) # or whatever you want to do with the values
else:
bounds = boundaries[0]
for i in range(*bounds):
dynamic_for_loop(boundaries[1:], *(vargs + (i,)))
Now we can use it like so:
In [2]: boundaries = [[0,5,1], [0,3,1], [0,3,1]]
In [3]: dynamic_for_loop(boundaries)
0 0 0
0 0 1
0 0 2
0 1 0
0 1 1
0 1 2
0 2 0
0 2 1
0 2 2
1 0 0
1 0 1
1 0 2
1 1 0
1 1 1
1 1 2
1 2 0
1 2 1
1 2 2
2 0 0
2 0 1
2 0 2
2 1 0
2 1 1
2 1 2
2 2 0
2 2 1
2 2 2
3 0 0
3 0 1
3 0 2
3 1 0
3 1 1
3 1 2
3 2 0
3 2 1
3 2 2
4 0 0
4 0 1
4 0 2
4 1 0
4 1 1
4 1 2
4 2 0
4 2 1
4 2 2

Python nested looping Idiom

I often find myself doing this:
for x in range(x_size):
for y in range(y_size):
for z in range(z_size):
pass # do something here
Is there a more concise way to do this in Python? I am thinking of something along the lines of
for x, z, y in ... ? :

You can use itertools.product:
>>> for x,y,z in itertools.product(range(2), range(2), range(3)):
... print x,y,z
...
0 0 0
0 0 1
0 0 2
0 1 0
0 1 1
0 1 2
1 0 0
1 0 1
1 0 2
1 1 0
1 1 1
1 1 2

If you've got numpy as a dependency already, numpy.ndindex will do the trick ...
>>> for x,y,z in np.ndindex(2,2,2):
... print x,y,z
...
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1

Use itertools.product():
import itertools
for x, y, z in itertools.product(range(x_size), range(y_size), range(z_size)):
pass # do something here
From the docs:
Cartesian product of input iterables.
Equivalent to nested for-loops in a generator expression.
...

It depends on what is inside the loop. If dealing with lists, you may be able to use a list comprehension
For the more general case, see this post on itertools.