Develop Reference

insert python list in all rows new pd.Dataframe column

I have python list:
my_list = [1, 'V']
I have pd.Dataframe:
A B C
0 f v b
1 f i n
2 f i m
I need to create new column in my dataframe with value = my_list:
A B C D
0 f v b [1, 'V']
1 f i n [1, 'V']
2 f i m [1, 'V']
As far as I understand python lists can be values, bc df.groupby with apply "list":
df = df.groupby(['A', 'B'], group_keys=True)['C'].apply(list).reset_index(name='H')
A B H
0 f i [n, m]
1 f v [b]
Its posible without convert my_list type? What the the easiest way to do that?
I tried:
df['D'] = my_list
df['D'] = pd.Series(my_list)
but they did not meet my expectations

You can try using: np.repeat and set its repeat parameter to number of rows which can be found out from the shape of the dataframe.
my_list = [1, 'V']
df = pd.DataFrame({'col1': ['f', 'f', 'f'], 'col2': ['v', 'i', 'i'], 'col3': ['b', 'n', 'm']})
df['new_col'] = np.repeat(my_list, df.shape[0])
This will repeat the values of my_list as many times as there are rows in the DataFrame.

You can do it by creating a new array with my_list through hstack and then forming a new DataFrame. The code below has been tested and works fine.
import numpy as np
import pandas as ph
a1 = np.array([['f','v','b'], ['f','i','n'], ['f','i','m']])
a2 = np.array([1, 'V']).repeat(3).reshape(2,3).transpose()
df = pd.DataFrame(np.hstack((a1,a2)))
Edit: Another code that has been tested is:
import pandas as pd
import numpy as np
a1 = np.array([['f','v','b'], ['f','i','n'], ['f','i','m']])
a2 = np.squeeze(np.dstack((np.array(1).repeat(3), np.array('V').repeat(3))))
df = pd.DataFrame(np.hstack((a1,a2)))

Reassigning numpy.array()

In the code below, I can easily reduce the array ['a','b','a','c','b','b','c','a'] to a binary array [0 1 0 1 1 1 1 0] so that 'a' -> 0 and 'b','c' -> 1. How do I transform it to a ternary array so that 'a' -> 0, 'b' -> 1, 'c' -> 2, without using for and if-else? Thanks.
import numpy as np
x = np.array(['a', 'b', 'a', 'c', 'b', 'b', 'c', 'a'])
y = np.where(x=='a', 0, 1)
print(y)

By doing:
np.where(x == 'a', 0, (np.where(x == 'b', 1, 2)))
note that this changes all the characters that are neither 'a' or 'b' to 2. I've assumed that you have only an array with a,b and c.

A more scalable version is using dictionary of conversion:
my_dict = {'a':0, 'b':1, 'c':2}
x = np.vectorize(my_dict.get)(x)
output:
[0 1 0 2 1 1 2 0]
Another approach is:
np.select([x==i for i in ['a','b','c']], np.arange(3))
For small dictionary #ypno's answer is going to be faster. For larger dictionary, use this answer.
Time Comparison:
Ternary alphabet:
lst = ['a','b','c']
my_dict = {k: v for v, k in enumerate(lst)}
##Ehsan's solution1
def m1(x):
return np.vectorize(my_dict.get)(x)
##ypno's solution
def m2(x):
return np.where(x == 'a', 0, (np.where(x == 'b', 1, 2)))
##SteBog's solution
def m3(x):
y = np.where(x=='a', 0, x)
y = np.where(x=='b', 1, y)
y = np.where(x=='c', 2, y)
return y.astype(np.integer)
##Ehsan's solution 2 (also suggested by user3483203 in comments)
def m4(x):
return np.select([x==i for i in lst], np.arange(len(lst)))
##juanpa.arrivillaga's solution suggested in comments
def m5(x):
return np.array([my_dict[i] for i in x.tolist()])
in_ = [np.random.choice(lst, size = n) for n in [10,100,1000,10000,100000]]
Same analysis for 8 letter alphabet:
lst = ['a','b','c','d','e','f','g','h']

Creating a new column from two columns with apply()

I want to creat a column s['C'] using apply() with a Pandas DataFrame.
My dataset is similiar to this:
[In]:
s=pd.DataFrame({'A':['hello', 'good', 'my', 'pandas','wrong'],
'B':[['all', 'say', 'hello'],
['good', 'for', 'you'],
['so','hard'],
['pandas'],
[]]})
[Out]:
A B
0 hello [all, say, hello]
1 good [good, for, you]
2 my [so, hard]
3 pandas [pandas]
4 wrong []
I need to creat a s['C'] column where the value of each row is a list with ones and zeros dependending if the word of column A is in the list of column B and the position of the element in the list of column B. My output should be like this:
[Out]:
A B C
0 hello [all, say, hello] [0, 0, 1]
1 good [good, for, you] [1, 0, 0]
2 my [so, hard] [0, 0]
3 pandas [pandas] [1]
4 wrong [] [0]
I've been trying with a función and apply, but I still have not realized where is the error.
[In]:
def func(valueA,listB):
new_list=[]
for i in listB:
if listB[i] == valueA:
new_list.append(1)
else:
new_list.append(0)
return new_list
s['C']=s.apply( lambda x: func(x.loc[:,'A'], x.loc[:,'B']))
The error is: Too many indexers
And I also tried with:
[In]:
list=[]
listC=[]
for i in s['A']:
for j in s['B'][i]:
if s['A'][i] == s['B'][i][j]:
list.append(1)
else:
list.append(0)
listC.append(list)
s['C']=listC
The error is: KeyError: 'hello'
Any suggests?

If you are working with pandas 0.25+, explode is an option:
(s.explode('B')
.assign(C=lambda x: x['A'].eq(x['B']).astype(int))
.groupby(level=0).agg({'A':'first','B':list,'C':list})
)
Output:
A B C
0 hello [all, say, hello] [0, 0, 1]
1 good [good, for, you] [1, 0, 0]
2 my [so, hard] [0, 0]
3 pandas [pandas] [1]
4 wrong [nan] [0]
Option 2: Based on your logic, you can do a list comprehension. This should work with any version of pandas:
s['C'] = [[x==a for x in b] if b else [0] for a,b in zip(s['A'],s['B'])]
Output:
A B C
0 hello [all, say, hello] [False, False, True]
1 good [good, for, you] [True, False, False]
2 my [so, hard] [False, False]
3 pandas [pandas] [True]
4 wrong [] [0]

With apply would be
s['c'] = s.apply(lambda x: [int(x.A == i) for i in x.B], axis=1)
s
A B c
0 hello [all, say, hello] [0, 0, 1]
1 good [good, for, you] [1, 0, 0]
2 my [so, hard] [0, 0]
3 pandas [pandas] [1]
4 wrong [] []

I could get your function to work with some minor changes:
def func(valueA, listB):
new_list = []
for i in range(len(listB)): #I changed your in listB with len(listB)
if listB[i] == valueA:
new_list.append(1)
else:
new_list.append(0)
return new_list
and adding the parameter axis = 1 to the apply function
s['C'] = s.apply(lambda x: func(x.A, x.B), axis=1)

Another approach that requires numpy for easy indexing:
import numpy as np
def create_vector(word, vector):
out = np.zeros(len(vector))
indices = [i for i, x in enumerate(vector) if x == word]
out[indices] = 1
return out.astype(int)
s['C'] = s.apply(lambda x: create_vector(x.A, x.B), axis=1)
# Output
# A B C
# 0 hello [all, say, hello] [0, 0, 1]
# 1 good [good, for, you] [1, 0, 0]
# 2 my [so, hard] [0, 0]
# 3 pandas [pandas] [1]
# 4 wrong [] []

Implementing k nearest neighbours from distance matrix?

I am trying to do the following:
Given a dataFrame of distance, I want to identify the k-nearest neighbours for each element.
Example:
A B C D
A 0 1 3 2
B 5 0 2 2
C 3 2 0 1
D 2 3 4 0
If k=2, it should return:
A: B D
B: C D
C: D B
D: A B
Distances are not necessarily symmetric.
I am thinking there must be something somewhere that does this in an efficient way using Pandas DataFrames. But I cannot find anything?
Homemade code is also very welcome! :)
Thank you!

The way I see it, I simply find n + 1 smallest numbers/distances/neighbours for each row and remove the 0, which would then give you n numbers/distances/neighbours. Keep in mind that the code will not work if you have a distance of zeroes! Only the diagonals are allowed to be 0.
import pandas as pd
import numpy as np
X = pd.DataFrame([[0, 1, 3, 2],[5, 0, 2, 2],[3, 2, 0, 1],[2, 3, 4, 0]])
X.columns = ['A', 'B', 'C', 'D']
X.index = ['A', 'B', 'C', 'D']
X = X.T
for i in X.index:
Y = X.nsmallest(3, i)
Y = Y.T
Y = Y[Y.index.str.startswith(i)]
Y = Y.loc[:, Y.any()]
for j in Y.index:
print(i + ": ", list(Y.columns))
This prints out:
A: ['B', 'D']
B: ['C', 'D']
C: ['D', 'B']
D: ['A', 'B']

Creating a subset of array from another array : Python

I have a basic question regarding working with arrays:
a= ([ c b a a b b c a a b b c a a b a c b])
b= ([ 0 1 0 1 0 0 0 0 2 0 1 0 2 0 0 1 0 1])
I) Is there a short way, to count the number of time 'c' in a corresponds to 0, 1, and 2 in b and 'b' in a corresponds to 0, 1, 2 and so on
II) How do I create a new array c (subset of a) and d(subset of b) such that it only contains those elements if the corresponding element in a is 'c' ?

In [10]: p = ['a', 'b', 'c', 'a', 'c', 'a']
In [11]: q = [1, 2, 1, 3, 3, 1]
In [12]: z = zip(p, q)
In [13]: z
Out[13]: [('a', 1), ('b', 2), ('c', 1), ('a', 3), ('c', 3), ('a', 1)]
In [14]: counts = {}
In [15]: for pair in z:
...: if pair in counts.keys():
...: counts[pair] += 1
...: else:
...: counts[pair] = 1
...:
In [16]: counts
Out[16]: {('a', 1): 2, ('a', 3): 1, ('b', 2): 1, ('c', 1): 1, ('c', 3): 1}
In [17]: sub_p = []
In [18]: sub_q = []
In [19]: for i, element in enumerate(p):
...: if element == 'a':
...: sub_p.append(element)
...: sub_q.append(q[i])
In [20]: sub_p
Out[20]: ['a', 'a', 'a']
In [21]: sub_q
Out[21]: [1, 3, 1]
Explanation
zip takes two lists and runs a figurative zipper between them. Resulting in a list of tuples
I've used a simplistic approach, I'm just maintaining a map/dictionary that makes not of how many times it has seen a pair of char-int tuples
Then I make 2 sub lists that you can modify to use the character in question and figure out what it maps to
Alternative methods
As abarnert suggested you could use A Counter from collections instead.
Or you could just a count method on z . eg: z.count('a',1). Or you can use a defaultdict instead.

The questions are a bit vague but here's a quick method (some would call it dirty) using Pandas though I think something written without recourse to Pandas should be preferred.
import pandas as pd
#create OP's lists
a= [ 'c', 'b', 'a', 'a', 'b', 'b', 'c', 'a', 'a', 'b', 'b', 'c', 'a', 'a', 'b', 'a', 'c', 'b']
b= [ 0, 1, 0, 1, 0, 0, 0, 0, 2, 0, 1, 0, 2, 0, 0, 1, 0, 1]
#dump lists to a Pandas DataFrame
df = pd.DataFrame({'a':a, 'b':b})
Question 1
provided I interpreted it correctly, you can cross-tabulate the two arrays:
pd.crosstab(df.a, df.b).stack(). Cross-tabulate basically counts the number of times each number corresponds to a particular letter. .stack is a command to turn output from .crosstab into a more legible format.
#question 1
pd.crosstab(df.a, df.b).stack()
## -- End pasted text --
Out[9]:
a b
a 0 3
1 2
2 2
b 0 4
1 3
2 0
c 0 4
1 0
2 0
dtype: int64
Question 2
Here, I use Pandas boolean indexing ability to only select the elements in array a that correspond to value 'c'. So df.a=='c' will return True for every value in a that is 'c' and False otherwise. df.loc[df.a=='c','a'] will return values from a for which the boolean statement was true.
c = df.loc[df.a == 'c', 'a']
d = df.loc[df.a == 'c', 'b']
In [15]: c
Out[15]:
0 c
6 c
11 c
16 c
Name: a, dtype: object
In [16]: d
Out[16]:
0 0
6 0
11 0
16 0
Name: b, dtype: int64

Python List : https://www.tutorialspoint.com/python/python_lists.htm has a count method.
I suggest you to first zip both lists, as said in comments, and then count occurances of tuple c, 1 and occurances of tuple c, 0 and sum them up, thats what you need for (I), basically.
For (II), if I understood you correctly, you have to take the zipped lists and apply filter on them with lambda x: x[0]==x[1]