Develop Reference

Merge two DataFrames by combining duplicates and concatenating nonduplicates

I have two DataFrames:
df = pd.DataFrame({'A':[1,2],
'B':[3,4]})
A B
0 1 3
1 2 4
df2 = pd.DataFrame({'A':[3,2,1],
'C':[5,6,7]})
A C
0 3 5
1 2 6
2 1 7
and I want to merge in a way that the column 'A' add the different values between DataFrames but merge the duplicates.
Desired output:
A B C
0 3 NaN 5
1 2 4 6
2 1 3 7

You can use combine_first:
df2 = df2.combine_first(df)
Output:
A B C
0 1 3.0 5
1 2 4.0 6
2 3 NaN 7

How to make an one-to-one merge on pandas dataframe

Let's say df1 looks like:
id x
a 1
b 2
b 3
c 4
and df2 looks like:
id y
b 9
b 8
How do I merge them so that the output is:
id x y
b 2 9
b 3 8
I've tried pd.merge(df1, df2, on='id') but it is giving me:
id x y
b 2 9
b 2 8
b 3 9
b 3 8
which is not what I want.

IIUC, GroupBy.cumcount + merge
new_df = (df1.assign(count=df1.groupby('id').cumcount())
.merge(df2.assign(count=df2.groupby('id').cumcount()),
on=['id', 'count'], how='inner')
.drop(columns='count'))
id x y
0 b 2 9
1 b 3 8

Addition-merging dataframes

What is the best way to add the contents of two dataframes, which have mostly equivalent indices:
df1:
A B C
A 0 3 1
B 3 0 2
C 1 2 0
df2:
A B C D
A 0 1 1 0
B 1 0 3 2
C 1 3 0 0
D 0 2 0 0
df1 + df2 =
A B C D
A 0 4 2 0
B 4 0 5 2
C 2 5 0 0
D 0 2 0 0

You can also concat both the dataframes since concatenation (by default) happens by index.
# sample dataframe
df1 = pd.DataFrame({'a': [1,2,3], 'b':[2,3,4]}, index=['a','c','e'])
df2 = pd.DataFrame({'a': [10,20], 'b':[11,22]}, index=['b','d'])
new_df= pd.concat([df1, df2]).sort_index()
print(new_df)
a b
a 1 2
b 10 11
c 2 3
d 20 22
e 3 4

I think you can just add:
In [625]: df1.add(df2,fill_value=0)
Out[625]:
A B C D
A 0.0 4.0 2.0 0.0
B 4.0 0.0 5.0 2.0
C 2.0 5.0 0.0 0.0
D 0.0 2.0 0.0 0.0

Variable shift in Pandas

having two columns A and B in a dataframe:
A B
0 1 6
1 2 7
2 1 8
3 2 9
4 1 10
I would like to create a column C. C must have values of B shifted by value of A:
A B C
0 1 6 NaN
1 2 7 NaN
2 1 8 7
3 2 9 7
4 1 10 9
The command:
df['C'] = df['B'].shift(df['A'])
does not work.
Do you have any other ideas?

I'd use help from numpy to avoid the apply
l = np.arange(len(df)) - df.A.values
df['C'] = np.where(l >=0, df.B.values[l], np.nan)
df
A B C
0 1 6 NaN
1 2 7 NaN
2 1 8 7.0
3 2 9 7.0
4 1 10 9.0
simple time test

This is tricky due to index alignment, you can define a user func and apply row-wise on your df, here the function will perform a shift on the B column and return the index value (using .name attribute to return the index) of the shifted column:
In [134]:
def func(x):
return df['B'].shift(x['A'])[x.name]
df['C'] = df.apply(lambda x: func(x), axis=1)
df
Out[134]:
A B C
0 1 6 NaN
1 2 7 NaN
2 1 8 7.0
3 2 9 7.0
4 1 10 9.0

Pandas: solve a crosstab issue

I have a situation where a user belongs to multiple categories:
UserID Category
1 A
1 B
2 A
3 A
4 C
2 C
4 A
A = 1,2,3,4
B = 1
C = 2,4
I want the crosstab which shows data like this using pandas:
A B C
A 4 1 2
B 1 2 0
C 2 0 2
I try:
df.groupby(UserID).agg(countDistinct('Category'))
I did the above but it returns 0 for elements not on the diagonal.

You can first create DataFrame from lists a, b, c. Then stack and merge it to original. Last use crosstab:
a = [1,2,3,4]
b = [1]
c = [2,4]
df1 = pd.DataFrame({'A':pd.Series(a), 'B':pd.Series(b), 'C':pd.Series(c)})
print (df1)
A B C
0 1 1.0 2.0
1 2 NaN 4.0
2 3 NaN NaN
3 4 NaN NaN
df2 = df1.stack()
.reset_index(drop=True, level=0)
.reset_index(name='UserID')
.rename(columns={'index':'newCat'})
print (df2)
newCat UserID
0 A 1.0
1 B 1.0
2 C 2.0
3 A 2.0
4 C 4.0
5 A 3.0
6 A 4.0
df3 = pd.merge(df, df2, on='UserID')
print (pd.crosstab(df3.newCat, df3.Category))
Category A B C
newCat
A 4 1 2
B 1 1 0
C 2 0 2