Develop Reference

Correlation analysis with multiple data in a single cell

I have a dataset with some rows containing singular answers and others having multiple answers. Like so:
year length Animation
0 1971 121 1,2,3
1 1939 71 1,3
2 1941 7 0,2
3 1996 70 1,2,0
4 1975 71 3,2,0
With the singular answers I managed to create a heatmap using df.corr(), but I can't figure out what is the best approach for multiple answers rows.
I could split them and add additional columns for each answer like:
year length Animation
0 1971 121 1
1 1971 121 2
2 1971 121 3
3 1939 71 1
4 1939 71 3 ...
and then do the exact same dr.corr(), or add additional Animation_01, Animation_02 ... columns, but there must be a smarter way to work around this issue?
EDIT: Actual data snippet

You should compute a frequency table between two categorical variables using pd.crosstab() and perform subsequent analyses based on this table. df.corr(x, y) is NOT mathematically meaningful when one of x and y is categorical, no matter it is encoded into number or not.
N.B.1 If x is categorical but y is numerical, there are two options to describe the linkage between them:
Group y into quantiles (bins) and treat it as categorical
Perform a linear regression of y against one-hot encoded dummy variables of x
Option 2 is more precise in general but the statistics is beyond the scope of this question. This post will focus on the case of two categorical variables.
N.B.2 For sparse matrix output please see this post.
Sample Solution
Data & Preprocessing
import pandas as pd
import io
import matplotlib.pyplot as plt
from seaborn import heatmap
df = pd.read_csv(io.StringIO("""
year length Animation
0 1971 121 1,2,3
1 1939 71 1,3
2 1941 7 0,2
3 1996 70 1,2,0
4 1975 71 3,2,0
"""), sep=r"\s{2,}", engine="python")
# convert string to list
df["Animation"] = df["Animation"].str.split(',')
# expand list column into new rows
df = df.explode("Animation")
# (optional)
df["Animation"] = df["Animation"].astype(int)
Frequency Table
Note: grouping of length is ignored for simplicity
ct = pd.crosstab(df["Animation"], df["length"])
print(ct)
# Out[65]:
# length 7 70 71 121
# Animation
# 0 1 1 1 0
# 1 0 1 1 1
# 2 1 1 1 1
# 3 0 0 2 1
Visualization
ax = heatmap(ct, cmap="viridis",
yticklabels=df["Animation"].drop_duplicates().sort_values(),
xticklabels=df["length"].drop_duplicates().sort_values(),
)
ax.set_title("Title", fontsize=20)
plt.show()
Example Analysis
Based on the frequency table, you can ask questions about the distribution of y given a certain (subset of) x value(s), or vice versa. This should better describe the linkage between two categorical variables, as the categorical variables have no order.
For example,
Q: What length does Animation=3 produces?
A: 66.7% chance to give 71
33.3% chance to give 121
otherwise unobserved

You want to break Animation (or Preferred_positions in your data snippet) up into a series of one-hot columns, one one-hot column for every unique string in the original column. Every column with have values of either zero or one, one corresponding to rows where that string appeared in the original column.
First, you need to get all the unique substrings in Preferred_positions (see this answer for how to deal with a column of lists).
positions = df.Preferred_positions.str.split(',').sum().unique()
Then you can create the positions columns in a loop based on whether the given position is in Preferred_positions for each row.
for position in positions:
df[position] = df.Preferred_positions.apply(
lambda x: 1 if position in x else 0
)

creating new dataframe with manhattan distance in python

I need to create a dataframe containing the manhattan distance between two dataframes with the same columns, and I need the indexes of each dataframe to be the index and column name, so for example lets say I have these two dataframes:
x_train :
index a b c
11 2 5 7
23 4 2 0
312 2 2 2
x_test :
index a b c
22 1 1 1
30 2 0 0
so the columns match but the size and indexes do not, the expected dataframe would look like this:
dist_dataframe:
index 11 23 312
22 11 5 3
30 12 4 4
and what I have right now is this:
def manhattan_distance(a, b):
return sum(abs(e1-e2) for e1, e2 in zip(a,b))
def calc_distance(X_test,X_train):
dist_dataframe = pd.DataFrame(index=X_test.index,columns = X_train.index)
for i in X_train.index:
for j in X_test.index:
dist_dataframe.loc[i,j]=manhattan_distance(X_train.loc[[i]],X_test.loc[[j]])
return dist_dataframe
what I get from the code I have is this dataframe:
dist_dataframe:
index
index 11 23 312
22 NaN NaN NaN
30 NaN NaN NaN
I get the right dataframe size except that it has 2 rows called indexes that I get from the creation of the new dataframe, and also I get an error no matter what I do in the manhattan calculation line, can anyone help me out here please?

Problem in your code
There is a very small problem in your code, i.e. accessing values in dist_dataframe. So,instead of dist_dataframe.loc[i,j], you should reverse the order of i and j and make it like dist_dataframe.loc[j,i]
More efficient solution
It will work fine but since you are a new contributor, I would also like to point out the efficiency of your code. Always try to replace loops with pandas in-built functions. Since they are written in C, it makes them much faster. So here is a more efficient solution:
def manhattan_distance(a, b):
return sum(abs(e1-e2) for e1, e2 in zip(a,b))
def xtrain_distance(row):
distances = {}
for i,each in x_train.iterrows():
distances[i] = manhattan_distance(each,row)
return distances
result = x_test.apply(xtrain_distance, axis=1)
# converting into dataframe
pd.DataFrame(dict(result)).transpose()
It also produces same output and on your example and you can't see any time difference. But when run on a larger size (same data scaled over 20 times), i.e. 60 x_train samples and 40 x_test samples, here is the time difference:
Your solution took: 929 ms
This solution took: 207 ms
It got 4x faster just by eliminating one for loop. Note that, it can be made more efficient but for the sake of demonstration, I have used this solution.

how to plot the variation of feature

I have dataset with 2 columns and I would like to show the variation of one feature according to the binary output value
data
id Column1 output
1 15 0
2 80 1
3 120 1
4 20 0
... ... ...
I would like to drop a plot with python where x-axis contains values of Column1 and y-axis contains the percent of getting positive values.
I know already that the form of my plot have the form of exponontial function where when column1 has smaller numbers I will get more positive output then when it have long values

exponential plot maybe need two list like this
try this
import matplotlib.pyplot as plt
# x-axis points list
xL = [5,10,15,20,25,30]
# y-axis points list
yL = [100,50,25,12,10]
plt.plot(xL, yL)
plt.axis([0, 35, 0, 200])
plt.show()

pandas plotting skipping xtick labels

I am trying to display two pandas Series objects together, which works, except all the labels are not displayed.
I am trying to plot the two Series together like this:
plt.figure()
sns.set_style('ticks')
ts86['Gene'].value_counts().plot(kind='area')
l97['Gene'].value_counts().plot(kind='area')
sns.despine(offset=10)
But only one of the indexes is displayed.
Here are the two Series that I have:
one
TIIIh 25
TET2-2 24
IDH2 15
TIIIa 14
TIIIb 12
TIIIj 11
TIIIp 9
p53-1 9
SF3B1 8
TIIIe 8
KRAS-1 7
TIIIo 6
TIIId 6
TET2-1 6
GATA1 5
p53-3 5
HRAS 5
NRAS-2 4
IDH1 4
TIIIq 4
JAK2 4
TIIIc 4
TIIIf 3
TIIIg 3
TIIIm 3
KRAS-2 3
p53-2 3
TIIIk 3
TIIIn 2
DNMT3a 1
and
two
p53-1 17
p53-2 2
NRAS-2 2
p53-3 1
KRAS-2 1

Your output graph shows value_counts of 2 dataframes but obviously the index orders are no longer the same, so there is no way to show xticks at this point (e.g. highest count in df1 is TIIIh while that of df2 is p53-1 and you are trying to plot them together by preserving the order).
Let's simply merge df1 and df2 first (I named TIIIh and so on as id for merge key):
combi = pd.merge(ts86, l97, on='id', how='left')
combi = combi.set_index('id')
And then, plot each column and show all xticks:
ax = combi['Gene_x'].plot(kind='area', figsize=(10, 3))
combi['Gene_y'].plot(kind='area', figsize=(10, 3))
ax.set_xticks(range(combi.shape[0]))
ax.set_xticklabels(combi.index, rotation=90)
Now you get this:
Hope this helps.

How to sum previous and next row values in pandas

After trying several hours i am still not able to perform the following task.
I would like to sum to my center points the previous and next row value as shown in the image below
Can you please provide me with an example how that can be done?
Thank you in advance for your time!

You can also use DF.rolling.sum() by providing center=True (Since by default the labels are set to the right edge of the window) and then take every third slice from it. Additionally, you can set the minimum number of observations, min_periods to be equal to 1 which basically says no output values will be set until at least min_periods non-null values are encountered.
df.A.rolling(window=3, min_periods=1, center=True).sum().iloc[::3].astype(int)
1 30
4 120
7 210
10 190
Name: A, dtype: int32

This will get it done
df = pd.DataFrame(dict(A=np.arange(10, 101, 10)), np.arange(1, 11))
pd.Series(np.convolve(df.A.values, [1, 1, 1])[1::3], df.index[0::3])
1 30
4 120
7 210
10 190
dtype: int64
​