I have a pandas dataframe of over 1000 lines that looks somewhat like this:
Copy name type ntv
G1 BA X 0.45
G1 BB X 0.878
G1 C Z 0.19
G1 LA1 Y 1.234
G1 L Y 0.09
G1 LB Y 1.056
F2 BA1 X -7.890
F2 BB X 2.345
F2 MA Y -0.871
F2 LB1 Y 0.737
In the example above (df1), there are two sets of the 'Copy' column, G1 and F2, with various names, and three types X,Y and Z.
I would like to create another data frame (df2) that looks like the one below, where they are grouped together in the form X-Y or Z-Y.
Model ntv_1 ntv_2
G1BA-LA1 0.45 1.234
G1BB-LB 0.878 1.056
G1C-L 0.19 0.09
F2BA1-MA -7.890 -0.871
F2BB-LB1 2.345 0.737
For group X-Y, they have the second character of df1['name'] in common. So, I decided to approach it this way:
c = df1[(df1['name'].str[0]=='B' & (df1['ntv'] != 0.0)]
h = df1[((df1['name'].str[0]=='L')|(df1['name'].str[0]=='M')) & (df['ntv'] != 0.0)]
b = (c.loc[:,c['name'].str[1]] == h.loc[:,h['name'].str[1]]).groupby('Copy')
df2['Model'] = c['Copy'].astype(str) + c['name'].astype(str) + '-' + h['name'].astype(str)
df2['ntv_1'] = c['ntv']
df2['ntv_2'] = h['ntv']
I got a KeyError message. So I decided to do this:
ca = c['name'].str[1].dropna()
ha = h['name'].str[1].dropna()
if ca == ha:
df2['Model'] = c['Copy'].astype(str) + c['name'].astype(str) + '-' + h['name'].astype(str)
df2['ntv_1'] = c['ntv']
df2['ntv_2'] = h['ntv']
But I got a ValueError: "Series length must match to compare."
Please how can I group the dataframe into the form X-Y or Z-Y? Thanks in advance!
There is problem c and h are not aligned, because different indices and possible different lenght:
#added condition for remove all rows with no second value in name
c = df1[(df1['name'].str[0]=='B') & (df1['ntv'] != 0.0) &
(df1['name'].str[1].notnull())].copy()
#created MultiIndex for align with Counter duplicates
ca = c['name'].str[1]
c.index = [ca, c.groupby(ca).cumcount()]
#added condition for remove all rows with no second value in name
h = df1[((df1['name'].str[0]=='L')|(df1['name'].str[0]=='M')) &
(df1['ntv'] != 0.0) & (df1['name'].str[1].notnull())].copy()
#created MultiIndex for align with Counter duplicates
ha = h['name'].str[1]
h.index = [ha, h.groupby(ha).cumcount()]
print (c)
copy name type ntv
name
A 0 G1 BA X 0.450
B 0 G1 BB X 0.878
A 1 F2 BA1 X -7.890
B 1 F2 BB X 2.345
print (h)
copy name type ntv
name
A 0 G1 LA1 Y 1.234
B 0 G1 LB Y 1.056
A 1 F2 MA Y -0.871
B 1 F2 LB1 Y 0.737
#join together DataFrames
df2 = pd.concat([c, h.add_suffix('_2')], axis=1)
#with real data is possible data are not aligned and get NaNs
#for remove all NaNs rows use
#df2 = df2.dropna()
df2['Model'] = df2['copy'].astype(str)+df2['name'].astype(str)+'-'+ df2['name_2'].astype(str)
#filter columns and remove MultiIndex
df2 = df2[['Model','ntv','ntv_2']].reset_index(drop=True)
print (df2)
Model ntv ntv_2
0 G1BA-LA1 0.450 1.234
1 G1BB-LB 0.878 1.056
2 F2BA1-MA -7.890 -0.871
3 F2BB-LB1 2.345 0.737
Related
variable value
G1 1
G1 2
G2 3
G2 4
G3 5
G3 6
How can I calculate sum of squared between groups ? For example;
SSB = ((G1)**2/n +(G2)**2/n + (G3)**2/n) - (21)**2/6
I'm interpreting your formula as wanting the following:
SSB = (
sum(G1)**2/len(G1) +
sum(G2)**2/len(G2) +
sum(G3)**2/len(G3) +
...
) - sum(Gall)**2/len(Gall)
I assume df is your DataFrame:
Gn = df.groupby('variable').value.agg(['sum', 'size'])
Gall = df.value.agg(['sum', 'size'])
SSB = (Gn['sum']**2/Gn['size']).sum() - Gall['sum']**2/Gall['size']
If I understand correctly, then SSB should contain the result you are interested in.
My problem is kind of a bit tricky (similar to sql merge/update), and not understanding how to fix: ( I am giving a small sample of the dataframes below)
I have two dataframes :
dfOld
A B C D E
x1 x2 g h r
q1 q2 x y s
t1 t2 h j u
p1 p2 r s t
AND
dfNew
A B C D E
x1 x2 a b c
s1 s2 p q r
t1 t2 h j u
q1 q2 x y z
We want to merge the dataframes with the following rule : ( we can think Col A & ColB as keys)
For any ColA & ColB combination if C/D/E are exact match then it takes value from any dataframe, however if any value has changed in Col C/D/E , it takes the value from new dataframe and if a new ColA/Col B combination is in DfNew then it takes those values and if the ColA/ColB combination does not exist in dfNew then it takes the value from dfOld:
So my OutPut should be like:
A B C D E
x1 x2 a b c
q1 q2 x y z
t1 t2 h j u
p1 p2 r s t
s1 s2 p q r
I was trying :
mydfL = (df.merge(df1,indicator = True, how='left').loc[lambda x : x['_merge']!='both'])
mydfR = (df1.merge(df,indicator = True, how='left').loc[lambda x : x['_merge']!='both'])
dfO = pd.concat([mydfL,mydfR])
dfO.drop("_merge", axis=1, inplace=True)
My output looks like: ( I kept the index for clarity)
A B C D E
0 x1 x2 a b c
2 s1 s2 p q r
3 q1 q2 x y z
0 x1 x2 g h r
2 q1 q2 x y s
3 p1 p2 r s t
However, this output does not serve my purpose. First and foremost it does not include the totally identical row (between dfOld & dfnew) which consists of :
t1 t2 h j u
and next it includes all the rows where for the ColA/Col x, y and q1, q2, where I just wanted the updated values in ColC/D/E in the new data frame ( dfNew). It includes data from both.
So can I get some help as to what am I missing and what may be a better and elegant way to do this. Thanks in advance.
You can use combine_first using A/B as temporary index:
out = (dfNew.set_index(['A', 'B'])
.combine_first(dfOld.set_index(['A', 'B']))
.reset_index()
)
I want to map values based how two columns are matched. For instance, the df below contains different labels, A or B. I want to assign a new column that describes these labels. How this occurs is comparing columns Z L and Z P. Z L will always contain either ['X1','X2','X3','X4']. While Z P will correspondingly contain ['LA','LB','LC','LD'].
These will always be in acceding order or reverse order. As in ascending order will mean X1 corresponds to LA, X2 corresponds to LB etc. Reverse order means X1 corresponds to LD, X2 corresponds to LC etc.
If ascending order I want to map an R. If reverse order I want to map an L.
X = ['X1','X2','X3','X4']
R = ['LA','LB','LC','LD']
L = ['LD','LC','LB','LA']
df = pd.DataFrame({
'Period' : [1,1,1,1,1,2,2,2,2,2],
'labels' : ['A','B','A','B','A','B','A','B','A','B'],
'Z L' : [np.nan,np.nan,'X3','X2','X4',np.nan,'X2','X3','X3','X1'],
'Z P' : [np.nan,np.nan,'LC','LC','LD',np.nan,'LC','LC','LB','LA'],
})
df = df.dropna()
This is the output dataset to determine the combinations. I have a large df with repeated combinations so I'm not too concerned with returning all of them. I'm mainly concerned with all unique Mapped values for each Period.
Period labels Z L Z P
2 1 A X3 LC
3 1 B X2 LC
4 1 A X4 LD
6 2 A X2 LC
7 2 B X3 LC
8 2 A X3 LB
9 2 B X1 LA
Attempt:
labels = df['labels'].unique().tolist()
I = df.loc[df['labels'] == labels[0]]
J = df.loc[df['labels'] == labels[1]]
I['Map'] = ((I['Z L'].isin(X)) | (I['Z P'].isin(R))).map({True:'R', False:'L'})
J['Map'] = ((J['Z L'].isin(X)) | (J['Z P'].isin(R))).map({True:'R', False:'L'})
If I drop duplicates from period and labels the intended df is:
Period labels Map
0 1 A R
1 1 B L
2 2 A L
3 2 B R
Here's my approach:
# the ascending orders
lst1,lst2 = ['X1','X2','X3','X4'], ['LA','LB','LC','LD']
# enumerate the orders
d1, d2 = ({v:k for k,v in enumerate(l)} for l in (lst1, lst2))
# check if the enumerations in `Z L` and `Z P` are the same
df['Map'] = np.where(df['Z L'].map(d1)== df['Z P'].map(d2), 'R', 'L')
Output:
Period labels Z L Z P Map
2 1 A X3 LC R
3 1 B X2 LC L
4 1 A X4 LD R
6 2 A X2 LC L
7 2 B X3 LC R
8 2 A X3 LB L
9 2 B X1 LA R
and df.drop_duplicates(['Period', 'labels']):
Period labels Z L Z P Map
2 1 A X3 LC R
3 1 B X2 LC L
6 2 A X2 LC L
7 2 B X3 LC R
You said your data is always either in ascending or reversed order. You only need to define a fix mapping between Z L and Z P as the R and check on this mapping. If True it is R, else L. I may be wrong, but I think solution may be reduced to this
r_dict = dict(zip(['X1','X2','X3','X4'], ['LA','LB','LC','LD']))
df1['Map'] = (df1['Z L'].map(r_dict) == df1['Z P']).map({True: 'R', False: 'L'})
Out[292]:
Period labels Z L Z P Map
2 1 A X3 LC R
3 1 B X2 LC L
4 1 A X4 LD R
6 2 A X2 LC L
7 2 B X3 LC R
8 2 A X3 LB L
9 2 B X1 LA R
For the bottom desired output, you just drop_duplicates as QuangHoang.
I have a pandas dataframe (NROWS x 1) where each row is a list , such as
y
0 [[aa, bb], 0000001]
1 [[uz, mk], 0000011]
I want to flatten the list and split into (in this case three) columns like so:
1 2 3
0 aa bb 0000001
1 uz mk 0000011
Further, different rows have unequal lengths:
y
0 [[aa, bb], 0000001]
1 [[mk], 0000011]
What I really want to end up with is, detect the max length over all rows and pad the rest to empty string ''. In this example,
1 2 3
0 aa bb 0000001
1 '' mk 0000011
I've toyed around with doing .values.tolist() but it doesn't do what I need.
Edit- the answers below are super neat and much appreciated. I'm editing to include a solution for a similar but simpler problem, for completeness.
Read data, use the trim() fn from Strip / trim all strings of a dataframe to make sure there is no left/right whitespace
df = pd.read_csv('data.csv',sep=',',dtype=str)
df = trim_all_columns(df)
Keep categorical/nominal ID and CODE columns, remove all NA
df.dropna(subset=['dg_cd'] , inplace=True) # drop dg_cd is NaN rows from df
df2 = df[['id','dg_cd']]
Turn CODE into sentences by ID keeping all repeated instances
x = df2.groupby('id').apply(lambda x: x['dg_cd'].values.tolist()).apply(pd.Series).replace(np.nan, '', regex=True)
The reason for doing all that is because that feeds into a k-modes cluster search, https://pypi.org/project/kmodes/. NA is not an acceptable input but empty strings
''
allow rows of same length while there is no spurious similarity. For example,
km = KModes(n_clusters=4, init='Cao', n_init=1, verbose=1)
clusters = km.fit_predict( x )
Setup
df = pd.DataFrame(dict(y=[
[['aa', 'bb'], '0000001'],
[['uz', 'mk'], '0000011'],
[['mk'], '0000111']
]))
df
y
0 [[aa, bb], 0000001]
1 [[uz, mk], 0000011]
2 [[mk], 0000111]
flatten
From #wim
def flatten(x):
try:
it = iter(x)
except TypeError:
yield x
return
if isinstance(x, str):
yield x
return
for elem in it:
yield from flatten(elem)
d = dict(zip(df.index, [dict(enumerate([*flatten(x)][::-1])) for x in df.y]))
d = pd.DataFrame.from_dict(d, 'index').fillna('')
d.iloc[:, ::-1].rename(columns=lambda x: d.shape[1] - x)
1 2 3
0 aa bb 0000001
1 uz mk 0000011
2 mk 0000111
After using the same function flatten the list
pd.DataFrame(list(map(lambda x : list(flatten(x)),df.y.tolist()))).apply(lambda x : pd.Series(sorted(x,key=pd.notna)),1)
Out[85]:
0 1 2
0 aa bb 0000001
1 uz mk 0000011
2 None mk 0000111
In case you want to have control over which side to pad the sublists from:
max_len = df['y'].apply(lambda row: len(row[0])).max()
pd.DataFrame([*df['y'].apply(lambda row: ['']*(max_len - len(row[0])) + row[0] + row[1:])])
Which, using #piRSquared's setup gives
0 1 2
0 aa bb 0000001
1 uz mk 0000011
2 mk 0000111
Or, alternatively
pd.DataFrame([*df['y'].apply(lambda row: row[0] + ['']*(max_len - len(row[0])) + row[1:])])
giving you
0 1 2
0 aa bb 0000001
1 uz mk 0000011
2 mk 0000111
I would like to compare two parts of two different columns from an Excel file that have a different number of elements. The comparison should be made between a part of Column 3 and a part of Column 2. Column 3 part has a length of j elements and Column 2 has a length of k elements(k>j). Column 2 part starts from row "j+1" and column 3 part starts from row 1. If an element from column 3 part is matching an element from column 2 part, then should check if the element from column1, before the j row, which has the same index as matched item from column 3 part is matching with the element from Column 1 part between j+1 and k, which has the same index as matched item from column 2 part. If yes, then should be written the element from Column 4 with the same index as matched element from column 2 part in a new Excel sheet.
Example: Column3[1]==Column2[2](which represents element 'A') => Column1[1]==Column1[j+2](which represents element 'P') => Column4[j+2] should be written in a new sheet.
Column 1 Column 2 Column 3 Column 4
P F A S
B G X T
C H K V
D I M W
P B R B
P A R D
C D H E
D E J k
E M K W
F F L Q
Q F K Q
For reading the Excel sheet cells from original sheet, I have used the df27.ix[:j-1,1].
One part of the code which reads the values of the mention part from column 3 and column 2 might be:
for j in range(1,j):
c3=sheet['B'+str(j)].value
for k in range(j,j+k):
c2=sheet['B'+str(k)].value
Any hint how I can accomplish this?
UPDATED
I have tried a new code which takes in consideration that we have '-', like joaquin mentioned in his example.
Joaquin's example:
C1 C2 C3 C4
0 P - A -
1 B - X -
2 C - K -
3 D - M -
4 P B - B
5 P A - D
6 C D - E
7 D E - k
8 E M - W
9 F F - Q
10 Q F - Q
New code:
from pandas import DataFrame as df
import pandas as pd
import openpyxl
wb=openpyxl.load_workbook('/media/sf_vboxshared/x.xlsx')
sheet=wb.get_sheet_by_name('Sheet1')
C13=[]
C12=[]
C1=[]
C2=[]
C3=[]
for s in range(2, sheet.max_row+1):
C1second=sheet['A'+str(s)].value
C2second=sheet['B'+str(s)].value
C3second=sheet['C'+str(s)].value
C1.append(C1second)
C2.append(C2second)
C3.append(C3second)
C1=[x.encode('UTF8') for x in C1]
for y in C2:
if y is not None:
C2=[x.encode('UTF8') if x is not None else None for x in C2]
for z in C3:
if z is not None:
C3=[x.encode('UTF8') if x is not None else None for x in C3]
for x in C1:
C13.append(x)
for x in C3:
C13.append(x)
for x in C1:
C12.append(x)
for x in C2:
C12.append(x)
tosave = pd.DataFrame()
df[C13]=pd.DataFrame(C13)
df[C12]=pd.DataFrame(C12)
for item in df[C13]:
if '-' in item: continue
new = df[df[C12] == item]
tosave = tosave.append(new)
But I still get the following error: df[C13]=pd.DataFrame(C13) TypeError: 'type' object does not support item assignment. Any idea what is wrong?
Many thanks in advance,
Dan
Given your df is
C1 C2 C3 C4
0 P - A -
1 B - X -
2 C - K -
3 D - M -
4 P B - B
5 P A - D
6 C D - E
7 D E - k
8 E M - W
9 F F - Q
10 Q F - Q
then, I combine C1 and C3 and C1 and C2
df['C13'] = df.apply(lambda x: x['C1'] + x['C3'], axis=1)
df['C12'] = df.apply(lambda x: x['C1'] + x['C2'], axis=1)
and compare which rows have the same pair of characters in columns C13 and C12, and save them in tosave
tosave = p.DataFrame()
for item in df['C13']:
if '-' in item: continue
new = df[df['C12'] == item]
tosave = tosave.append(new)
this gives you a tosave dataframe with the rows matching:
C1 C2 C3 C4 C13 C12
5 P A - D P- PA
That can be directly saved as it is or you can save just column C4
UPDATE: If you have data on each row, then you can not use the '-' detection (or any other kind of detection based on the differences between empty and filled columns). On the other hand, if j,k are not defined (for any j and k), your problem is actually reduced to find, for each row, identical pairs below that row. In consecuence, this:
tosave = p.DataFrame()
for idx, item in enumerate(df['C13']):
new = df[df['C12'] == item]
tosave = tosave.append(new.loc[idx+1:])
solves the problem given your labels and data is like:
C1 C2 C3 C4
0 P F A S
1 B G X T
2 C H K V
3 D I M W
4 P B R B
5 P A R D
6 C D H E
7 D E J k
8 E M K W
9 F F L Q
10 Q F K Q
This code also produces the same output as before:
C1 C2 C3 C4 C13 C12
5 P A R D PR PA
Note this probably needs some refinenment (p.e when a row produces 2 matches, the second row with produce 1 match, and you will need to remove replicates from the final output).