df_1 = {'budget_id':['1', '2', '3', '4'],
'budget_amount':[200, 300, 400, 500]}
df_2 = {'budget_id':['1', '2', '3', '4', '5'],
'budget_amount':[200, 300, 400, 550, 700]}
df_1.compare(df_2, align_axis=0, keep_equal=True).rename(index={'self': 'Prev', 'other': 'New'}, level=1)
Desired output of df.compare():
budget_id budget_amount
4 550
5 700
I have two data frames that I wish to compare using df.compare. They both have the same columns and index labels.
However, I can not guarantee they have the same number of rows. This causes issues as compare expects a two DFs with the same shape.
I need to know if a new row has been added as part of the compare.
Is the best solution would be to append blank rows to either data frame until they're equal? How would you do that?
Is there a more elegant way?
Does mergework for you:
(df_1.merge(df_2, on='budget_id', how='right')
.query('budget_amount_x != budget_amount_y')
)
Output:
budget_id budget_amount_x budget_amount_y
3 4 500.0 550
4 5 NaN 700
This is the solution I wrote based on Giovanni Frison's comment.
def compare_dataframes(df_1, df_2):
if df_1.equals(df_2):
return pandas.DataFrame()
else:
#Get indexes of rows present in df_2, but not in df_1
new_row_indexes = df_2.index.difference(df_1.index)
new_rows = df_2.loc[new_row_indexes]
#Create second index to match df.compare output
new_rows[''] = 'New'
new_rows = new_rows.set_index('',append=True)
#Drop new rows from df_2 to create same shape for df.compare
df_2 = df_2.drop(new_row_indexes)
compare_df = df_1.compare(df_2, align_axis=0, keep_equal=True).rename(index={'self': 'Prev', 'other': 'New'}, level=1)
compare_df = compare_df.append(new_rows)
return compare_df
Related
The data I have to work with is a bit messy.. It has header names inside of its data. How can I choose a row from an existing pandas dataframe and make it (rename it to) a column header?
I want to do something like:
header = df[df['old_header_name1'] == 'new_header_name1']
df.columns = header
In [21]: df = pd.DataFrame([(1,2,3), ('foo','bar','baz'), (4,5,6)])
In [22]: df
Out[22]:
0 1 2
0 1 2 3
1 foo bar baz
2 4 5 6
Set the column labels to equal the values in the 2nd row (index location 1):
In [23]: df.columns = df.iloc[1]
If the index has unique labels, you can drop the 2nd row using:
In [24]: df.drop(df.index[1])
Out[24]:
1 foo bar baz
0 1 2 3
2 4 5 6
If the index is not unique, you could use:
In [133]: df.iloc[pd.RangeIndex(len(df)).drop(1)]
Out[133]:
1 foo bar baz
0 1 2 3
2 4 5 6
Using df.drop(df.index[1]) removes all rows with the same label as the second row. Because non-unique indexes can lead to stumbling blocks (or potential bugs) like this, it's often better to take care that the index is unique (even though Pandas does not require it).
This works (pandas v'0.19.2'):
df.rename(columns=df.iloc[0])
It would be easier to recreate the data frame.
This would also interpret the columns types from scratch.
headers = df.iloc[0]
new_df = pd.DataFrame(df.values[1:], columns=headers)
To rename the header without reassign df:
df.rename(columns=df.iloc[0], inplace = True)
To drop the row without reassign df:
df.drop(df.index[0], inplace = True)
You can specify the row index in the read_csv or read_html constructors via the header parameter which represents Row number(s) to use as the column names, and the start of the data. This has the advantage of automatically dropping all the preceding rows which supposedly are junk.
import pandas as pd
from io import StringIO
In[1]
csv = '''junk1, junk2, junk3, junk4, junk5
junk1, junk2, junk3, junk4, junk5
pears, apples, lemons, plums, other
40, 50, 61, 72, 85
'''
df = pd.read_csv(StringIO(csv), header=2)
print(df)
Out[1]
pears apples lemons plums other
0 40 50 61 72 85
Keeping it Python simple
Padas DataFrames have columns attribute why not use it with standard Python, it is much clearer what you are doing:
table = [['name', 'Rf', 'Rg', 'Rf,skin', 'CRI'],
['testsala.cxf', '86', '95', '92', '87'],
['testsala.cxf: 727037 lm', '86', '95', '92', '87'],
['630.cxf', '18', '8', '11', '18'],
['Huawei stk-lx1.cxf', '86', '96', '88', '83'],
['dedo uv no filtro.cxf', '52', '93', '48', '58']]
import pandas as pd
data = pd.DataFrame(table[1:],columns=table[0])
or in the case is not the first row, but the 10th for instance:
columns = table.pop(10)
data = pd.DataFrame(table,columns=columns)
I have a tricky problem to select column in a dataframe. I have a dataframe and multiple columns in it have the same name "PTime".
This is my dataframe:
PTime first_column PTime third_column PTime fourth_column
0 4 first_value 1 first_value 6 first_value
1 4 second_value 2 second_value 7 second_value
This is what I want:
PTime first_column PTime fourth_column
0 4 first_value 6 first_value
1 4 second_value 7 second_value
I will select my columns from a list:
My code:
data = {'PTime': ['1', '1'],
'first_column': ['first_value', 'second_value'],
'PTime': ['2', '2'],
'third_column': ['first_value', 'second_value'],
'PTime': ['4', '4'],
'fourth_column': ['first_value', 'second_value'],
}
list_c = ['PTime','first_column','fourth_column']
df = pd.DataFrame(data)
#df = df[df.columns.intersection(list_c)]
df = df[list_c]
df
So my goal is to select the column that is in the list and select the column to the left of the one in the list. I if you have any idea to do that, thank you really much. Regards
I don't exactly know how to get left of one in list
But i have a trick to get desired table which you want as shown
PTime first_column PTime fourth_column
0 4 first_value 6 first_value
1 4 second_value 7 second_value
what we can do is simply remove the column by index
But here as there are same name pandas will to try to delete the first row
But you can simply rename the columns if there are duplicates name and then you can use indexing to delete columns..
So here find some logic to rename it like PTime1 .. PTime2 .. PTime3 ..
and then use indexes to remove it
df.drop(df.columns[i], axis=1,inplace=True)
// or //
df = df.drop(df.columns[i], axis=1)
Here you have to pass the list of indices . In your case it will be like
df.drop(df.columns[[2,3]],axis=1)
After renaming columns
In my dataframe I will not have multiple columns with the same name. All names will be distinct.
So in the case I have ten columns to select it will be difficult to list them all in a list.
data = {'PTime1': ['1', '1'],
'first_column': ['first_value', 'second_value'],
'PTime2': ['2', '2'],
'third_column': ['first_value', 'second_value'],
'PTime3': ['4', '4'],
'fourth_column': ['first_value', 'second_value'],
}
list_c = ['first_column','fourth_column'] #define column to select
df = pd.DataFrame(data) #create dataframe
list_index = [] #create list to store index column
for col in list_c:
index_no = df.columns.get_loc(col) #get index column
list_index.append(index_no-1) #insert index-1 in a list. Get column from the left
list_index.append(index_no) #insert index from the column in the list.
df = df.iloc[:, list_index] #Subset the dataframe with the list of column selected.
df
Like this I can select the column from my list and the column on the left of each element in my list.
I would like to merge three dataframes. I have tried to simplify the problem to explain: I have df with columns ['1', '2', '3'] and df1 with columns ['1', '2', '3'] and df2 with columns ['1', '2', '3'].
I want to merge the dataframes on keys 1 & 2.
I have tried the following (simplified):
new = pd.merge(df, df1, how = 'left', on = [ '1', '2'])
new1 = pd.merge(new, df2, how = 'left', on = ['1', '2'])
Now you get as outcome
new with columns ['1', '2', '3_x', '3_y']
new1 with columns ['1', '2', '3_x', '3_y', '3_z']
while i would like
new with columns ['1', '2', '3']
new1 with columns ['1', '2', '3']
Any help is welcome! I don't want to use a loop.
Thanks in advance.
Merge will 'merge' only the columns (and only on the columns) that you are specifying in the 'on'. That is, this is working as expected.
If you have matching columns 1 and 2 but different columns 3 and you make the merge, what should be in each column in our new output? 1 and 2 will be whatever they were in both of the originals, but there are two different options for column 3. The function of merge is to split those by different columns.
What you may want here instead is append. Append will put one dataframe below another, as explained on this page here: https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html#concatenating-using-append.
This will give you an output that has the columns '1', '2', '3'.
new = df.append(df1)
new1 = new.append(df2)
You may even be able to do new = df.append([df1,df2]), but I didn't try it and don't have 3 dataframes handy.
You did say you want to just merge on columns 1 and 2, so I may be completely missing your point here. What would you want with the data in column 3 in such a case? You may be able to achieve that by using append and then removing some duplicates or otherwise cleaning up this output.
The data I have to work with is a bit messy.. It has header names inside of its data. How can I choose a row from an existing pandas dataframe and make it (rename it to) a column header?
I want to do something like:
header = df[df['old_header_name1'] == 'new_header_name1']
df.columns = header
In [21]: df = pd.DataFrame([(1,2,3), ('foo','bar','baz'), (4,5,6)])
In [22]: df
Out[22]:
0 1 2
0 1 2 3
1 foo bar baz
2 4 5 6
Set the column labels to equal the values in the 2nd row (index location 1):
In [23]: df.columns = df.iloc[1]
If the index has unique labels, you can drop the 2nd row using:
In [24]: df.drop(df.index[1])
Out[24]:
1 foo bar baz
0 1 2 3
2 4 5 6
If the index is not unique, you could use:
In [133]: df.iloc[pd.RangeIndex(len(df)).drop(1)]
Out[133]:
1 foo bar baz
0 1 2 3
2 4 5 6
Using df.drop(df.index[1]) removes all rows with the same label as the second row. Because non-unique indexes can lead to stumbling blocks (or potential bugs) like this, it's often better to take care that the index is unique (even though Pandas does not require it).
This works (pandas v'0.19.2'):
df.rename(columns=df.iloc[0])
It would be easier to recreate the data frame.
This would also interpret the columns types from scratch.
headers = df.iloc[0]
new_df = pd.DataFrame(df.values[1:], columns=headers)
To rename the header without reassign df:
df.rename(columns=df.iloc[0], inplace = True)
To drop the row without reassign df:
df.drop(df.index[0], inplace = True)
You can specify the row index in the read_csv or read_html constructors via the header parameter which represents Row number(s) to use as the column names, and the start of the data. This has the advantage of automatically dropping all the preceding rows which supposedly are junk.
import pandas as pd
from io import StringIO
In[1]
csv = '''junk1, junk2, junk3, junk4, junk5
junk1, junk2, junk3, junk4, junk5
pears, apples, lemons, plums, other
40, 50, 61, 72, 85
'''
df = pd.read_csv(StringIO(csv), header=2)
print(df)
Out[1]
pears apples lemons plums other
0 40 50 61 72 85
Keeping it Python simple
Padas DataFrames have columns attribute why not use it with standard Python, it is much clearer what you are doing:
table = [['name', 'Rf', 'Rg', 'Rf,skin', 'CRI'],
['testsala.cxf', '86', '95', '92', '87'],
['testsala.cxf: 727037 lm', '86', '95', '92', '87'],
['630.cxf', '18', '8', '11', '18'],
['Huawei stk-lx1.cxf', '86', '96', '88', '83'],
['dedo uv no filtro.cxf', '52', '93', '48', '58']]
import pandas as pd
data = pd.DataFrame(table[1:],columns=table[0])
or in the case is not the first row, but the 10th for instance:
columns = table.pop(10)
data = pd.DataFrame(table,columns=columns)
I have the pandas.DataFrame below:
One of the columns from the Dataframe, pontos, holds a dict for each of the rows.
What I want to do is add one column to the DataFrame for each key from this dict. So the new columns would be, in this example: rodada, mes, etc, and for each row, these columns would be populated with the respective value from the dict.
So far I've tried the following for one of the keys:
df_times["rodada"] = [df_times["pontos"].get('rodada') for d in df_times["pontos"]]
However, as a result I'm getting a new column rodada filled with None values:
Any hints on what I'm doing wrong?
You can create a new dataframe and concat it to the current one like:
Code:
df2 = pd.concat([df, pd.DataFrame(list(df.pontos))], axis=1)
Test Code:
import pandas as pd
df = pd.DataFrame([
['A', dict(col1='1', col2='2')],
['B', dict(col1='3', col2='4')],
], columns=['X', 'D'])
print(df)
df2 = pd.concat([df, pd.DataFrame(list(df.D))], axis=1)
print(df2)
Results:
X D
0 A {'col2': '2', 'col1': '1'}
1 B {'col2': '4', 'col1': '3'}
X D col1 col2
0 A {'col2': '2', 'col1': '1'} 1 2
1 B {'col2': '4', 'col1': '3'} 3 4
You just need a slight change in your comprehension to extract that data.
It should be:
df_times["rodada"] = [d.get('rodada') for d in
df_times["pontos"]]
You want the values of the dictionary key 'rodada' to be the basis of your new column. So you iterate over those dictionary entries in the loop- in other words, d, and then extract the value by key to make the new column.
you can also use join and pandas apply function:
df=df.join(df['pontos'].apply(pd.Series))