Related
This question already has answers here:
Create pandas dataframe from nested dict with outer keys as df index and inner keys column headers
(2 answers)
Closed 5 months ago.
I have a dictionary with the following keys and values
my_dict={'1':{'name':'one',
'f_10':[1,10,20,30],
'f_20':[1,20,40,60]},
'2':{'name':'two',
'f_10':[2,12,22,32],
'f_20':[2,22,42,62]}}
How do I convert it to a Pandas DataFrame that will look like:
name name f_10 f_20
1 one [1,10,20,30] [1,10,20,60]
2 two [2,12,22,32] [2,22,42,62]
The lists need to be considered in one column based on the key, if I try to concat these get converted to separate rows in the data frame.
Simply use orient=index when importing your data using from_dict:
df = pd.DataFrame.from_dict(my_dict, orient = 'index')
This returns:
name f_10 f_20
1 one [1, 10, 20, 30] [1, 20, 40, 60]
2 two [2, 12, 22, 32] [2, 22, 42, 62]
I would parse that dictionary to DataFrame and have it transposed. For example,
pd.DataFrame(my_dict).T
Result
name f_10 f_20
1 one [1, 10, 20, 30] [1, 20, 40, 60]
2 two [2, 12, 22, 32] [2, 22, 42, 62]
I aim to create a plot similar to the image where the '2020 Q4' data is in the same column as '2020'.
So far I was only able to place the 2020 Q4 data simply as an extra column.
The data is provided as a DataFrame like in the code below:
# DataFrame using arrays.
import pandas as pd
# initialize data of lists.
data = {'A':[10, 15, 20, 26, 27, 35, 15],
'B':[20, 25, 32, 33, 50, 52, 8],
'C':[30, 35, 41, 49, 52, 53, 25]}
# Creates pandas DataFrame.
df = pd.DataFrame(data, index =['2015',
'2016',
'2017',
'2018',
'2019',
'2020',
'2020 Q4',])
# plotting the data
df.plot(kind='bar',stacked=True)
Two problems have to be addressed here. First, one has to transpose the Q4 data into the row above. Second, the corresponding columns A A-Q4 etc. need similar colors to make it clear that they belong to the same category. Matplotlib's tab20 colormap comes in handy. Here is one approach:
from matplotlib import pyplot as plt
# DataFrame using arrays.
import pandas as pd
import numpy as np
# initialize data of lists.
data = {'A':[10, 15, 20, 26, 27, 35, 15],
'B':[20, 25, 32, 33, 50, 52, 8],
'C':[30, 35, 41, 49, 52, 53, 25]}
# Creates pandas DataFrame.
df = pd.DataFrame(data, index =['2015',
'2016',
'2017',
'2018',
'2019',
'2020',
'2020 Q4',])
#get column names
columns = df.columns
#store data of last row and create a new dataframe without the last row
val_q4 = df.iloc[-1].values
df1 = df.iloc[:-1]
#alternatively, one can simply overwrite df it doesn't matter to remove the Q4 row
#df = df.iloc[:-1]
#generate additional columns for Q4 data
new_columns = [f(item) for item in columns for f in (lambda x: x, lambda x: x+" Q4")]
df1 = df1.reindex(columns=new_columns)
#store Q4 data in last row
df1.iloc[-1, range(1, 2*len(columns), 2)] = val_q4
#create corresponding color pairs using the tab20 colormap
colors = plt.cm.tab20(np.linspace(0, 1, 20))
df1.plot(kind='bar',stacked=True, color=colors)
plt.show()
Sample output:
Restrictions: It relies on your data structure that the rows are already sorted and the last two rows are "Year" and "Year Q4". Tab20 limits the use to 10 columns A, B, C,...,J because beginning with K, the colors will not be unique. However, stacked bar graphs with more than 10 categories should be outlawed anyhow.
You can introduce Q4 values for A, B and C rows and initialise it to zero for all years except 2020. This should give you the desired result.
For example, see this updated code:
# DataFrame using arrays.
import pandas as pd
# initialize data of lists.
data = {'A':[10, 15, 20, 26, 27, 35, 15],
'B':[20, 25, 32, 33, 50, 52, 8],
'C':[30, 35, 41, 49, 52, 53, 25]}
# additional data initialise with zero for all years
add_data = {'AQ4':[0]*5,
'BQ4':[0]*5,
'CQ4':[0]*5}
# take the last element in list A, B, C and append it to add_data dict
add_data['AQ4'].append(data['A'].pop())
add_data['BQ4'].append(data['B'].pop())
add_data['CQ4'].append(data['C'].pop())
# concatenate the 2 dicts
data.update(add_data)
# Creates pandas DataFrame.
df = pd.DataFrame(data, index =['2015',
'2016',
'2017',
'2018',
'2019',
'2020'])
# plotting the data
df.plot(kind='bar',stacked=True)
Note that I have removed 2020 Q4 index when creating the data frame.
I am pretty new to Python and hence I need your help on the following:
I have two tables (dataframes):
Table 1 has all the data and it looks like that:
GenDate column has the generation day.
Date column has dates.
Column D and onwards has different values
I also have the following table:
Column I has "keywords" that can be found in the header of Table 1
Column K has dates that should be in column C of table 1
My goal is to produce a table like the following:
I have omitted a few columns for Illustration purposes.
Every column on table 1 should be split base on the Type that is written on the Header.
Ex. A_Weeks: The Weeks corresponds to 3 Splits, Week1, Week2 and Week3
Each one of these slits has a specific Date.
in the new table, 3 columns should be created, using A_ and then the split name:
A_Week1, A_Week2 and A_Week3.
for each one of these columns, the value that corresponds to the Date of each split should be used.
I hope the explanation is good.
Thanks
You can get the desired table with the following code (follow comments and check panda api reference to learn about functions used):
import numpy as np
import pandas as pd
# initial data
t_1 = pd.DataFrame(
{'GenDate': [1, 1, 1, 2, 2, 2],
'Date': [10, 20, 30, 10, 20, 30],
'A_Days': [11, 12, 13, 14, 15, 16],
'B_Days': [21, 22, 23, 24, 25, 26],
'A_Weeks': [110, 120, 130, 140, np.NaN, 160],
'B_Weeks': [210, 220, 230, 240, np.NaN, 260]})
# initial data
t_2 = pd.DataFrame(
{'Type': ['Days', 'Days', 'Days', 'Weeks', 'Weeks'],
'Split': ['Day1', 'Day2', 'Day3', 'Week1', 'Week2'],
'Date': [10, 20, 30, 10, 30]})
# create multiindex
t_1 = t_1.set_index(['GenDate', 'Date'])
# pivot 'Date' level of MultiIndex - unstack it from index to columns
# and drop columns with all NaN values
tt_1 = t_1.unstack().dropna(axis=1)
# tt_1 is what you need with multi-level column labels
# map to rename columns
t_2 = t_2.set_index(['Type'])
mapping = {
type_: dict(zip(
t_2.loc[type_, :].loc[:, 'Date'],
t_2.loc[type_, :].loc[:, 'Split']))
for type_ in t_2.index.unique()}
# new column names
new_columns = list()
for letter_type, date in tt_1.columns.values:
letter, type_ = letter_type.split('_')
new_columns.append('{}_{}'.format(letter, mapping[type_][date]))
tt_1.columns = new_columns
I have a dataframe from an import csv using pandas. This dataframe has 160 variables and I would like to keep only 5, 9, 10, 46, 89.
I try this:
dataf2 = dataf[[5] + [9] + [10] + [46] + [89]]
but I take this error:
KeyError: '[ 5 9 10 46 89] not in index'
If you want to refer to columns not by their names but by their positions in the dataset, you need to use df.iloc:
dataf.iloc[:, [5, 9, 10, 46, 89]]
Row indices are specified before the comma, column indices are specified after the comma.
If the columns that you would like to keep are: 5, 9, 10, 46, 89, then you can index just these ones like so:
dataf2 = dataf[[5, 9, 10, 46, 89]]
My data (in a pandas DataFrame) looks like this:
a = pd.DataFrame({"age" : [25, 25, 25, 26, 26, 25, 25, 27, 27, 25, 26, 26, 25, 25],
"category" : [1, 2, 2, 3, 1, 3, 1, 1, 2, 3, 2, 2, 1, 2]})
I want a stacked barplot - each bar should reflect how many are in an age group and what percentage within the agegroup belongs to which category. The code
a.pivot_table('category', 'age').plot(kind='bar')
returns only the mean of the category instead of stacking it the way I want it.
Perhaps you mean to do something like the following... You need to change the aggregating function in pivot_table (which defaults to 'mean') and then use stacked=True when you plot:
table = a.pivot_table(index='category', columns='age', aggfunc='size')
table.plot(kind='bar', stacked=True)
This yields: