Say I have a data frame like this:
from pandas import DataFrame
example = {'year_month': [201801,201802,201803,201801,201802,201803],
'store_id': [101,101,101,102,102,102],
'tot_employees': [100,200,150,6,7,10],
'hrs_per_employee': [30,35,20,20,18,15]
}
df = DataFrame(example,columns=["year_month", "store_id", "tot_employees", "hrs_per_employee"])
df
and i want to have stacked subplots with a different subplot for each store_id with:
x axis: year_month
plots line 1: tot employees
plot line 2: hrs per
employee
is this possible with df.plot()? I haven't been able to find the correct x,y inputs to get the result i'm looking for. if not is there a close alternative? thanks in advance
import pandas as pd
df = pd.DataFrame(example)
df.year_month = pd.to_datetime(df.year_month, format='%Y%m', exact=True)
df.set_index('year_month', drop=True, inplace=True)
for x in df.store_id.unique():
df[['tot_employees', 'hrs_per_employee']][df.store_id == x].plot(title=f'Store ID: {x}')
Using only df.groupby
df.groupby('store_id').plot(y=['tot_employees', 'hrs_per_employee'])
Related
I want to create a plot chart with forecasted figures for next 2 months. The below is the code I wrote.
import pandas as pd
from datetime import datetime
df= pd.read_csv(r'C:\Users\Desktop\Customers.csv')
parsed = pd.to_datetime(df["Date"], errors="coerce").fillna(pd.to_datetime(df["Date"],format="%Y-%d-%m",errors="coerce"))
ordinal = pd.to_numeric(df["Date"], errors="coerce").apply(lambda x: pd.Timestamp("1899-12-30")+pd.Timedelta(x, unit="D"))
df["Date"] = parsed.fillna(ordinal)
df['Amount currency'] = df['Amount currency'].str.replace(r'[^0-9\.]', '', regex=True)
df['Amount'] = df['Amount'].str.replace(r'[^0-9\.]', '', regex=True)
df['Amount currency'] = pd.to_numeric(df['Amount currency'])
df['Amount'] = pd.to_numeric(df['Amount'])
#df.Date = pd.to_datetime(df.Date).dt.to_period('m')
df['Date'] = df['Date'].dt.to_period('M').dt.to_timestamp() + pd.offsets.MonthEnd()
columns = ['Date', 'Type', 'Amount']
df = df[columns]
and it is required to pivot the figures
df2=pd.pivot_table(df,index='Date',values = 'Amount', columns = 'Type',aggfunc='sum')
So the final output columns are,
Date
Customer Credit Note
Payment
Sales Invoice
Based on the above code, I wanted to create a plot with 2 months of forecast
import matplotlib.pyplot as plt
import seaborn as sns
sns.lineplot(x='Date',y='Payment',data=dataset)
plt.title("Monthly_cases")
plt.xlabel("Month end date")
plt.ylabel("Payment")
plt.show()
But the above code returns with an error named KeyError:Date
What would be the reason for this? Can anyone help me? Also can anyone help me to modify the above code to get next two months forecasted values?
Thanks
I have weekly based trade export time-series data that I need to make a stacked bar plot for visualizing trade activity. To do so, I aggregated my data for sum-up of each columns for all rows, then use nlargest() to select top n columns. However, doing this way might not be quite accurate because I made stacked plot for different years in the loop and top n columns for each year can be different. But what I did, take the total sum of each column for all rows (a.k.a, including all years) then select top n columns, which is biased. So, I am looking at the different way of doing this, perhaps, I might group the time series data by each year then make the stacked plot. Is there other way around selecting top n columns from time-series data instead of using nlargest? Does anyone know any possible way of doing this? What other way we could select top n columns from time-series data? Any idea?
my current attempt:
this is my current attempt to manipulate time series data, where I aggregate each columns for all rows then select top n columns using nlargest():
import pandas as pd
# load the data
url = 'https://gist.githubusercontent.com/adamFlyn/a6048e547b5a963c7af356c964d15af6/raw/c57c7915cf14f81edc9d5eadaf14efbd43d3e58a/trade_df.csv'
df_ = pd.read_csv(url, parse_dates=['weekly'])
df_.set_index('weekly', inplace=True)
df_.loc['Total',:]= df_.sum(axis=0)
df1 = df_.T
df1 =df1.nlargest(6, columns=['Total'])
df1.drop('Total', axis=1, inplace=True)
df2 = df1.T
df2.reset_index(inplace=True)
df2['weekly'] = pd.to_datetime(df2['weekly'])
df2['year'] = df2['weekly'].dt.year
df2['week'] = df2['weekly'].dt.strftime('%W').astype('int')
then I visualize the plotting data with matplotlib as follow:
import matplotlib.pyplot as plt
plt_df = df2.set_index(['year','week'])
plt_df.drop("weekly", axis=1, inplace=True)
for n, g in plt_df.groupby(level=0):
ax = g.loc[n].plot.bar(stacked=True, title=f'{n} Year', figsize=(8,5))
plt.show()
although the output of current approach in stacked plot is fine, but selecting top n columns using nlargest() is not quite accurate.for example, in 2019 USDA report, China wasn't top trade partner of US, but in late 2020, China was getting more products from US, and if I use nlargest() to select top column (or trade partners), it is going to be problematic and China won't be in list and not in the plot.
update
As #Vaishali suggested in the comment with this post, using head() might be good idea to extract top columns, so I tried like this:
for n, g in plt_df.groupby(level=0):
for i in g:
gg = g[i].sort_values(g[i].values,ascending = False).groupby('week').head(5)
ax = gg.loc[n].plot.bar(stacked=True, title=f'{n} Year', figsize=(8,5))
but this is not working. Can anyone point me out how to select top n columns from time series data? any idea?
You can try something like this:
url = 'https://gist.githubusercontent.com/adamFlyn/a6048e547b5a963c7af356c964d15af6/raw/c57c7915cf14f81edc9d5eadaf14efbd43d3e58a/trade_df.csv'
df_ = pd.read_csv(url, parse_dates=['weekly'])
df_['weekly'] = pd.to_datetime(df_['weekly'])
df_.set_index('weekly', inplace=True)
for g, n in df_.groupby(df_.index.year):
ng = n.loc[:, n.sum().rank(ascending=False, method='min')<5]
ng.div(ng.sum(axis=1), axis=0).plot.area(title=f'{g}')
Output:
Bar chart:
import matplotlib.ticker as mticker
url = 'https://gist.githubusercontent.com/adamFlyn/a6048e547b5a963c7af356c964d15af6/raw/c57c7915cf14f81edc9d5eadaf14efbd43d3e58a/trade_df.csv'
df_ = pd.read_csv(url, parse_dates=['weekly'])
df_['weekly'] = pd.to_datetime(df_['weekly'])
df_.set_index('weekly', inplace=True)
for g, n in df_.groupby(df_.index.year):
ng = n.loc[:, n.sum().rank(ascending=False, method='min')<5]
ng.index = ng.index.strftime('%m/%d/%Y')
ax = ng.plot.bar(stacked=True, figsize=(10,8))
Output:
Staked 100% Bar chart:
#(previous code)
ax = ng.div(ng.sum(axis=1), axis=0).plot.bar(stacked=True, figsize=(10,8))
Output:
I am not sure I understand the requirement correctly here, but this is based on your output charts:
find top n countries using sum and nlargest
filter df by top_countries, groupby year and week, sum
for each unique year, plot stacked chart
df.columns = df.columns.str.strip()
top_countries = df.iloc[:, 1:].sum().nlargest(6).index.tolist()
df['weekly'] = pd.to_datetime(df['weekly'])
agg = df[top_countries].groupby([df['weekly'].dt.year.rename('year'),df['weekly'].dt.week.rename('week')]).sum()
for year in df['weekly'].dt.year.unique():
agg[agg.index.get_level_values(0) == year].droplevel(level=0).plot.bar(stacked = True, figsize = (10,5), title = year)
Edit:
If you want to filter top countries by year, move the part where you are filtering df into the loop,
df.columns = df.columns.str.strip()
df['weekly'] = pd.to_datetime(df['weekly'])
for year in df['weekly'].dt.year.unique():
top_countries = df.iloc[:, 1:].sum().nlargest(6).index.tolist()
agg = df[top_countries].groupby([df['weekly'].dt.year.rename('year'),df['weekly'].dt.week.rename('week')]).sum()
agg[agg.index.get_level_values(0) == year].droplevel(level=0).plot.bar(stacked = True, figsize = (10,5), title = year)
You can try this
import pandas as pd
# load the data
url = 'https://gist.githubusercontent.com/adamFlyn/a6048e547b5a963c7af356c964d15af6/raw/c57c7915cf14f81edc9d5eadaf14efbd43d3e58a/trade_df.csv'
df = pd.read_csv(url, parse_dates=['weekly'])
df['weekly'] = pd.to_datetime(df['weekly'])
df['year'] = df['weekly'].dt.year
df['week'] = df['weekly'].dt.strftime('%W').astype('int')
df.set_index(['year', 'week'], inplace=True)
df.drop('weekly', axis=1, inplace=True)
df_year_sums = df.groupby(level='year').sum().T
for year in df_year_sums.columns:
largest = list(df_year_sums[year].nlargest(6).index)
df_plot = df.xs(year, level='year')[largest]
df_plot.plot.bar(stacked=True, title=f'{year} Year', figsize=(8,5))
df=pd.read_csv('trade_df.csv',parse_dates=['weekly'])
df['weekly']=pd.to_datetime(df['weekly'])
df['Total']=0
df.reset_index()
for key,row in df.iterrows():
sum=0.0
for row_value in row:
if type(row_value)==float:
sum+=row_value
df.loc[key,'Total']=sum
results=df.sort_values(by="Total",ascending=False)
print(results.head(5))
#grouped=df.groupby('year').sum().T.plot.bar(stacked=True)
#plt.show()
filter=df['year'].isin([2018])
results_2018=df[filter].sort_values(by=['total'],ascending=False).head(5)
filter=df['year'].isin([2019])
results_2019=df[filter].sort_values(by=['total'],ascending=False).head(5)
filter=df['year'].isin([2020])
results_2020=df[filter].sort_values(by=['total'],ascending=False).head(5)
grouped=df.groupby('year').sum().T.plot.bar(stacked=True)
plt.show()
fp=results_2018.pivot_table(index=['week'],aggfunc='sum').fillna(0)
fp = fp[(fp.T != 0).any()]
fp2=results_2019.pivot_table(index=['week'],aggfunc='sum').fillna(0)
fp2 = fp2[(fp2.T != 0).any()]
fp3=results_2020.pivot_table(index=['week'],aggfunc='sum').fillna(0)
fp3 = fp3[(fp3.T != 0).any()]
#print(fp)
fig,ax=plt.subplots(3,1,figsize=(16,16))
fp.plot.bar(stacked=True,ax=ax[0])
fp2.plot.bar(stacked=True,ax=ax[1])
fp3.plot.bar(stacked=True,ax=ax[2])
plt.show()
df = pd.DataFrame(np.random.randint(1,100,(100)),columns=["column1"])
results=np.array(df.sort_values(by="column1",ascending=False)).flatten()
print(results[:5])
This is my code:
import pandas as pd
cols= ['DD','MM','YYYY','HH'] #names
DD,MM,YYYY,HH=[1,2,None,4,5,5],[1,1,1,2,2,3],[2014,2014,2014,2014,2014,2014],[20,20,20,18,18,18] #data
df = pd.DataFrame(list(zip(DD,MM,YYYY,HH)), columns =cols )
print (df)
a=pd.crosstab(df .HH, df .MM,margins=True)
print (a)
I would like to view results in a table format. Table borders or at least the same number of digits would solve the problem.
I want to see the table on console without any graphical window.
If you want a nicely looking crosstab you can use seaborn.heatmap
An example
>>> import numpy as np; np.random.seed(0)
>>> import seaborn as sns; sns.set()
>>> uniform_data = np.random.rand(10, 12)
>>> ax = sns.heatmap(uniform_data)
result would look like this:
You can find many examples that show how to apply this, e.g.:
https://www.science-emergence.com/Codes/How-to-plot-a-confusion-matrix-with-matplotlib-and-seaborn/
https://gist.github.com/shaypal5/94c53d765083101efc0240d776a23823
Update
In order to simply display the crosstab in a formatted way you can skip the print and display like this
import pandas as pd
cols= ['DD','MM','YYYY','HH'] #names
DD,MM,YYYY,HH=[1,2,None,4,5,5],[1,1,1,2,2,3],[2014,2014,2014,2014,2014,2014],[20,20,20,18,18,18] #data
df = pd.DataFrame(list(zip(DD,MM,YYYY,HH)), columns =cols )
print (df)
a = pd.crosstab(df .HH, df .MM,margins=True)
display(a)
which will yield the same result as:
pd.crosstab(df .HH, df .MM,margins=True)
I have a dataset like the one shown below.
Date;Time;Global_active_power;Global_reactive_power;Voltage;Global_intensity;Sub_metering_1;Sub_metering_2;Sub_metering_3
16/12/2006;17:24:00;4.216;0.418;234.840;18.400;0.000;1.000;17.000
16/12/2006;17:25:00;5.360;0.436;233.630;23.000;0.000;1.000;16.000
16/12/2006;17:26:00;5.374;0.498;233.290;23.000;0.000;2.000;17.000
16/12/2006;17:27:00;5.388;0.502;233.740;23.000;0.000;1.000;17.000
16/12/2006;17:28:00;3.666;0.528;235.680;15.800;0.000;1.000;17.000
16/12/2006;17:29:00;3.520;0.522;235.020;15.000;0.000;2.000;17.000
16/12/2006;17:30:00;3.702;0.520;235.090;15.800;0.000;1.000;17.000
16/12/2006;17:31:00;3.700;0.520;235.220;15.800;0.000;1.000;17.000
16/12/2006;17:32:00;3.668;0.510;233.990;15.800;0.000;1.000;17.000
I've used pandas to get the data into a DataFrame. The dataset has data for multiple days with an interval of 1 min for each row in the dataset.
I want to plot separate graphs for the voltage with respect to the time(shown in column 2) for each day(shown in column 1) using python. How can I do that?
txt = '''Date;Time;Global_active_power;Global_reactive_power;Voltage;Global_intensity;Sub_metering_1;Sub_metering_2;Sub_metering_3
16/12/2006;17:24:00;4.216;0.418;234.840;18.400;0.000;1.000;17.000
16/12/2006;17:25:00;5.360;0.436;233.630;23.000;0.000;1.000;16.000
16/12/2006;17:26:00;5.374;0.498;233.290;23.000;0.000;2.000;17.000
16/12/2006;17:27:00;5.388;0.502;233.740;23.000;0.000;1.000;17.000
16/12/2006;17:28:00;3.666;0.528;235.680;15.800;0.000;1.000;17.000
16/12/2006;17:29:00;3.520;0.522;235.020;15.000;0.000;2.000;17.000
16/12/2006;17:30:00;3.702;0.520;235.090;15.800;0.000;1.000;17.000
16/12/2006;17:31:00;3.700;0.520;235.220;15.800;0.000;1.000;17.000
16/12/2006;17:32:00;3.668;0.510;233.990;15.800;0.000;1.000;17.000'''
from io import StringIO
f = StringIO(txt)
df = pd.read_table(f,sep =';' )
plt.plot(df['Time'],df['Voltage'])
plt.show()
gives output :
I believe this will do the trick (I edited the dates so we have two dates)
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline #If you use Jupyter Notebook
df = pd.read_csv('test.csv', sep=';', usecols=['Date','Time','Voltage'])
unique_dates = df.Date.unique()
for date in unique_dates:
print('Date: ' + date)
df.loc[df.Date == date].plot.line('Time', 'Voltage')
plt.show()
You will get this:
X = df.Date.unique()
for i in X: #iterate over unique days
temp_df = df[df.Date==i] #get df for specific day
temp_df.plot(x = 'Time', y = 'Voltage') #plot
If you want to change x values you can use
x = np.arange(1, len(temp_df.Time), 1)
group by hour and minute after creating a DateTime variable to handle multiple days. you can filter the grouped for a specific day.
txt =
'''Date;Time;Global_active_power;Global_reactive_power;Voltage;Global_intensity;Sub_metering_1;Sub_metering_2;Sub_metering_3
16/12/2006;17:24:00;4.216;0.418;234.840;18.400;0.000;1.000;17.000
16/12/2006;17:25:00;5.360;0.436;233.630;23.000;0.000;1.000;16.000
16/12/2006;17:26:00;5.374;0.498;233.290;23.000;0.000;2.000;17.000
16/12/2006;17:27:00;5.388;0.502;233.740;23.000;0.000;1.000;17.000
16/12/2006;17:28:00;3.666;0.528;235.680;15.800;0.000;1.000;17.000
16/12/2006;17:29:00;3.520;0.522;235.020;15.000;0.000;2.000;17.000
16/12/2006;17:30:00;3.702;0.520;235.090;15.800;0.000;1.000;17.000
16/12/2006;17:31:00;3.700;0.520;235.220;15.800;0.000;1.000;17.000
16/12/2006;17:32:00;3.668;0.510;233.990;15.800;0.000;1.000;17.000'''
from io import StringIO
f = StringIO(txt)
df = pd.read_table(f,sep =';' )
df['DateTime']=pd.to_datetime(df['Date']+"T"+df['Time']+"Z")
df.set_index('DateTime',inplace=True)
filter=df['Date']=='16/12/2006'
grouped=df[filter].groupby([df.index.hour,df.index.minute])['Voltage'].mean()
grouped.plot()
plt.show()
I am trying to combine a series of stock tick data based on the dates.
But it wont work. Please help.
import pandas as pd
import tushare as ts
def get_all_tick(stockID):
dates=pd.date_range('2016-01-01',periods=5,freq='D')
append_data=[]
for i in dates:
stock_tick=pd.DataFrame(ts.get_tick_data(stockID,date=i))
stock_tick.sort('volume',inplace=True, ascending=False)
stock_tick=stock_tick[:10]
stock_tick.sort('time',inplace=True, ascending=False)
append_data.append(stock_tick.iterrows())
get_all_tick('300243')
I figure it out myself.
def get_all_tick(stockID):
.........
df = pd.DataFrame()
for i in get_date:
stock_tick = ts.get_tick_data(stockID, date=i)
stock_tick['Date']=i
stock_tick.sort('volume', inplace=True, ascending=False)
stock_tick = stock_tick[:10]
stock_tick.sort('time', inplace=True, ascending=False)
df = df.append(stock_tick)
df.to_excel('tick.xlsx',sheet_name='Sheet1')
get_all_tick('300243')