Develop Reference

Time series data visualization issue

I have a time series data like below where the data consists of year and week. So, the data is from 2014 1st week to 2015 52 weeks.
Now, below is the line plot of the above mentioned data
As you can see the x axis labelling is not quite what I was trying to achieve since the point after 201453 should be 201501 and there should not be any straight line and it should not be up to 201499. How can I rescale the xaxis exactly according to Due_date column? Below is the code
rand_products = np.random.choice(Op_2['Sp_number'].unique(), 3)
selected_products = Op_2[Op_2['Sp_number'].isin(rand_products)][['Due_date', 'Sp_number', 'Billing']]
plt.figure(figsize=(20,10))
plt.grid(True)
g = sns.lineplot(data=selected_products, x='Due_date', y='Billing', hue='Sp_number', ci=False, legend='full', palette='Set1');

the issue is because 201401... etc. are read as numbers and that is the reason the line chart has that gap. To fix it, you will need to change the numbers to date format and plot it.
As the full data is not available, below is the two column dataframe which has the Due_date in the form of integer YYYYWW. Billing column is a bunch of random numbers. Use the method here to convert the integers to dateformat and plot. The gap will be removed....
import numpy as np
import pandas as pd
import random
import matplotlib.pyplot as plt
import seaborn as sns
Due_date = list(np.arange(201401,201454)) #Year 2014
Due_date.extend(np.arange(201501,201553)) #Year 2915
Billing = random.sample(range(500, 1000), 105) #billing numbers
df = pd.DataFrame({'Due_date': Due_date, 'Billing': Billing})
df.Due_date = df.Due_date.astype(str)
df.Due_date = pd.to_datetime(df['Due_date']+ '-1',format="%Y%W-%w") #Convert to date
plt.figure(figsize=(20,10))
plt.grid(True)
ax = sns.lineplot(data=df, x='Due_date', y='Billing', ci=False, legend='full', palette='Set1')
Output graph

How to extract appropriate data in Plotly Grouped Bar Chart?

Well, I am trying to plot a Bar Graph in Plotly where I need to show 3 years of data in a grouped bar chart though the chart is displaying the data in the chart not showing data correctly all the bars are equal in the graph Something like this:
Here is my code for plotting:
data=[go.Bar(x=nasdaq['Sector'],y=recent_ipos['IPO Year'],textangle=-45,name='2015'),
go.Bar(x=nasdaq['Sector'],y=recent_ipos['IPO Year'],textangle=-45,name='2016'),
go.Bar(x=nasdaq['Sector'],y=recent_ipos['IPO Year'],textangle=-45,name='2017')
]
layout=go.Layout(title="NASDAQ Market Capitalization IPO yEAR (million USD)",barmode='group')
fig=go.Figure(data=data,layout=layout)
fig.show(renderer="colab")
Here is my code which I am using to extract the data for 3 years:
recent_ipos = nasdaq[nasdaq['IPO Year'] > 2014]
recent_ipos['IPO Year'] = recent_ipos['IPO Year'].astype(int)
I tried to extract the 2015 data here using an array but I don't find an appropriate method here to extract an element from the array
ipo2015=np.array(recent_ipos['IPO Year'])
ipo2015
I am not sure if this is the right way to extract the particular year data or not??
Things I want to know here are :
How to extract year data appropriately in the graph using Plotly?
What changes I should make to solve this inconsistency in the graph?
What should I put in Y= in all the three groups bars??
How to extract the years dynamically rather than manually?
Hope to receive help from this amazing community on StackOverflow.
Thanks in advance.!!

I wrote the code under the assumption that the data on which the question is based is in data frame format. The data is taken from plotly. The query() can also be used as a variable using # as shown in the code.
import plotly.graph_objects as go
import plotly.express as px
# yyyy = [1992,1997,2002]
df = px.data.gapminder()
continent = df['continent'].unique().tolist()
yyyy = df['year'].unique().tolist()[-3:] # update
data = []
for y in yyyy:
# tmp_df = df.query('year == #y')
tmp_df = df[df['year'] == y].groupby('continent')['pop'].sum()
data.append(go.Bar(x=tmp_df.index, y=tmp_df, name=y))
# Change the bar mode
fig = go.Figure(data)
fig.update_layout(barmode='group')
fig.show()

How to make a line plot from a dataframe with multiple categorical columns in matplotlib

I want to make line chart for the different categories where one is a different country, and one is a different country for weekly based line charts. Initially, I was able to draft line plots using seaborn but it is not quite handy like setting its label, legend, color palette and so on. I am wondering is there any way to easily reshape this data with multiple categorical variables and render line charts. In initial attempt, I tried seaborn.relplot but it is not easy to tune its parameter and hard to customize the resulted plot. Can anyone point me to any efficient way to reshape dataframe with multiple categorical columns and render a clear line chart? Any thoughts?
reproducible data & my attempt:
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
url = 'https://gist.githubusercontent.com/adamFlyn/cb0553e009933574ac7ec3109ffb5140/raw/a277bc00dc08e526a7d5b7ead5425905f7206bfa/export.csv'
dff = pd.read_csv(url, parse_dates=['weekly'])
dff.drop('Unnamed: 0', axis=1, inplace=True)
df2_bf = dff.groupby(['destination', 'weekly'])['FCF_Beef'].sum().unstack()
df2_bf = df2_bf.fillna(0)
mm = df2_bf.T
mm.columns.name = None
mm = mm[~(mm.isna().sum(1)/mm.shape[1]).gt(0.9)].fillna(0)
#Total sum per column:
mm.loc['Total',:]= mm.sum(axis=0)
mm1 = mm.T
mm1 = mm1.nlargest(6, columns=['Total'])
mm1.drop('Total', axis=1, inplace=True)
mm2 = mm1.T
mm2.reset_index(inplace=True)
mm2['weekly'] = pd.to_datetime(mm2['weekly'])
mm2['year'] = mm2['weekly'].dt.year
mm2['week'] = mm2['weekly'].dt.isocalendar().week
df = mm2.melt(id_vars=['weekly','week','year'], var_name='country')
df_ = df.groupby(['country', 'year', 'week'], as_index=False)['value'].sum()
sns.relplot(data=df_, x='week', y='value', hue='year', row='country', kind='line', height=6, aspect=2, facet_kws={'sharey': False, 'sharex': False}, sizes=(20, 10))
current plot
this is one of current plot that I made with seaborn.relplot
structure of plot is okay for me, but in seaborn.replot, it is hard to tune parameter and it is as flexible as using matplotlib. Also, I realized that the way of aggregating my data is not very efficient. I think there might be a shortcut to make the above code snippet more efficient like:
plt_data = []
for i in dff.loc[:, ['FCF_Beef','FCF_Beef']]:
...
but doing this way I faced a couple of issues to make the right plot. Can anyone point me out how to make this simple and efficient in order to make the expected line chart with matplotlib? Does anyone know any better way of doing this? Any idea? Thanks
desired output
In my desired plot, first I need to iterate list of countries, where each country has one subplot, in each subplot, x-axis shows 52 weeks and y-axis shows weeklyExport amount of different years for each country. Here is draft plot that I made with seaborn.relplot.
note that, I don't like the output from seaborn.relplot, so I am wondering how can I make above attempt more efficient with matplotlib attempt. Any idea?

As requested by the OP, following is an iterative way to plot the data.
The following example plots each year, for a given 'destination' in a single figure
This is similar to the answer for this question.
import pandas as pd
import matplotlib.pyplot as plt
# load the data
url = 'https://gist.githubusercontent.com/adamFlyn/cb0553e009933574ac7ec3109ffb5140/raw/a277bc00dc08e526a7d5b7ead5425905f7206bfa/export.csv'
df = pd.read_csv(url, parse_dates=['weekly'], usecols=range(1, 6))
# groupby destination and iterate through for plotting
for g, d in df.groupby(['destination']):
# create the figure
fig, ax = plt.subplots(figsize=(7, 4))
# add lines for specific years
for year in d.weekly.dt.year.unique():
data = d[d.weekly.dt.year == year].copy() # select the data from d, by year
data['week'] = data.weekly.dt.isocalendar().week # create a week column
data.sort_values('weekly', inplace=True)
display(data.head()) # display is for jupyter, if it causes an error, use pring
data.plot(x='week', y='FCF_Beef', ax=ax, label=year)
plt.show()
Single sample plot
If we look at the tail of one of the dataframes, data.weekly.dt.isocalendar().week as putting the last day of the year as week 1, so a line is drawn back to the last data point being placed at week 1.
This function rests on datetime.datetime(2018, 12, 31).isocalendar() and is the expected behavior from the datetime module, as per this closed pandas bug.
Removing the last row with .iloc[:-1, :], is a work around
Alternatively, replace data['week'] = data.weekly.dt.isocalendar().week with data['week'] = data.weekly.dt.strftime('%W').astype('int')
data.iloc[:-1, :].plot(x='week', y='FCF_Beef', ax=ax, label=year)
Updated with all code from OP
# load the data
url = 'https://gist.githubusercontent.com/adamFlyn/cb0553e009933574ac7ec3109ffb5140/raw/a277bc00dc08e526a7d5b7ead5425905f7206bfa/export.csv'
dff = pd.read_csv(url, parse_dates=['weekly'], usecols=range(1, 6))
df2_bf = dff.groupby(['destination', 'weekly'])['FCF_Beef'].sum().unstack()
df2_bf = df2_bf.fillna(0)
mm = df2_bf.T
mm.columns.name = None
mm = mm[~(mm.isna().sum(1)/mm.shape[1]).gt(0.9)].fillna(0)
#Total sum per column:
mm.loc['Total',:]= mm.sum(axis=0)
mm1 = mm.T
mm1 = mm1.nlargest(6, columns=['Total'])
mm1.drop('Total', axis=1, inplace=True)
mm2 = mm1.T
mm2.reset_index(inplace=True)
mm2['weekly'] = pd.to_datetime(mm2['weekly'])
mm2['year'] = mm2['weekly'].dt.year
mm2['week'] = mm2['weekly'].dt.strftime('%W').astype('int')
df = mm2.melt(id_vars=['weekly','week','year'], var_name='country')
# groupby destination and iterate through for plotting
for g, d in df.groupby(['country']):
# create the figure
fig, ax = plt.subplots(figsize=(7, 4))
# add lines for specific years
for year in d.weekly.dt.year.unique():
data = d[d.weekly.dt.year == year].copy() # select the data from d, by year
data.sort_values('weekly', inplace=True)
display(data.head()) # display is for jupyter, if it causes an error, use pring
data.plot(x='week', y='value', ax=ax, label=year, title=g)
plt.show()

Plotly: How to style a plotly figure so that it doesn't display gaps for missing dates?

I have a plotly graph of the EUR/JPY exchange rate across a few months in 15 minute time intervals, so as a result, there is no data from friday evenings to sunday evenings.
Here is a portion of the data, note the skip in the index (type: DatetimeIndex) over the weekend:
Plotting this data in plotly results in a gap over the missing dates Using the dataframe above:
import plotly.graph_objs as go
candlesticks = go.Candlestick(x=data.index, open=data['Open'], high=data['High'],
low=data['Low'], close=data['Close'])
fig = go.Figure(layout=cf_layout)
fig.add_trace(trace=candlesticks)
fig.show()
Ouput:
As you can see, there are gaps where the missing dates are. One solution I've found online is to change the index to text using:
data.index = data.index.strftime("%d-%m-%Y %H:%M:%S")
and plotting it again, which admittedly does work, but has it's own problem. The x-axis labels look atrocious:
I would like to produce a graph that plots a graph like in the second plot where there are no gaps, but the x-axis is displayed like as it is on the first graph. Or at least displayed in a much more concise and responsive format, as close to the first graph as possible.
Thank you in advance for any help!

Even if some dates are missing in your dataset, plotly interprets your dates as date values, and shows even missing dates on your timeline. One solution is to grab the first and last dates, build a complete timeline, find out which dates are missing in your original dataset, and include those dates in:
fig.update_xaxes(rangebreaks=[dict(values=dt_breaks)])
This will turn this figure:
Into this:
Complete code:
import plotly.graph_objects as go
from datetime import datetime
import pandas as pd
import numpy as np
# sample data
df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv')
# remove some dates to build a similar case as in the question
df = df.drop(df.index[75:110])
df = df.drop(df.index[210:250])
df = df.drop(df.index[460:480])
# build complete timepline from start date to end date
dt_all = pd.date_range(start=df['Date'].iloc[0],end=df['Date'].iloc[-1])
# retrieve the dates that ARE in the original datset
dt_obs = [d.strftime("%Y-%m-%d") for d in pd.to_datetime(df['Date'])]
# define dates with missing values
dt_breaks = [d for d in dt_all.strftime("%Y-%m-%d").tolist() if not d in dt_obs]
# make fiuge
fig = go.Figure(data=[go.Candlestick(x=df['Date'],
open=df['AAPL.Open'], high=df['AAPL.High'],
low=df['AAPL.Low'], close=df['AAPL.Close'])
])
# hide dates with no values
fig.update_xaxes(rangebreaks=[dict(values=dt_breaks)])
fig.update_layout(yaxis_title='AAPL Stock')
fig.show()

Just in case someone here wants to remove gaps for outside trading hours and weekends,
As shown below, using rangebreaks is the way to do it.
fig = go.Figure(data=[go.Candlestick(x=df['date'], open=df['Open'], high=df['High'], low=df['Low'], close=df['Close'])])
fig.update_xaxes(
rangeslider_visible=True,
rangebreaks=[
# NOTE: Below values are bound (not single values), ie. hide x to y
dict(bounds=["sat", "mon"]), # hide weekends, eg. hide sat to before mon
dict(bounds=[16, 9.5], pattern="hour"), # hide hours outside of 9.30am-4pm
# dict(values=["2020-12-25", "2021-01-01"]) # hide holidays (Christmas and New Year's, etc)
]
)
fig.update_layout(
title='Stock Analysis',
yaxis_title=f'{symbol} Stock'
)
fig.show()
here's Plotly's doc.

thanks for the amazing sample! works on daily data but with intraday / 5min data rangebreaks only leave one day on chart
# build complete timepline
dt_all = pd.date_range(start=df.index[0],end=df.index[-1], freq="5T")
# retrieve the dates that ARE in the original datset
dt_obs = [d.strftime("%Y-%m-%d %H:%M:%S") for d in pd.to_datetime(df.index, format="%Y-%m-%d %H:%M:%S")]
# define dates with missing values
dt_breaks = [d for d in dt_all.strftime("%Y-%m-%d %H:%M:%S").tolist() if not d in dt_obs]

To fix problem with intraday data, you can use the dvalue parameter of rangebreak with the right ms value.
For example, 1 hour = 3.6e6 ms, so use dvalue with this value.
Documentation here : https://plotly.com/python/reference/layout/xaxis/
fig.update_xaxes(rangebreaks=[dict(values=dt_breaks, dvalue=3.6e6)])

Managing hourly data into monthly data for the whole year Pandas

Background
The data links which I uploaded is the time series data of the con for the whole year at a monitoring station. The format of the data is shown like this:
My target
To investigate the temporal pattern of the samples, I want to plot the variation of the monthly sample.
Like the figure below which I downloaded from plot.ly. Each box represent the daily average sample of the raw data. And the monthly average values are outlined by the lines.
With groupby function or pd.pivot function, I can get the subset of certain month or daily data easily.
But I found out that it's hard to generate a bunch of dataframes. Each one should contains the daily average data for certain month.
By pre-defining 12 empty dataframes, I can generate 12 dataframes which feed my need. But is there any neat way to divide the original dataframe and then generate multliple dataframes by user-defined conditions.
EDIT
Inspired by the answer of #alexis. I tried to achieve my target with these code. And it works for me.
## PM is the original dataset with date, hour, and values.
position = np.arange(1,13,1)
monthDict = {1:'Jan', 2:'Feb', 3:'Mar', 4:'Apr', 5:'May', 6:'Jun',
7:'Jul', 8:'Aug', 9:'Sep', 10:'Oct', 11:'Nov', 12:'Dec'}
pm['label'] = np.nan
for i in range(0,len(pm),1):
pm['label'].iloc[i] = monthDict.get(int(pm['date'].str[4:6].iloc[i]))
## Create an empty dataframe for containing the daily mean value.
df = pd.DataFrame(np.nan, index=np.arange(0,31,1), columns=['A'])
for i,t in enumerate(pm.label.unique()):
df[str(t)] = np.nan
df = df.drop(['A'],1)
mean_ = []
for i in range(0,len(pm.label.unique()),1):
month_data = pm.groupby(['label']).get_group(pm.label.unique()[i]).groupby(pm['date'].str[6:8])['pm25'].mean()
mean_.append(month_data.mean())
for j in range(0,len(month_data),1):
df[pm.label.unique()[i]].iloc[j] = month_data[j]
#### PLOT
fig = plt.figure(figsize=(12,5))
ax = plt.subplot()
bp = ax.boxplot( df.dropna().values, patch_artist=True, showfliers=False)
mo_me = plt.plot(position,mean_, marker = 'o', color ='k',markersize =6, label = 'Monthly Mean', lw = 1.5,zorder =3)
cs = ['#9BC4E1','k']
for box in bp['boxes']:
box.set(color = 'b', alpha = 1)
box.set(facecolor = cs[0], alpha = 1)
for whisker in bp['whiskers']:
whisker.set(color=cs[1], linewidth=1,linestyle = '-')
for cap in bp['caps']:
cap.set(color=cs[1], linewidth=1)
for median in bp['medians']:
median.set(color=cs[1], linewidth=1.5)
ax.set_xticklabels(pm.label.unique(), fontsize = 14)
# ax.set_yticklabels(ax.get_yticks(), fontsize = 12)
for label in ax.yaxis.get_ticklabels()[::2]:
label.set_visible(False)
for tick in ax.yaxis.get_major_ticks():
tick.label.set_fontsize(14)
plt.ylabel('Concentration', fontsize = 16, labelpad =14)
plt.xlabel('Month', fontsize = 16, labelpad =14)
plt.legend(fontsize = 14, frameon = False)
ax.set_ylim(0.0, 178)
plt.grid()
plt.show()
And this is my output figure.
Any suggestion about my code on data management or visualization would be appreciate!

Don't generate 12 dataframes. Instead of splitting your data into multiple similar variables, add a column that indicates which group each row should belong to. This is standard practice (with good reason) for database tables, dataframes, etc.
Use groupby on your dataset to group the data by month, then use apply() on the resulting DataFrameGroupBy object to restrict whatever analysis you want (e.g., the average to each group. This will also make it easy to plot the monthly results together.
You don't provide any code, so it's hard to be more specific than that. Show how you group your data by month and what you want to do to the monthly dataframes, and I'll show you how to restrict it to each month via the groupby object.