Develop Reference

Z-score calculation/standardisation using pandas

I come across this video and it bugs me.
Essentially, at 5:50, they calculate Z-score for the whole data frame by the following snippet:
df_z = (df - df.describle.T['mean'])/df.describle.T['std']
It is a neat and beautiful line.
However, df.describle.T looks like this and df looks like this
df.describle.T['mean'] and df.describle.T['std'] are two individual series, which take the df columns name as index and describle statistic parameters as columns, and df is an ordinary pd.DataFramewhich has numercial index and columns names in the right places.
My question is: how does that line make sense when they are not matching at all, in particular, how do they ensure that every variable example (x_i) matches their mean or std?
Thank you.

Pandas - select lowest value to date

I'm new to Pandas.
I've got a dataframe where I want to group by user and then find their lowest score up until that date in the their speed column.
So I can't just use df.groupby(['user'])['speed'].transform('min) as this would give the min of all values not just form the current row to the first.
What can I use to get what I need?

Without seeing your dataset it's hard to help you directly. The problem does boil down to the following. You need to select the range of data you want to work with (so select rows for the date range and columns for the user/speed).
That would look something like x = df.loc[["2-4-2018","2-4-2019"], ['users', 'speed']]
From there you could do a simple x['users'].min() for the value or x['users'].idxmin() for the index of the value.
I haven't played around for a bit with Dataframes, but you're looking for how to slice Dataframes.

loc in dataframe filtering taking lots of time

I have a dataframe Emp (Details of employess) having 3,500,000 rows and 5 columns. I have to filter Dataframe based on Emp_Name=="John". I am using loc for this purpose. But this step is taking several hours. What is the best and fastest way to filter dataframe with huge dataset?
Emp_subset=Emp.loc[Emp['Emp_Name'] == "John"]

It shouldn't be taking that long. There's no need to use loc here.
Try this and see how much it speeds things up:
emp_subset=Emp[Emp['Emp_Name'] == "John"]
Also try not to use capitals for df object names as it could lead to confusion: https://www.python.org/dev/peps/pep-0008/

How to find and add frequency column for ID?

I am a beginner at python, so bear with me!
My dataset is from excel and I was curious how to find and add a frequency column for my ID.
I first performed the groupby function for ID and date by doing:
dfcount = dfxyz.groupby(["ID", "Date"])
and then found the mean by doing:
dfcount1 = dfcount.mean()
The output i got was:
What I am trying to do is get the frequency number beside it like this:
I did not know how to copy python code, so I uploaded pictures! Sorry! Any help is appreciated for what code I can use to count the frequency for each ID AFTER I find the mean of the groupby columns.
Thank you in advance!

You can using groupby with cumcount
df['Freq']=(df.groupby(level=0).cumcount()+1).values

You can use this:
df['column_name'].value_counts()
value_counts - Returns object containing counts of unique values

Generating a list of values from a pandas DataFrame column for a range of values in another column

For a list of daily maximum temperature values from 5 to 27 degrees celsius, I want to calculate the corresponding maximum ozone concentration, from the following pandas DataFrame:
I can do this by using the following code, by changing the 5 then 6, 7 etc.
df_c=df_b[df_b['Tmax']==5]
df_c.O3max.max()
Then I have to copy and paste the output values into an excel spreadsheet. I'm sure there must be a much more pythonic way of doing this, such as by using a list comprehension. Ideally I would like to generate a list of values from the column 03max. Please give me some suggestions.

use pd.Series.map with another pd.Series
pd.Series(list_of_temps).map(df_b.set_index('Tmax')['O3max'])
You can get a dataframe
result_df = pd.DataFrame(dict(temps=list_of_temps))
result_df['O3max'] = result_df.temps.map(df_b.set_index('Tmax')['O3max'])

I had another play around and think the following piece of code seems to do the job:
df_c=df_b.groupby(['Tmax'])['O3max'].max()
I would appreciate any thoughts on whether this is correct