This question already has answers here:
How is numpy's fancy indexing implemented?
(3 answers)
Closed 5 years ago.
Please I don't understant how this syntax works: current_x = X[Y == c].
I've printed out in Ipython some sample with the code (I use pandas here as pd):
In [120]: pd.DataFrame(X)
Out[120]:
0 1 2 3 4 5 6 7 8 9 ... 475 \
0 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
1 0.070588 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
2 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
3 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
4 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
5 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
6 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
7 0.596078 0.596078 0.062745 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
8 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
9 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
476 477 478 479 480 481 482 483 484
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
[10 rows x 485 columns]
In [121]: pd.DataFrame(Y)
Out[121]:
0
0 1
1 7
2 9
3 4
4 5
5 1
6 1
7 6
8 1
9 1
In [122]: pd.DataFrame(X[Y == 5])
Out[122]:
0 1 2 3 4 5 6 7 8 9 ... 475 476 477 478 \
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0
479 480 481 482 483 484
0 0.0 0.0 0.0 0.0 0.0 0.0
[1 rows x 485 columns]
I don't How X[Y == 5] has come with that result.
Please hep-lp
# this returns a list of True or False depending if Y[i] == 5 for all i in Y
Y==5 => [False, False, False, False, True, False, False, False, False, False]
# this is known as boolean indexing ...
X[ [False, False, False, False, True, False, False, False, False, False] ] => your output
Related
Here I have a dataset:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
14876 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14877 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14878 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14879 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14880 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
The Y-axis represents seconds, and the X-axis represents binned size ranges. This is data for cloud particle concentration. So for every second, there are 42 pieces of data that represent the number of particles that exist within a certain size range.
Each column represents a certain size range as I already said and those ranges for example, are:
(micrometers)
0 = 0.0 to 20.0
1 = 20.0 to 40.0
2 = 40.0 to 60.0
3 = 60.0 to 80.0
4 = 80.0 to 100.0
5 = 100.0 to 125.0
6 = 125.0 to 150.0
7 = 150.0 to 200.0
8 = 200.0 to 250.0
9 = 250.0 to 300.0
10 = 300.0 to 350.0
11 = 350.0 to 400.0
12 = 400.0 to 475.0
ect...
The reason I included so many is I want to show how the bins are spaced. The width of the bins increases, and the increase in width does not follow any sort of formula.
What I am wanting to do is replace the index for each column on the X-axis with these binned size ranges, and create a filled contour plot very similar to this:
I am using a pandas dataframe to store the dataset and I am currently using pyplot to attempt some plotting using pcolormesh.
Edit: Here is my attempt at starting with this.
#reading dataset extracted using h5py into pandas dataframe
df = pd.DataFrame(ds_arr)
df = df.replace(-999.0, 0)
#creating list for bin midpoints
strcols = [10.0, 30.0, 50.0, 70.0, 90.0, 112.5, 137.5, 175.0, 225.0, 275.0, 325.0, 375.0, 437.5, 512.5, 587.5, 662.5, 750.0, 850.0, 950.0, 1100.0, 1300.0, 1500.0, 1700.0, 2000.0, 2400.0, 2800.0, 3200.0, 3600.0, 4000.0, 4400.0, 4800.0, 5500.0, 6500.0, 7500.0, 8500.0, 9500.0, 11000.0, 13000.0, 15000.0, 17000.0, 19000.0, 22500.0]
#add new column to end of df
newdf = df
newdf['midpoints'] = strcols
#set the index to be the new column, and delete the name.
newdf.set_index('midpoints', drop=True, inplace=True)
newdf.index.name=None
#getting bins on the X-axis
newdf = newdf.T
print(newdf)
#check data type of indicies
print('\ncolumns are:',type(newdf.columns),)
print('rows are:',type(newdf.index))
#creating figure
fig, ax = plt.subplots(figsize=(13, 5))
fig.tight_layout(pad = 6)
#setting colormap, .copy so you can modify it inplace without an error
cmap = cm.gnuplot2.copy()
cmap.set_bad(color='black')
#plotting data using pcolormesh
plot = ax.pcolormesh(newdf, norm = mpl.colors.LogNorm(), cmap = cmap)
plt.title('Number Concentration', pad=12.8)
plt.xlabel("Bins", rotation=0, labelpad=17.5)
plt.ylabel("Time(Seconds)", labelpad=8.5)
cb = plt.colorbar(plot, shrink=1, aspect=25, location='right')
cb.ax.set_title('#/m4', pad=10, fontsize=10.5)
plt.show()
The resulting dataframe, where I use the midpoints of my desired bins as the header labels, looks like this:
10.0 30.0 50.0 70.0 90.0 112.5 137.5 175.0 225.0 275.0 325.0 375.0 437.5 ... 4400.0 4800.0 5500.0 6500.0 7500.0 8500.0 9500.0 11000.0 13000.0 15000.0 17000.0 19000.0 22500.0
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
14876 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14877 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14878 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14879 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14880 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
And this is the plot I generated:
plot
The problem here is that the tick marks on the X-axis are not matching with the column headers in my resulting dataset.
Here is the output when I check what type of data my indices are:
columns are: <class 'pandas.core.indexes.numeric.Float64Index'>
rows are: <class 'pandas.core.indexes.base.Index'>
To be clear, my goal is to be able to give each column a bin width, where the range of size data on the X-axis starts at 0 and ends at the endpoint of my last bin. I was to be able to hardcode the bin widths for each column individually. I would also be able to display the bins logarithmically scaled, or anything similar.
How should I configure my dataframe to be able to output a plot similar to the example plot, with unevenly spaced yet logarithmically scaled binned data?
I have dataframe df_reps
RepID RepText
================
1328 Hello, this is me, ..
5744 This is a test reoprt, ..
8417 The UK has begun sending ventilators and oxygen ..
I am tryingto extract TFIDF values from RepText
and Here is my code
def TFIDF2(df, use_idf, smooth_idf, ngram_range, stop_words):
tf_idf_vec = TfidfVectorizer(use_idf=use_idf,
smooth_idf=smooth_idf,
ngram_range=ngram_range,stop_words=stop_words) # to use only bigrams ngram_range=(2,2)
tf_idf_data = tf_idf_vec.fit_transform(df)
tf_idf_dataframe = pd.DataFrame(tf_idf_data.toarray(),columns=tf_idf_vec.get_feature_names())
return tf_idf_dataframe
df_tfidf = TFIDF2(df_reps["RepText"], True, False, (1,1), "english")
But df_tfidf looks like this
df_tfidf
Out[25]:
UK reoort test ... begun sending Hello
0 0.0 0.0 0.0 ... 0.0 0.0 0.0
1 0.0 0.0 0.0 ... 0.0 0.0 0.0
2 0.0 0.0 0.0 ... 0.0 0.0 0.0
3 0.0 0.0 0.0 ... 0.0 0.0 0.0
4 0.0 0.0 0.0 ... 0.0 0.0 0.0
The problem is the index in df_tfidf is not related to RepID in df_report
I want to get the index of df_tfidf to be the RepID so it lools like this
RepID UK reoort test ... begun sending Hello
1328 0.0 0.0 0.0 ... 0.0 0.0 0.0
5744 0.0 0.0 0.0 ... 0.0 0.0 0.0
8417 0.0 0.0 0.0 ... 0.0 0.0 0.0
8823 0.0 0.0 0.0 ... 0.0 0.0 0.0
9938 0.0 0.0 0.0 ... 0.0 0.0 0.0
I have dataframe which looks like below:
df:
Review_Text Noun Thumbups
Would be nice to be able to import files from ... [My, Tracks, app, phone, Google, Drive, import... 1.0
No Offline Maps! It used to have offline maps ... [Offline, Maps, menu, option, video, exchange,... 18.0
Great application. Designed with very well tho... [application, application] 16.0
Great App. Nice and simple but accurate. Wish ... [Great, App, Nice, Exported] 0.0
Save For Offline - This does not work. The rou... [Save, Offline, route, filesystem] 12.0
Since latest update app will not run. Subscrip... [update, app, Subscription, March, application] 9.0
Great app. Love it! And all the things it does... [Great, app, Thank, work] 1.0
I have paid for subscription but keeps telling... [subscription, trial, period] 0.0
Error: The route cannot be save for no locatio... [Error, route, i, GPS] 0.0
When try to restore my tracks it says "unable ... [try, file, locally-1] 0.0
Was a good app but since the update it only re... [app, update, metre] 2.0
based on 'Noun' Column values, I want to create other columns. For example, all values of noun column from first row become columns and those columns contain value of 'Thumbups' column value. If the column name already present in dataframe then it adds 'Thumbups' value into the existing value of the column.
I was trying to implement by using pivot_table :
pd.pivot_table(latest_review,columns='Noun',values='Thumbups')
But got following error:
TypeError: unhashable type: 'list'
Could anyone help me in fixing the issue?
Use Series.str.join with Series.str.get_dummies for dummies and then multiple by column Thumbups by DataFrame.mul:
df1 = df['Noun'].str.join('|').str.get_dummies().mul(df['Thumbups'], axis=0)
print (df1)
App Drive Error Exported GPS Google Great Maps March My Nice \
0 0.0 10.0 0.0 0.0 0.0 10.0 0.0 0.0 0.0 10.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 180.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 90.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
Offline Save Subscription Thank Tracks app application exchange \
0 0.0 0.0 0.0 0.0 10.0 10.0 0.0 0.0
1 180.0 0.0 0.0 0.0 0.0 0.0 0.0 180.0
2 0.0 0.0 0.0 0.0 0.0 0.0 160.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 120.0 120.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 90.0 0.0 0.0 90.0 90.0 0.0
6 0.0 0.0 0.0 10.0 0.0 10.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 NaN NaN NaN NaN NaN NaN NaN NaN
file filesystem i import locally-1 menu metre option period \
0 0.0 0.0 0.0 10.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 180.0 0.0 180.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 120.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 NaN NaN NaN NaN NaN NaN NaN NaN NaN
phone route subscription trial try update video work
0 10.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 180.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 120.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 90.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.0
7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 NaN NaN NaN NaN NaN NaN NaN NaN
rows = []
#_unpacking Noun column row list values and storing it in rows list
_ = df.apply(lambda row: [rows.append([row['Review_Text'],row['Thumbups'], nn])
for nn in row.Noun], axis=1)
#_creates new dataframe with unpacked values
df_new = pd.DataFrame(rows, columns=df.columns)
#_now doing pivot operation on df_new
pivot_df = df_new.pivot(index='Review_Text', columns='Noun')
I have a Pandas Dataframe which tells me monthly sales of items in shops
df.head():
ID month sold
0 150983 0 1.0
1 56520 0 13.0
2 56520 1 7.0
3 56520 2 13.0
4 56520 3 8.0
I want to remove all IDs where there were no sales last month. I.e. month == 33 & sold == 0. Doing the following
unwanted_df = df[((df['month'] == 33) & (df['sold'] == 0.0))]
I just get 46 rows, which is far too little. But nevermind, I would like to have the data in different format anyway. Pivoted version of above table is just what I want:
pivoted_df = df.pivot(index='month', columns = 'ID', values = 'sold').fillna(0)
pivoted_df.head()
ID 0 2 3 5 6 7 8 10 11 12 ... 214182 214185 214187 214190 214191 214192 214193 214195 214197 214199
month
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Question. How to remove columns with the value 0 in the last row in pivoted_df?
You can do this with one line:
pivoted_df= pivoted_df.drop(pivoted_df.columns[pivoted_df.iloc[-1,:]==0],axis=1)
I want to remove all IDs where there were no sales last month
You can first calculate the IDs satisfying your condition:
id_selected = df.loc[(df['month'] == 33) & (df['sold'] == 0), 'ID']
Then filter these from your dataframe via a Boolean mask:
df = df[~df['ID'].isin(id_selected)]
Finally, use pd.pivot_table with your filtered dataframe.
I have the following numpy matrix:
0 1 2 3 4 5 6 7 8 9
0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 5.0 0.0 9.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 7.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 5.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0
8 2.0 0.0 0.0 0.0 3.0 0.0 6.0 0.0 8.0 0.0
9 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
I want to calculate the non-zero values average of every row and column separately. So my result should be something like this:
average_rows = [1.0,7.0,2.0,5.0,0.0,4.0,0.0,5.5,4.75,1.0,0.0]
average_cols = [3.5,1.0,4.33333,0.0,4.33333,0.0,4.0,6.0,6.5,0.0]
I can't figure out how to iterate over them, and I keep getting TypeError: unhashable type
Also, I'm not sure if iterating is the best solution, I also tried something like R[:,i] to grab each column and sum it using sum(R[:,i]), but keep getting the same error.
It is better to use 2d np.array instead of matrix.
import numpy as np
data = np.array([[1, 2, 0], [0, 0, 1], [0, 2, 4]], dtype='float')
data[data == 0] = np.nan
# replace all zeroes with `nan`'s to skip them
# [[ 1. 2. nan]
# [ nan nan 1.]
# [ nan 2. 4.]]
np.nanmean(data, axis=0)
# array([ 1. , 2. , 2.5])
np.nanmean(data, axis=1)
# array([ 1.5, 1. , 3. ])