This question already has answers here:
Right way to reverse a pandas DataFrame?
(6 answers)
Closed 2 years ago.
I know how to iterate through the rows of a pandas DataFrame:
for id, value in df.iterrows():
but now I'd like to go through the rows in reverse order (id is numeric, but doesn't coincide with row number). Firstly I thought of doing a sort on index data.sort(ascending = False) and then running the same iteration procedure, but it didn't work (it seem to still go from smaller id to larger).
How can I accomplish this?
Iterating through a DataFrame is usually a bad idea, unless you use Cython. If you really have to, you can use the slice notation to reverse the DataFrame:
In [8]: import pandas as pd
In [9]: pd.DataFrame(np.arange(20).reshape(4,5))
Out[9]:
0 1 2 3 4
0 0 1 2 3 4
1 5 6 7 8 9
2 10 11 12 13 14
3 15 16 17 18 19
In [10]: pd.DataFrame(np.arange(20).reshape(4,5))[::-1]
Out[10]:
0 1 2 3 4
3 15 16 17 18 19
2 10 11 12 13 14
1 5 6 7 8 9
0 0 1 2 3 4
In [11]: for row in pd.DataFrame(np.arange(20).reshape(4,5))[::-1].iterrows():
...: print row
...:
(3, 0 15
1 16
2 17
3 18
4 19
Name: 3)
(2, 0 10
1 11
2 12
3 13
4 14
Name: 2)
(1, 0 5
1 6
2 7
3 8
4 9
Name: 1)
(0, 0 0
1 1
2 2
3 3
4 4
Name: 0)
Related
This question already has answers here:
Sort rows of DataFrame by duplicate
(2 answers)
Closed last year.
I have a mixed list of integers from 1 to 4 (for example 100 numbers like this). I want to sort this list like (1-2-3-4-1-2-3-4 .... 1-2-3-4). Is there a simple way I can do this with numpy or pandas?
[Edited]
#Corralien's solution helped me.
import pandas as pd
df = pd.read_csv("students.csv", encoding='utf-8')
df['A'] = df['Class']
df = df.assign(B=df.groupby('A').cumcount()).sort_values(['B', 'A']).drop(columns=['A','B'])
print(df.head(20))
Result:
Create a temporary column to sort your dataframe:
>>> df.assign(B=df.groupby('A').cumcount()).sort_values(['B', 'A']).drop(columns='B')
A
1 1
8 2
3 3
0 4
2 1
9 2
5 3
4 4
11 1
10 2
12 3
6 4
14 1
17 2
13 3
7 4
18 1
19 2
16 3
15 4
Setup:
import pandas as pd
import numpy as np
np.random.seed(2022)
df = pd.DataFrame({'A': [1,2,3,4]*5})
df = df.sample(frac=1).reset_index(drop=True)
print(df)
# Output
A
0 4
1 1
2 1
3 3
4 4
5 3
6 4
7 4
8 2
9 2
10 2
11 1
12 3
13 3
14 1
15 4
16 3
17 2
18 1
19 2
I have rand_df1:
np.random.seed(1)
rand_df1 = pd.DataFrame(np.random.randint(0, 40, size=(3, 2)), columns=list('AB'))
print(rand_df1, '\n')
A B
0 37 12
1 8 9
2 11 5
Also, rand_df2:
rand_df2 = pd.DataFrame(np.random.randint(0, 40, size=(3, 2)), columns=list('AB'))
rand_df2 = rand_df2.loc[rand_df2.index.repeat(rand_df2['B'])]
print(rand_df2, '\n')
A B
1 16 1
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
I need to reassign values in the first dataframe col 'A' with values in 'A' of the second dataframe by index. Desired output of rand_df1:
A B
0 37 12
1 16 1
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
If I've interpreted your question correctly, you are looking to append new rows onto rand_df2. These rows are to be selected from rand_df1 where they have an index which does not appear in rand_df2. Is that correct?
This will do the trick:
rand_df2_new = rand_df2.append(rand_df1[~rand_df1.index.isin(rand_df2.index)]).sort_index()
Thanks to Henry Yik for his solution:
rand_df2.combine_first(rand_df1)
A B
0 37 12
1 16 1
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
2 12 7
Also, tested this with extra column in one dataframe, that doesn't appears in second dataframe and backward situation. It works good.
I have this table:
import pandas as pd
list1 = [1,1,2,2,3,3,3,3,4,1,1,1,1,2,2]
df = pd.DataFrame(list1)
df.columns = ['A']
I want to keep maximum 3 consecutive duplicates, or keep all in case there's less than 3 (or no) duplicates.
The result should look like this:
list2 = [1,1,2,2,3,3,3,4,1,1,1,2,2]
result = pd.DataFrame(list2)
result.columns = ['A']
Use GroupBy.head with consecutive Series create by compare shifted values for not equal and cumulative sum by Series.cumsum:
df1 = df.groupby(df.A.ne(df.A.shift()).cumsum()).head(3)
print (df1)
A
0 1
1 1
2 2
3 2
4 3
5 3
6 3
8 4
9 1
10 1
11 1
13 2
14 2
Detail:
print (df.A.ne(df.A.shift()).cumsum())
0 1
1 1
2 2
3 2
4 3
5 3
6 3
7 3
8 4
9 5
10 5
11 5
12 5
13 6
14 6
Name: A, dtype: int32
Last us do
df[df.groupby(df[0].diff().ne(0).cumsum())[0].cumcount()<3]
0
0 1
1 1
2 2
3 2
4 3
5 3
6 3
8 4
9 1
10 1
11 1
13 2
14 2
Solving with itertools.groupby which groups only consecutive duplicates , then slicing 3 elements:
import itertools
pd.Series(itertools.chain.from_iterable([*g][:3] for i,g in itertools.groupby(df['A'])))
0 1
1 1
2 2
3 2
4 3
5 3
6 3
7 4
8 1
9 1
10 1
11 2
12 2
dtype: int64
Say I have a data frame that looks like this:
Id ColA
1 2
2 2
3 3
4 5
5 10
6 12
7 18
8 20
9 25
10 26
I would like my code to create a new column at the end of the DataFrame that divides the total # of obvservations by 5 ranging from 5 to 1.
Id ColA Segment
1 2 5
2 2 5
3 3 4
4 5 4
5 10 3
6 12 3
7 18 2
8 20 2
9 25 1
10 26 1
I tried the following code but doesn't work:
df['segment'] = pd.qcut(df['Id'],5)
I also want to know what would happpen if the total of my observations was not dividable by 5.
Actually, you were closer to the answer than you think. This will work regardless of whether len(df) is a multiple of 5 or not.
bins = 5
df['Segment'] = bins - pd.qcut(df['Id'], bins).cat.codes
df
Id ColA Segment
0 1 2 5
1 2 2 5
2 3 3 4
3 4 5 4
4 5 10 3
5 6 12 3
6 7 18 2
7 8 20 2
8 9 25 1
9 10 26 1
Where,
pd.qcut(df['Id'], bins).cat.codes
0 0
1 0
2 1
3 2
4 3
5 4
6 4
dtype: int8
Represents the categorical intervals returned by pd.qcut as integer values.
Another example, for a DataFrame with 7 rows.
df = df.head(7).copy()
df['Segment'] = bins - pd.qcut(df['Id'], bins).cat.codes
df
Id ColA Segment
0 1 2 5
1 2 2 5
2 3 3 4
3 4 5 3
4 5 10 2
5 6 12 1
6 7 18 1
This should work:
df['segment'] = np.linspace(1, 6, len(df), False, dtype=int)
It creates a list of int between 1 and 5 of the size of your array. If you want from 5 to 1, just add [::-1] at the end of the line.
I have a pandas datastructure, which I create like this:
test_inputs = pd.read_csv("../input/test.csv", delimiter=',')
Its shape
print(test_inputs.shape)
is this
(28000, 784)
I would like to print a subset of its rows, like this:
print(test_inputs[100:200, :])
print(test_inputs[100:200, :].shape)
However, I am getting:
TypeError: unhashable type: 'slice'
Any idea what could be wrong?
Indexing in pandas is really confusing, as it looks like list indexing but it is not. You need to use .iloc, which is indexing by position
print(test_inputs.iloc[100:200, :])
And if you don't use column selection you can omit it
print(test_inputs.iloc[100:200])
P.S. Using .loc is not what you want, as it would look not for the row number, but for the row index (which can be filled we anything, not even numbers, not even unique). Ranges in .loc will find rows with index value 100 and 200, and return the lines between. If you just created the DataFrame .iloc and .loc may give the same result, but using .loc in this case is a very bad practice as it will lead you to difficult to understand problem when the index will change for some reason (for example you'll select some subset of rows, and from that moment the row number and index will not be the same).
P.P.S. You can use test_inputs[100:200], but not test_inputs[100:200, :] because pandas designers tried to combine different popular approaches into one construction. And test_input['column'] equals to test_input.loc[:, 'column'], but surprisingly slicing with integers test_input[100:200] equals to test_inputs.iloc[100:200] (while slicing with not integer values is equivalent to loc row slicing). And if you pass a pair of values to [] it considers as a tuple for multilevel column indexing so multi_level_columns_df['level_1', 'level_2'] is equivalent to multi_level_columns_df.loc[:, ('level_1', 'level_2')]. That is why your original construction led to the error: slice can't be used as a part of multilevel index.
There is more possible solutions, but output is not same:
loc selects by labels, but iloc and slicing without function, the start bounds is included, while the upper bound is excluded, docs - select by positions:
test_inputs = pd.DataFrame(np.random.randint(10, size=(28, 7)))
print(test_inputs.loc[10:20])
0 1 2 3 4 5 6
10 3 2 0 6 6 0 0
11 5 0 2 4 1 5 2
12 5 3 5 4 1 3 5
13 9 5 6 6 5 0 1
14 7 0 7 4 2 2 5
15 2 4 3 3 7 2 3
16 8 9 6 0 5 3 4
17 1 1 0 7 2 7 7
18 1 2 2 3 5 8 7
19 5 1 1 0 1 8 9
20 3 6 7 3 9 7 1
print(test_inputs.iloc[10:20])
0 1 2 3 4 5 6
10 3 2 0 6 6 0 0
11 5 0 2 4 1 5 2
12 5 3 5 4 1 3 5
13 9 5 6 6 5 0 1
14 7 0 7 4 2 2 5
15 2 4 3 3 7 2 3
16 8 9 6 0 5 3 4
17 1 1 0 7 2 7 7
18 1 2 2 3 5 8 7
19 5 1 1 0 1 8 9
print(test_inputs[10:20])
0 1 2 3 4 5 6
10 3 2 0 6 6 0 0
11 5 0 2 4 1 5 2
12 5 3 5 4 1 3 5
13 9 5 6 6 5 0 1
14 7 0 7 4 2 2 5
15 2 4 3 3 7 2 3
16 8 9 6 0 5 3 4
17 1 1 0 7 2 7 7
18 1 2 2 3 5 8 7
19 5 1 1 0 1 8 9
print(test_inputs.values[100:200, :])
print(test_inputs.values[100:200, :].shape)
This code is also working for me.
I was facing the same problem. Even the above solutions couldn't fix it. It was some problem with pandas, What I did was I changed the array into a numpy array that fixed the issue.
import pandas as pd
import numpy as np
test_inputs = pd.read_csv("../input/test.csv", delimiter=',')
test_inputs = np.asarray(test_inputs)