Develop Reference

How to create lists from pandas columns

I have created a pandas dataframe using this code:
import numpy as np
import pandas as pd
ds = {'col1': [1,2,3,3,3,6,7,8,9,10]}
df = pd.DataFrame(data=ds)
The dataframe looks like this:
print(df)
col1
0 1
1 2
2 3
3 3
4 3
5 6
6 7
7 8
8 9
9 10
I need to create a field called col2 that contains in a list (for each record) the last 3 elements of col1 while iterating through each record. So, the resulting dataframe would look like this:
Does anyone know how to do it by any chance?

Here is a solution using rolling and list comprehension
df['col2'] = [x.tolist() for x in df['col1'].rolling(3)]
col1 col2
0 1 [1]
1 2 [1, 2]
2 3 [1, 2, 3]
3 3 [2, 3, 3]
4 3 [3, 3, 3]
5 6 [3, 3, 6]
6 7 [3, 6, 7]
7 8 [6, 7, 8]
8 9 [7, 8, 9]
9 10 [8, 9, 10]

Use a list comprehension:
N = 3
l = df['col1'].tolist()
df['col2'] = [l[max(0,i-N+1):i+1] for i in range(df.shape[0])]
Output:
col1 col2
0 1 [1]
1 2 [1, 2]
2 3 [1, 2, 3]
3 3 [2, 3, 3]
4 3 [3, 3, 3]
5 6 [3, 3, 6]
6 7 [3, 6, 7]
7 8 [6, 7, 8]
8 9 [7, 8, 9]
9 10 [8, 9, 10]

Upon seeing the other answers, I'm affirmed my answer is pretty stupid.
Anyways, here it is.
import pandas as pd
ds = {'col1': [1,2,3,3,3,6,7,8,9,10]}
df = pd.DataFrame(data=ds)
df['col2'] = df['col1'].shift(1)
df['col3'] = df['col2'].shift(1)
df['col4'] = (df[['col3','col2','col1']]
.apply(lambda x:','.join(x.dropna().astype(str)),axis=1)
)
The last column contains the resulting list.
col1 col4
0 1 1.0
1 2 1.0,2.0
2 3 1.0,2.0,3.0
3 3 2.0,3.0,3.0
4 3 3.0,3.0,3.0
5 6 3.0,3.0,6.0
6 7 3.0,6.0,7.0
7 8 6.0,7.0,8.0
8 9 7.0,8.0,9.0
9 10 8.0,9.0,10.0

lastThree = []
for x in range(len(df)):
lastThree.append([df.iloc[x - 2]['col1'], df.iloc[x - 1]['col1'], df.iloc[x]['col1']])
df['col2'] = lastThree

Pandas replace with dict and condition

In Pandas in Python you have the function df.replace(), which you can give a dict to change the values in a column:
df = pd.DataFrame({'A': [0, 1, 2, 3, 4],
'B': [5, 6, 7, 8, 9],
'C': ['a', 'b', 'c', 'd', 'e']})
df.replace('A': {0: 10, 3: 100})
Is it possible to add a condition to this? For example that it will only replace the values in the A column if the value in the B column is smaller than 8.

Using where:
df['A'] = df['A'].replace({0: 10, 3: 100}).where(df['B'].lt(8), df['A'])
output:
A B C
0 10 5 a
1 1 6 b
2 2 7 c
3 3 8 d
4 4 9 e

Try this:
df.update(df['A'][df['B'] < 8].replace({0: 10, 3: 100}))
Output:
>>> df
A B C
0 10.0 5 a
1 1.0 6 b
2 2.0 7 c
3 3.0 8 d
4 4.0 9 e
Notice how A at row 3 is not 100, but 3.0 (the old value). Because B at row 3 is 8, which, per your condition, is not less then 8.

Add new column in the Dataset with condition?

I am new in pandas and python, but I want to add new column that collect all row data and paste it in new column for example:
df_final = pd.read_csv('df_final.csv')
House_No = df_final['House_No_'].copy()
Street = df_final['Street'].copy()
City = df_final['City'].copy()
District = df_final['District'].copy()
Postl_Code = df_final['Postl_Code'].copy()
df_final['Full_Address']=(House_No +' , '+ Street +' , '+ City +' , '+ District +' , '+ str(Postl_Code))
the output is :
when the House No is null the new cell become null .. find it in row 7,8 and 9 in the image.
how can ignore the null cell and just take the rest of the roe ??
Thank you in advance.

import numpy as np
import pandas as pd
# sample data
np.random.seed(1)
df = pd.DataFrame(np.random.randint(1,10, size=(5,5)), columns=list('abcde'))
df.iloc[2, 0] = np.nan
# use np.where with join
df['new_col'] = np.where(df['a'].isna(), df.iloc[:, 1:].astype(str).apply(', '.join, axis=1),
df.astype(str).apply(', '.join, axis=1))
a b c d e new_col
0 6.0 9 6 1 1 6.0, 9, 6, 1, 1
1 2.0 8 7 3 5 2.0, 8, 7, 3, 5
2 NaN 3 5 3 5 3, 5, 3, 5
3 8.0 8 2 8 1 8.0, 8, 2, 8, 1
4 7.0 8 7 2 1 7.0, 8, 7, 2, 1
or if you do not care if the nan is in the final output simply do:
df['new_col1'] = df.astype(str).apply(', '.join, axis=1)
a b c d e new_col1
0 6.0 9 6 1 1 6.0, 9, 6, 1, 1
1 2.0 8 7 3 5 2.0, 8, 7, 3, 5
2 NaN 3 5 3 5 nan, 3, 5, 3, 5
3 8.0 8 2 8 1 8.0, 8, 2, 8, 1
4 7.0 8 7 2 1 7.0, 8, 7, 2, 1

Divide numbers in same column in python

In Python, I have a dataframe and want to divide the numbers in same column.(such as divide 6 by 9 in column c)
df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),columns=['a', 'b', 'c'])
df2
a b c
0 1 2 3
1 4 5 6
2 7 8 9

create a dataframe from 3 other dataframes in python

I am trying to create a new df which summarises my key information, by taking that information from 3 (say) other dataframes.
dfdate = {'x1': [2, 4, 7, 5, 6],
'x2': [2, 2, 2, 6, 7],
'y1': [3, 1, 4, 5, 9]}
dfdate = pd.DataFrame(df, index=range(0:4))
dfqty = {'x1': [1, 2, 6, 6, 8],
'x2': [3, 1, 1, 7, 5],
'y1': [2, 4, 3, 2, 8]}
dfqty = pd.DataFrame(df2, range(0:4))
dfprices = {'x1': [0, 2, 2, 4, 4],
'x2': [2, 0, 0, 3, 4],
'y1': [1, 3, 2, 1, 3]}
dfprices = pd.DataFrame(df3, range(0:4))
Let us say the above 3 dataframes are my data. Say, some dates, qty, and prices of goods. My new df is to be constructed from the above data:
rng = len(dfprices.columns)*len(dfprices.index) # This is the len of new df
dfnew = pd.DataFrame(np.nan,index=range(0,rng),columns=['Letter', 'Number', 'date', 'qty', 'price])
Now, this is where I struggle to piece my stuff together. I am trying to take all the data in dfdate and put it into a column in the new df. same with dfqty and dfprice. (so 3x5 matricies essentially goto a 1x15 vector and are placed into the new df).
As well as that, I need a couple of columns in dfnew as identifiers, from the names of the columns of the old df.
Ive tried for loops but to no avail, and don't know how to convert a df to series. But my desired output is:
dfnew:
'Lettercol','Numbercol', 'date', 'qty', 'price'
0 X 1 2 1 0
1 X 1 4 2 2
2 X 1 7 6 2
3 X 1 5 6 4
4 X 1 6 8 4
5 X 2 2 3 2
6 X 2 2 1 0
7 X 2 2 1 0
8 X 2 6 7 3
9 X 2 7 5 4
10 Y 1 3 2 1
11 Y 1 1 4 3
12 Y 1 4 3 2
13 Y 1 5 2 1
14 Y 1 9 8 3
where the numbers 0-14 are the index.
letter = letter from col header in DFs
number = number from col header in DFs
next 3 columns are data from the orig df's
(don't ask why the original data is in that funny format :)
thanks so much. my last Q wasn't well received so have tried to make this one better, thanks

Use:
#list of DataFrames
dfs = [dfdate, dfqty, dfprices]
#list comprehension with reshape
comb = [x.unstack() for x in dfs]
#join together
df = pd.concat(comb, axis=1, keys=['date', 'qty', 'price'])
#remove second level of MultiIndex and index to column
df = df.reset_index(level=1, drop=True).reset_index().rename(columns={'index':'col'})
#extract all values without first by indexing [1:] and first letter by [0]
df['Number'] = df['col'].str[1:]
df['Letter'] = df['col'].str[0]
cols = ['Letter', 'Number', 'date', 'qty', 'price']
#change order of columns
df = df.reindex(columns=cols)
print (df)
Letter Number date qty price
0 x 1 2 1 0
1 x 1 4 2 2
2 x 1 7 6 2
3 x 1 5 6 4
4 x 1 6 8 4
5 x 2 2 3 2
6 x 2 2 1 0
7 x 2 2 1 0
8 x 2 6 7 3
9 x 2 7 5 4
10 y 1 3 2 1
11 y 1 1 4 3
12 y 1 4 3 2
13 y 1 5 2 1
14 y 1 9 8 3