Develop Reference

How Can I combine two columns is one dataframe?

I have a dataset like this.
A B C A2
1 2 3 4
5 6 7 8
and I want to combine A and A2.
A B C
1 2 3
5 6 7
4
8
how can I combine two columns?
Hope for help. Thank you.

I don't think it is possible directly. But you can do it with a few lines of code:
df = pd.DataFrame({'A':[1,5],'B':[2,6],'C':[3,7],'A2':[4,8]})
df_A2 = df[['A2']]
df_A2.columns = ['A']
df = pd.concat([df.drop(['A2'],axis=1),df_A2])
You will get this if you print df:
A B C
0 1 2.0 3.0
1 5 6.0 7.0
0 4 NaN NaN
1 8 NaN NaN

You could append the last columns after renaming it:
df.append(df[['A2']].set_axis(['A'], axis=1)).drop(columns='A2')
it gives as expected:
A B C
0 1 2.0 3.0
1 5 6.0 7.0
0 4 NaN NaN
1 8 NaN NaN

if the index is not important to you:
import pandas as pd
pd.concat([df[['A','B','C']], df[['A2']].rename(columns={'A2': 'A'})]).reset_index(drop=True)

If dataframe length exceeds threshold, make a new row python

I have a data frame with columns a,b,c,d
a b c d
1 2 nan nan
2 3 4 5
4 5 nan nan
how do i reshape into 2 columns, when i am not aware of the number of rows that the result will give. (big data)
output:
a b
1 2
2 3
4 5
4 5

Numpy's reshape
pd.DataFrame(df.values.reshape(-1, 2), columns=['a', 'b']).dropna()
a b
0 1.0 2.0
2 2.0 3.0
3 4.0 5.0
4 4.0 5.0

Backfill values by distributing values across prior NaNs in a timeseries with pandas

I have a time series where each observation represents the total amount of something since the last observation, if there is no observation in that timestep then the value is reported as NaN. An example of the format:
Timestep Value
1 10
2 NaN
3 NaN
4 9
5 NaN
6 NaN
7 NaN
8 16
9 NaN
10 NaN
What I would like to do is distribute the observed values across the NaNs prior to it. For example, a sequence like [5, NaN, NaN, 6] would become [5, 2, 2, 2] with the final observation, 6, distributed over the last 2 NaN values. Applied to the dataframe above the desired output would be:
Timestep Value
1 10
2 3
3 3
4 3
5 4
6 4
7 4
8 4
9 NaN
10 NaN
I've tried doing this with some of the pandas backfill and interpolate methods but haven't found anything which quite does what I want.

transform
df.Value.bfill().div(
df.groupby(df.Value.notna()[::-1].cumsum()).Value.transform('size')
)
0 10.0
1 3.0
2 3.0
3 3.0
4 4.0
5 4.0
6 4.0
7 4.0
8 NaN
9 NaN
Name: Value, dtype: float64
np.bincount and pd.factorize
a = df.Value.notna().values
f, u = pd.factorize(a[::-1].cumsum()[::-1])
df.Value.bfill().div(np.bincount(f)[f])
0 10.0
1 3.0
2 3.0
3 3.0
4 4.0
5 4.0
6 4.0
7 4.0
8 NaN
9 NaN
Name: Value, dtype: float64
Alternative shorter version. This works because cumsum naturally gives me what factorize does.
a = df.Value.notna().values[::-1].cumsum()[::-1]
df.Value.bfill().div(np.bincount(a)[a])
Details
In both options above, we need to identify where the null values are and use cumsum on the reversed series to define groups. In the transform option, I use groupby and size to count the size of those groups.
The second option uses bin counting and slicing to get at the same series.
Thank you #ScottBoston for reminding me to mention the reversed element [::-1]

Count the cumulative NA, then we do update
s=df.Value.notnull().cumsum().shift(1)
df.Value.update(df.Value.bfill()/s.groupby(s).transform('count'))
df
Out[885]:
Timestep Value
0 1 10.0
1 2 3.0
2 3 3.0
3 4 3.0
4 5 4.0
5 6 4.0
6 7 4.0
7 8 4.0
8 9 NaN
9 10 NaN

Find observations in which both columns are NaN and replace them with 0 in pandas DataFrame

Here is a dataframe
a b c d
nan nan 3 5
nan 1 2 3
1 nan 4 5
2 3 7 9
nan nan 2 3
I want to replace the observations in both columns 'a' and 'b' where both of them are NaNs with 0s. Rows 2 and 5 in columns 'a' and 'b' have both both NaN, so I want to replace only those rows with 0's in those matching NaN columns.
so my output must be
a b c d
0 0 3 5
nan 1 2 3
1 nan 4 5
2 3 7 9
0 0 2 3

There might be a easier builtin function in Pandas, but this one should work.
df[['a', 'b']] = df.ix[ (np.isnan(df.a)) & (np.isnan(df.b)), ['a', 'b'] ].fillna(0)
Actually the solution from #Psidom much easier to read.

You can create a boolean series based on the conditions on columns a/b, and then use loc to modify corresponding columns and rows:
df.loc[df[['a','b']].isnull().all(1), ['a','b']] = 0
df
# a b c d
#0 0.0 0.0 3 5
#1 NaN 1.0 2 3
#2 1.0 NaN 4 5
#3 2.0 3.0 7 9
#4 0.0 0.0 2 3
Or:
df.loc[df.a.isnull() & df.b.isnull(), ['a','b']] = 0

Missing data, insert rows in Pandas and fill with NAN

I'm new to Python and Pandas so there might be a simple solution which I don't see.
I have a number of discontinuous datasets which look like this:
ind A B C
0 0.0 1 3
1 0.5 4 2
2 1.0 6 1
3 3.5 2 0
4 4.0 4 5
5 4.5 3 3
I now look for a solution to get the following:
ind A B C
0 0.0 1 3
1 0.5 4 2
2 1.0 6 1
3 1.5 NAN NAN
4 2.0 NAN NAN
5 2.5 NAN NAN
6 3.0 NAN NAN
7 3.5 2 0
8 4.0 4 5
9 4.5 3 3
The problem is,that the gap in A varies from dataset to dataset in position and length...

set_index and reset_index are your friends.
df = DataFrame({"A":[0,0.5,1.0,3.5,4.0,4.5], "B":[1,4,6,2,4,3], "C":[3,2,1,0,5,3]})
First move column A to the index:
In [64]: df.set_index("A")
Out[64]:
B C
A
0.0 1 3
0.5 4 2
1.0 6 1
3.5 2 0
4.0 4 5
4.5 3 3
Then reindex with a new index, here the missing data is filled in with nans. We use the Index object since we can name it; this will be used in the next step.
In [66]: new_index = Index(arange(0,5,0.5), name="A")
In [67]: df.set_index("A").reindex(new_index)
Out[67]:
B C
0.0 1 3
0.5 4 2
1.0 6 1
1.5 NaN NaN
2.0 NaN NaN
2.5 NaN NaN
3.0 NaN NaN
3.5 2 0
4.0 4 5
4.5 3 3
Finally move the index back to the columns with reset_index. Since we named the index, it all works magically:
In [69]: df.set_index("A").reindex(new_index).reset_index()
Out[69]:
A B C
0 0.0 1 3
1 0.5 4 2
2 1.0 6 1
3 1.5 NaN NaN
4 2.0 NaN NaN
5 2.5 NaN NaN
6 3.0 NaN NaN
7 3.5 2 0
8 4.0 4 5
9 4.5 3 3

Using the answer by EdChum above, I created the following function
def fill_missing_range(df, field, range_from, range_to, range_step=1, fill_with=0):
return df\
.merge(how='right', on=field,
right = pd.DataFrame({field:np.arange(range_from, range_to, range_step)}))\
.sort_values(by=field).reset_index().fillna(fill_with).drop(['index'], axis=1)
Example usage:
fill_missing_range(df, 'A', 0.0, 4.5, 0.5, np.nan)

In this case I am overwriting your A column with a newly generated dataframe and merging this to your original df, I then resort it:
In [177]:
df.merge(how='right', on='A', right = pd.DataFrame({'A':np.arange(df.iloc[0]['A'], df.iloc[-1]['A'] + 0.5, 0.5)})).sort(columns='A').reset_index().drop(['index'], axis=1)
Out[177]:
A B C
0 0.0 1 3
1 0.5 4 2
2 1.0 6 1
3 1.5 NaN NaN
4 2.0 NaN NaN
5 2.5 NaN NaN
6 3.0 NaN NaN
7 3.5 2 0
8 4.0 4 5
9 4.5 3 3
So in the general case you can adjust the arange function which takes a start and end value, note I added 0.5 to the end as ranges are open closed, and pass a step value.
A more general method could be like this:
In [197]:
df = df.set_index(keys='A', drop=False).reindex(np.arange(df.iloc[0]['A'], df.iloc[-1]['A'] + 0.5, 0.5))
df.reset_index(inplace=True)
df['A'] = df['index']
df.drop(['A'], axis=1, inplace=True)
df.reset_index().drop(['level_0'], axis=1)
Out[197]:
index B C
0 0.0 1 3
1 0.5 4 2
2 1.0 6 1
3 1.5 NaN NaN
4 2.0 NaN NaN
5 2.5 NaN NaN
6 3.0 NaN NaN
7 3.5 2 0
8 4.0 4 5
9 4.5 3 3
Here we set the index to column A but don't drop it and then reindex the df using the arange function.

This question was asked a long time ago, but I have a simple solution that's worth mentioning. You can simply use NumPy's NaN. For instance:
import numpy as np
df[i,j] = np.NaN
will do the trick.