Develop Reference

Calculating and storing the division between current row and previous row

I wanted to calculate the fractional difference between x(current value) and x(previous value) and store it into a new Pandas row for a huge table.
ID
x
1
1400
2
1500
3
1600
fractional_diff = (1500 / 1400) - 1
End Result should look something like this where first row is 0:
ID
x
fractional_diff
1
1400
0
2
1500
0.071428571
3
1600
0.066666667
I did try the following code by the way...
# Dataframe if anyone needs it real quick:
data = [['1', 1400], ['2', 1500], ['3', 1600]]
data = pd.DataFrame(data, columns=['ID', 'x'])
# Initialize the variables
n = 1
# data['fractional_diff'] = 0
# for loop to calculate fractional_diff for all rows:
for values in data['x']:
data['x'][n] = (data['x'].loc[n] / data['x'].loc[n-1]) - 1
n = n + 1
if n > len(data['x']) - 1:
break;
data.head()
But for some reason I keep getting traceback errors. Any help is appreciated.

Let us do pct_change on column x
df['change'] = df['x'].pct_change().fillna(0)
ID x change
0 1 1400 0.000000
1 2 1500 0.071429
2 3 1600 0.066667

Merging pandas.DataFrame

I need help. I've to merge this DataFrames(examples) by adding new column and put percents there.
If 'level'<5000 it's NaN, if 5000<'level'<=7000 it's 5%, if 7000<'level'<=10000 it's 7% and etc..
import pandas as pd
levels = pd.DataFrame({'lev':[5000,7000,10000],'proc':['5%','7%','10%']})
data = pd.DataFrame({'name':['A','B','C','D','E','F','G'],'sum':[6500,3000,15000,1400,8600,5409,9999]})
My efforts do solve this... It doesn't work and I don't understand how to solve this.
temp = data[data['sum'] >= levels['lev'][2]]
temp['proc']=levels['proc'][2]
lev3 = temp
temp = data[levels['lev'][1]<=data['sum'] and data['sum']<=levels['lev'][2]]
temp['proc']=levels['proc'][1]
lev2 = temp
temp = data[levels['lev'][0]<=data['sum'] and data['sum']<=levels['lev'][1]]
temp['proc']=levels['proc'][0]
lev1 = temp
data = pd.concat([lev1,lev2,lev3,data])

You can apply a function to each row like this:
import pandas as pd
def levels(s):
if 5000 < s <= 7000:
return '5%'
elif 7000 < s <= 10000:
return '7%'
elif s > 10000:
return '10%'
df = pd.DataFrame({'name':['A','B','C','D','E','F','G'],'sum':[6500,3000,15000,1400,8600,5409,9999]})
df['Percent'] = df.apply(lambda x: levels(x['sum']), axis=1)
print(df)
name sum Percent
0 A 6500 5%
1 B 3000 None
2 C 15000 10%
3 D 1400 None
4 E 8600 7%
5 F 5409 5%
6 G 9999 7%

Create dynamic column based on data in other columns

Original data:
Data Set
Day 100 200 300 400
1 2 4 6 8
2 3 5 7 9
3 4 6 8 10
Desired output:
Day Lookup Val Value
1 100 2
2 400 9
3 200 6
Basically trying to figure out how to create the second dataframe that queries the first based on the lookup value provided. Thanks in advance.

Below is one of the possible solutions. For a very large data frames it may be slow, but should work fine with average size data. I am assuming that the lookup info is loaded into some data frame. Please let me know if you have questions.
days = [1,2,3]
_100 = [2,3,4]
_200 = [4,5,6]
_300 = [6,7,8]
_400 = [8,9,10]
# this is your actual data
df = pd.DataFrame({"Day":days, "100": _100, "200": _200, "300": _300, "400": _400})
df.set_index("Day", inplace=True)
days = [1,2,3]
lookup = [100,400,200]
#this is your look up table
dfLookup = pd.DataFrame({"Day":days, "Lookup Val": lookup})
def get_value(row):
row_id = row["Day"]
lookup_id = str(row["Lookup Val"])
dfRow = df.loc[row_id].copy()
dfRow = pd.DataFrame(dfRow)
dfRow.reset_index(inplace=True)
dfRow.columns = ["lookup", "values"]
dictLookUp = dict(zip(dfRow["lookup"], dfRow["values"]))
value = dictLookUp[lookup_id]
dfRow = None
return value
dfLookup["Value"] = dfLookup.apply(lambda row: get_value(row), axis=1)

Python: How to change same numbers in a Series/Column to other values?

I am trying to change the values of a very long column (about 1mio entries) in a data frame. I have something like
####ID_Orig
3452
3452
3452
6543
6543
...
I want something like
####ID_new
0
0
0
1
1
...
At the moment I'm doing this:
j=0
for i in range(0,1199531):
if data.ID_orig[i]==data.ID_orig[i+1]:
data.ID_orig[i] = j
else:
data.ID_orig[i] = j
j=j+1
Which takes about ages... Is there a faster way to do this?
I don't know what values ID_orig has and how often a single value comes up.

Use factorize, but if duplicated groups then output values are set to same number.
Another solution with comparing by ne (!=) of shifted values with cumsum is more general - create always new values, also if repeating group values:
df['ID_new1'] = pd.factorize(df['ID_Orig'])[0]
df['ID_new2'] = df['ID_Orig'].ne(df['ID_Orig'].shift()).cumsum() - 1
print (df)
ID_Orig ID_new1 ID_new2
0 3452 0 0
1 3452 0 0
2 3452 0 0
3 6543 1 1
4 6543 1 1
5 100 2 2
6 100 2 2
7 6543 1 3 <-repeating group
8 6543 1 3 <-repeating group

You can do this …
import collections
l1 = [3452, 3452, 3452, 6543, 6543]
c = collections.Counter(l1)
l2 = list(c.items())
l3 = []
for i, t in enumerate(l2):
for x in range(t[1]):
l3.append(i)
for x in l3:
print(x)
This is the output:
0
0
0
1
1

You can use the following. In the following implementation duplicate ids in the original id will get same ids. The implementation is based on dropping duplicates from the column and assigning a different number to each unique id to form the enw ids. These new ids are then merged into the original dataset
import numpy as np
import pandas as pd
from time import time
num_rows = 119953
input_data = np.random.randint(1199531, size=(num_rows,1))
data = pd.DataFrame(input_data)
data.columns = ["ID_orig"]
data2 = pd.DataFrame(input_data)
data2.columns = ["ID_orig"]
t0 = time()
j=0
for i in range(0,num_rows-1):
if data.ID_orig[i]==data.ID_orig[i+1]:
data.ID_orig[i] = j
else:
data.ID_orig[i] = j
j=j+1
t1 = time()
id_new = data2.loc[:,"ID_orig"].drop_duplicates().reset_index().drop("index", axis=1)
id_new.reset_index(inplace=True)
id_new.columns = ["id_new"] + id_new.columns[1:].values.tolist()
data2 = data2.merge(id_new, on="ID_orig")
t2 = time()
print("Previous: ", round(t1-t0, 2), " seconds")
print("Current : ", round(t2-t1, 2), " seconds")
The output of the above program using only 119k rows is
Previous: 12.16 seconds
Current : 0.06 seconds
The runtime difference increases even more as the number of rows are increased.
EDIT
Using the same number of rows:
>>> print("Previous: ", round(t1-t0, 2))
Previous: 11.7
>>> print("Current : ", round(t2-t1, 2))
Current : 0.06
>>> print("jezrael's answer : ", round(t3-t2, 2))
jezrael's answer : 0.02

Python pandas resampling

I have the following dataframe:
Timestamp S_time1 S_time2 End_Time_1 End_time_2 Sign_1 Sign_2
0 2413044 0 0 0 0 x x
1 2422476 0 0 0 0 x x
2 2431908 0 0 0 0 x x
3 2441341 0 0 0 0 x x
4 2541232 2526631 2528631 2520631 2530631 10 80
5 2560273 2544946 2546496 2546496 2548496 40 80
6 2577224 2564010 2566010 2566010 2568010 null null
7 2592905 2580959 2582959 2582959 2584959 null null
The table goes on and on like that. The first column is a timestamp which is in milliseconds. S_time1 and End_time_1 are the duration where a particular sign (number) appear. For example, if we take the 5th row, S_time1 is 2526631, End_time_1 is 2520631, and the corresponding sign_1 is 10, which means from 2526631 to 2520631 the sign 10 will be displayed. And the same thing goes to S_time2 and End_time_2. The corresponding values in sign_2 will appear in the duration from S_time2 to End_time_2.
I want to resample the index column (Timestamp) in 100-millisecond bin time and check in which bin times the signs belong. For instance, between each start time and end time there is 2000 milliseconds difference. So the corresponding sign number will appear repeatedly in around 20 consecutive bin times because each bin time is 100 millisecond. So I need to have two columns only: one with the bin times and the second with the signs. Looks like the following table: (I am just making up the bin time just for example)
Bin_time signs
...100 0
...200 0
...300 10
...400 10
...500 10
...600 10
The sign 10 will be for the duration of the corresponding S_time1 to End_time_1. Then the next sign which is 80 continues for the duration of S_time2 to End_time_2. I am not sure if this can be done in pandas or not. But I really need help either in pandas or other methods.
Thanks for your help and suggestion in advance.

Input:
print df
Timestamp S_time1 S_time2 End_Time_1 End_time_2 Sign_1 Sign_2
0 2413044 0 0 0 0 x x
1 2422476 0 0 0 0 x x
2 2431908 0 0 0 0 x x
3 2441341 0 0 0 0 x x
4 2541232 2526631 2528631 2520631 2530631 10 80
5 2560273 2544946 2546496 2546496 2548496 40 80
6 2577224 2564010 2566010 2566010 2568010 null null
7 2592905 2580959 2582959 2582959 2584959 null null
2 approaches:
In [231]: %timeit s(df)
1 loops, best of 3: 2.78 s per loop
In [232]: %timeit m(df)
1 loops, best of 3: 690 ms per loop
def m(df):
#resample column Timestamp by 100ms, convert bak to integers
df['Timestamp'] = df['Timestamp'].astype('timedelta64[ms]')
df['i'] = 1
df = df.set_index('Timestamp')
df1 = df[[]].resample('100ms', how='first').reset_index()
df1['Timestamp'] = (df1['Timestamp'] / np.timedelta64(1, 'ms')).astype(int)
#felper column i for merging
df1['i'] = 1
#print df1
out = df1.merge(df,on='i', how='left')
out1 = out[['Timestamp', 'Sign_1']][(out.Timestamp >= out.S_time1) & (out.Timestamp <= out.End_Time_1)]
out2 = out[['Timestamp', 'Sign_2']][(out.Timestamp >= out.S_time2) & (out.Timestamp <= out.End_time_2)]
out1 = out1.rename(columns={'Sign_1':'Bin_time'})
out2 = out2.rename(columns={'Sign_2':'Bin_time'})
df = pd.concat([out1, out2], ignore_index=True).drop_duplicates(subset='Timestamp')
df1 = df1.set_index('Timestamp')
df = df.set_index('Timestamp')
df = df.reindex(df1.index).reset_index()
#print df.head(10)
def s(df):
#resample column Timestamp by 100ms, convert bak to integers
df['Timestamp'] = df['Timestamp'].astype('timedelta64[ms]')
df = df.set_index('Timestamp')
out = df[[]].resample('100ms', how='first')
out = out.reset_index()
out['Timestamp'] = (out['Timestamp'] / np.timedelta64(1, 'ms')).astype(int)
#print out.head(10)
#search start end
def search(x):
mask1 = (df.S_time1<=x['Timestamp']) & (df.End_Time_1>=x['Timestamp'])
#if at least one True return first value of series
if mask1.any():
return df.loc[mask1].Sign_1[0]
#check second start and end time
else:
mask2 = (df.S_time2<=x['Timestamp']) & (df.End_time_2>=x['Timestamp'])
if mask2.any():
#if at least one True return first value
return df.loc[mask2].Sign_2[0]
else:
#if all False return NaN
return np.nan
out['Bin_time'] = out.apply(search, axis=1)
#print out.head(10)