Develop Reference

Subtracting value from column gives NaN only

I have multiple column csv file and I want to subtract values of column X31-X27,Y31-Y27,Z31-Z27 from the same dataframe but when I am subtracting it gives me NaN values.
Here is the values of csv file:
It gives me the result as shown in picture
Help me to figure out this problem
import pandas as pd
import os
import numpy as np
df27 = pd.read_csv('D:27.txt', names=['No27','X27','Y27','Z27','Date27','Time27'], sep='\s+')
df28 = pd.read_csv('D:28.txt', names=['No28','X28','Y28','Z28','Date28','Time28'], sep='\s+')
df29 = pd.read_csv('D:29.txt', names=['No29','X29','Y29','Z29','Date29','Time29'], sep='\s+')
df30 = pd.read_csv('D:30.txt', names=['No30','X30','Y30','Z30','Date30','Time30'], sep='\s+')
df31 = pd.read_csv('D:31.txt', names=['No31','X31','Y31','Z31','Date31','Time31'], sep='\s+')
total=pd.concat([df27,df28,df29,df30,df31], axis=1)
total.to_csv('merge27-31.csv', index = False)
print(total)
df2731 = pd.read_csv('C:\\Users\\finalmerge27-31.csv')
df2731.reset_index(inplace=True)
print(df2731)
df227 = df2731[['X31', 'Y31', 'Z31']] - df2731[['X27', 'Y27', 'Z27']]
print(df227)

# input data
df = pd.DataFrame({'x27':[-1458.88, 181.78, 1911.84, 3739.3, 5358.19], 'y27':[-5885.8, -5878.1,-5786.5,-5735.7, -5545.6],
'z27':[1102,4139,4616,4108,1123], 'x31':[-1458, 181, 1911, np.nan, 5358], 'y31':[-5885, -5878, -5786, np.nan, -5554],
'z31':[1102,4138,4616,np.nan,1123]})
df
x27 y27 z27 x31 y31 z31
0 -1458.88 -5885.8 1102 -1458.0 -5885.0 1102.0
1 181.78 -5878.1 4139 181.0 -5878.0 4138.0
2 1911.84 -5786.5 4616 1911.0 -5786.0 4616.0
3 3739.30 -5735.7 4108 NaN NaN NaN
4 5358.19 -5545.6 1123 5358.0 -5554.0 1123.0
pd.DataFrame(df1.values - df2.values).rename(columns={0:'x32-x27', 1:'y31-y27', 2:'z31-x31'})
Out:
x32-x27 y31-y27 z31-x31
0 -0.88 -0.8 0.0
1 0.78 -0.1 1.0
2 0.84 -0.5 0.0
3 NaN NaN NaN
4 0.19 8.4 0.0

Pandas Query for date

I was looking through the pandas.query documentation but couldn't find anything specific about this.
Is it possible to perform a query on a date based on the closest date to the one given, instead of a specific date?
For example lets say we use the wine dataset and creates some random dates.
import pandas as pd
import numpy as np
from sklearn import datasets
dir(datasets)
df = pd.DataFrame(datasets.load_wine().data)
df.columns = datasets.load_wine().feature_names
df.columns=df.columns.str.strip()
def random_dates(start, end, n, unit='D'):
ndays = (end - start).days + 1
return pd.to_timedelta(np.random.rand(n) * ndays, unit=unit) + start
np.random.seed(0)
start = pd.to_datetime('2015-01-01')
end = pd.to_datetime('2022-01-01')
datelist=random_dates(start, end, 178)
df['Dates'] = datelist
if you perform a simple query on hue
df.query('hue == 0.6')
you'll receive three rows with three random dates. Is it possible to pick the query result that's closest to let's say 2017-1-1?
so something like
df.query('hue==0.6').query('Date ~2017-1-1')
I hope this makes sense!

You can use something like:
df.query("('2018-01-01' < Dates) & (Dates < '2018-01-31')")
# Output
alcohol malic_acid ... proline Dates
6 14.39 1.87 ... 1290.0 2018-01-24 08:21:14.665824000
41 13.41 3.84 ... 1035.0 2018-01-22 22:15:56.547561600
51 13.83 1.65 ... 1265.0 2018-01-26 22:37:26.812156800
131 12.88 2.99 ... 530.0 2018-01-01 18:58:05.118441600
139 12.84 2.96 ... 590.0 2018-01-08 13:38:26.117376000
142 13.52 3.17 ... 520.0 2018-01-19 22:37:10.170825600
[6 rows x 14 columns]
Or using #variables:
date = pd.to_datetime('2018-01-01')
offset = pd.DateOffset(days=10)
start = date - offset
end = date + offset
df.query("Dates.between(#start, #end)")
# Output
alcohol malic_acid ... proline Dates
131 12.88 2.99 ... 530.0 2018-01-01 18:58:05.118441600
139 12.84 2.96 ... 590.0 2018-01-08 13:38:26.117376000

Given a series, find the entries closest to a given date:
def closest_to_date(series, date, n=5):
date = pd.to_datetime(date)
return abs(series - date).nsmallest(n)
Then we can use the index of the returned series to select further rows (or you change the api to suit you):
(df.loc[df.hue == 0.6]
.loc[lambda df_: closest_to_date(df_.Dates, "2017-1-1", n=1).index]
)

I'm not sure if you have to use query, but this will give you the results you are looking for
df['Count'] = (df[df['hue'] == .6].sort_values(['Dates'], ascending=True)).groupby(['hue']).cumcount() + 1
df.loc[df['Count'] == 1]

matplotlib data stack on Y axis

i get this code:
fig = plt.figure(figsize=(20,8))
ax1 = fig.add_subplot(111, ylabel='Price in $')
df['Close'].plot(ax=ax1, color='r', lw=2.)
signals[['short_mavg', 'long_mavg']].plot(ax=ax1, lw=2.)
ax1.plot(signals.loc[signals.positions == 1.0].index,
signals.short_mavg[signals.positions == 1.0],
'^', markersize=10, color='m')
ax1.plot(signals.loc[signals.positions == -1.0].index,
signals.short_mavg[signals.positions == -1.0],
'v', markersize=10, color='k')
plt.show()
the problem is : all of the '^' and 'v' and 'Date" values from df, placed on Y axis =(
Ill added all code part of my jupyter notebook and .csv sample
csv data:
2013.12.17,00:00,0.89469,0.89571,0.88817,0.88973,4
2013.12.18,00:00,0.88974,0.89430,0.88200,0.88595,4
code:
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
df = pd.read_csv("AUDUSD.csv",header = None)
df.columns = ['Date', 'Time', 'Open', 'High', 'Low', 'Close', 'Volume']
df=df.set_index('Date')
second df:
short_window = 20
long_window = 90
signals = pd.DataFrame(index=df.index)
signals['signal'] = 0.0
#calculating MAs
signals['short_mavg'] = df['Close'].rolling(short_window).mean()
signals['long_mavg'] = df['Close'].rolling(long_window).mean()
signals['signal'][short_window:] = np.where(signals['short_mavg'][short_window:]
> signals['long_mavg'][short_window:], 1.0, 0.0)
signals['positions'] = signals['signal'].diff()

I created a graph with your code using Yahoo Finance currency data. It may be that the time series data is not indexed. Please check your data and my data content.
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
import yfinance as yf
ticker = yf.Ticker("AUDUSD=X")
df = ticker.history(start='2013-01-01', end='2021-01-01')
short_window = 20
long_window = 90
signals = pd.DataFrame(index=df.index)
signals['signal'] = 0.0
#calculating MAs
signals['short_mavg'] = df['Close'].rolling(short_window).mean()
signals['long_mavg'] = df['Close'].rolling(long_window).mean()
signals['signal'][short_window:] = np.where(signals['short_mavg'][short_window:]
> signals['long_mavg'][short_window:], 1.0, 0.0)
signals['positions'] = signals['signal'].diff()
fig = plt.figure(figsize=(20,8))
ax1 = fig.add_subplot(111, ylabel='Price in $')
df['Close'].plot(ax=ax1, color='r', lw=2.)
signals[['short_mavg', 'long_mavg']].plot(ax=ax1, lw=2.)
ax1.plot(signals.loc[signals.positions == 1.0].index,
signals.short_mavg[signals.positions == 1.0],
'^', markersize=10, color='m')
ax1.plot(signals.loc[signals.positions == -1.0].index,
signals.short_mavg[signals.positions == -1.0],
'v', markersize=10, color='k')
plt.show()
signals
signal short_mavg long_mavg positions
Date
2012-12-31 0.0 NaN NaN NaN
2013-01-01 0.0 NaN NaN 0.0
2013-01-02 0.0 NaN NaN 0.0
2013-01-03 0.0 NaN NaN 0.0
2013-01-04 0.0 NaN NaN 0.0
... ... ... ... ...
2020-12-25 1.0 0.749732 0.727486 0.0
2020-12-28 1.0 0.750791 0.727987 0.0
2020-12-29 1.0 0.751951 0.728454 0.0
2020-12-30 1.0 0.753096 0.728910 0.0
2020-12-31 1.0 0.754453 0.729403 0.0

Pivoting a panda data set reports multiple value error

So I have a set of data loaded into python using pandas, it looks like :
V n I
0 -0.400 0.0 -6.611865e-05
1 -0.384 0.0 -6.340880e-05
2 -0.368 0.0 -6.063826e-05
3 -0.352 0.0 -5.789697e-05
4 -0.336 0.0 -5.512495e-05
... ... ... ...
4483 0.336 83.0 1.905807e-10
4484 0.352 83.0 2.146759e-10
4485 0.368 83.0 2.452133e-10
4486 0.384 83.0 2.511581e-10
4487 0.400 83.0 2.704376e-10
[4488 rows x 3 columns]
Each data set is marked by an n value, I want to use that n value to sepearate the I and V from each other so I can plot them on the same graph. The V range are pretty much identical in each set, while the I varies.
To plot all 84 data sets on one graph to do that I used:
#store data using pandas
data = pd.read_csv( f, sep = '\t', comment = '#', names = ['V','n','I'] )
#observe data format
print(data)
#plot data
fig, ax = plt.subplots()
data = data.pivot(index='V', columns='n', values='I')
data.plot()
plt.legend(loc='best')
plt.show()
But this gives me :
ValueError: Index contains duplicate entries, cannot reshape
I tried something similar for another data set with the same structure and that worked fine but not here.
I kinda need those values even if they are identical, does anybody have ideas I can try? Thnx!

A pivot table by definition cannot accept duplicate rows or columns so you have to specify an aggregating function for these duplicated features. And you better use pivot_table as explained here
Is this what you're after?
data.csv
V n I
-0.85 0 0.060058
-0.85 0 -0.022989
-0.85 0 0.061704
-0.85 0 0.077374
-0.85 0 -0.03107
0.96 22 -0.07421
0.96 22 -0.011674
0.96 22 -0.090547
0.96 22 -0.018355
0.96 22 0.096896
-0.2 88 0.011591
-0.2 88 0.030667
-0.2 88 0.095687
-0.2 88 -0.030725
import pandas as pd, numpy as np
data = pd.read_csv( r'D:\path\data.csv' )
data_piv = pd.pivot_table(data, index=['V'], columns=['n'], values='I', aggfunc=np.mean, fill_value=0)
print(data_piv)
data_piv.plot()
You can use any aggregating function from numpy and fill_value with any value (you'll get nans if you don't specify it).

this works:
#store data using pandas
data = pd.read_csv( f, sep = '\t', comment = '#', names = ['V','n','I'] )
#observe data format
print(data)
data['n']=data['n'].astype('category')
fig, ax = plt.subplots()
sns.lineplot(data = data, x = 'V', y = 'I', hue = 'n', ax=ax)
plt.legend( bbox_to_anchor=(.05, 1),loc='upper left')
plt.show()

Tiling in groupby on dataframe

I have a data frame that contains returns, size and sedols for a couple of dates.
My goal is to identify the top and bottom values for a certain condition per date, i.e I want the top decile largest size entries and the bottom decile smallest size entries for each date and flag them in a new column by 'xx' and 'yy'.
I am confused how to apply the tiling while grouping as well as creating a new column, here is what I already have.
import pandas as pd
import numpy as np
import datetime as dt
from random import choice
from string import ascii_uppercase
def create_dummy_data(start_date, days, entries_pday):
date_sequence_lst = [dt.datetime.strptime(start_date,'%Y-%m-%d') +
dt.timedelta(days=x) for x in range(0,days)]
date_sequence_lst = date_sequence_lst * entries_pday
returns_lst = [round(np.random.uniform(low=-0.10,high=0.20),2) for _ in range(entries_pday*days)]
size_lst = [round(np.random.uniform(low=10.00,high=10000.00),0) for _ in range(entries_pday*days)]
rdm_sedol_lst = [(''.join(choice(ascii_uppercase) for i in range(7))) for x in range(entries_pday)]
rdm_sedol_lst = rdm_sedol_lst * days
dates_returns_df = pd.DataFrame({'Date':date_sequence_lst , 'Sedols':rdm_sedol_lst, 'Returns':returns_lst,'Size':size_lst})
dates_returns_df = dates_returns_df.sort_values('Date',ascending=True)
dates_returns_df = dates_returns_df.reset_index(drop=True)
return dates_returns_df
def order_df_by(df_in,column_name):
df_out = df_in.sort_values(['Date',column_name],ascending=[True,False])
return df_out
def get_ntile(df_in,ntile):
df_in['Tiled'] = df_in.groupby(['Date'])['Size'].transform(lambda x : pd.qcut(x,ntile))
return df_in
if __name__ == "__main__":
# create dummy returns
data_df = create_dummy_data('2001-01-01',31,10)
# sort by attribute
data_sorted_df = order_df_by(data_df,'Size')
#ntile data per date
data_ntiled = get_ntile(data_sorted_df, 10)
for key, item in data_ntiled:
print(data_ntiled.get_group(key))
so far I would be expecting deciled results based on 'Size' for each date, the next step would be to filter only for decile 1 and decile 10 and flag the entries 'xx' and 'yy' respectively.
thanks

Consider using transform on the pandas.qcut method with labels 1 through ntile+1 for a decile column, then conditionally set flag with np.where using decile values:
...
def get_ntile(df_in, ntile):
df_in['Tiled'] = df_in.groupby(['Date'])['Size'].transform(lambda x: pd.qcut(x, ntile, labels=list(range(1, ntile+1))))
return df_in
if __name__ == "__main__":
# create dummy returns
data_df = create_dummy_data('2001-01-01',31,10)
# sort by attribute
data_sorted_df = order_df_by(data_df,'Size')
#ntile data per date
data_ntiled = get_ntile(data_sorted_df, 10)
data_ntiled['flag'] = np.where(data_ntiled['Tiled']==1.0, 'YY',
np.where(data_ntiled['Tiled']==10.0, 'XX', np.nan))
print(data_ntiled.reset_index(drop=True).head(15))
# Date Returns Sedols Size Tiled flag
# 0 2001-01-01 -0.03 TEEADVJ 8942.0 10.0 XX
# 1 2001-01-01 -0.03 PDBWGBJ 7142.0 9.0 nan
# 2 2001-01-01 0.03 QNVVPIC 6995.0 8.0 nan
# 3 2001-01-01 0.04 NTKEAKB 6871.0 7.0 nan
# 4 2001-01-01 0.20 ZVVCLSJ 6541.0 6.0 nan
# 5 2001-01-01 0.12 IJKXLIF 5131.0 5.0 nan
# 6 2001-01-01 0.14 HVPDRIU 4490.0 4.0 nan
# 7 2001-01-01 -0.08 XNOGFET 3397.0 3.0 nan
# 8 2001-01-01 -0.06 JOARYWC 2582.0 2.0 nan
# 9 2001-01-01 0.12 FVKBQGU 723.0 1.0 YY
# 10 2001-01-02 0.03 ZVVCLSJ 9291.0 10.0 XX
# 11 2001-01-02 0.14 HVPDRIU 8875.0 9.0 nan
# 12 2001-01-02 0.08 PDBWGBJ 7496.0 8.0 nan
# 13 2001-01-02 0.02 FVKBQGU 7307.0 7.0 nan
# 14 2001-01-02 -0.01 QNVVPIC 7159.0 6.0 nan