Develop Reference

How to multiply two data frame column by column?

I have to similarity between columns within the one data frame. The expected result is the same as the correlation matrix output, but the calculation function is different(I wrote my self calc function). So the calc function should get calc_func(column1, column2). The idea is to get similarities between columns. row size is not important. as an Output I expect (937,937) matrix.
Sample data
0011 0012 0013 0014 0015 0019 0111 0112 0121 0122 0123 0125 0129 0161 0168 0172 0174 0175 0176 0221 0222 0223 0224 0230 0241 0242 0243 0249 0251 0252 0341 0342 0344 0345 0351 0352 0353 0361 0362 0363 0371 0372 0411 0412 0421 0422 0423 0430 0441 0449 0452 0453 0459 0461 0471 0472 0481 0482 0483 0484 0485 0541 0542 0544 0545 0546 0547 0548 0561 0564 0566 0567 0571 0572 0573 0574 0575 0576 0577 0579 0581 0583 \
Reporter ISO
AFG 0.149474 0.699753 0.000000e+00 0.000000 0.000000 6.084805 0.000000e+00 0.013655 0.123035 0.000000e+00 0.000000 0.011263 0.000000 0.000000e+00 0.000000 0.000000e+00 0.000000 0.000000 0.000000 0.000000 0.000000 0.040835 0.009775 0.000000 0.000000 0.000000 1.902343e-04 0.003110 0.000000 0.000000 9.900480e-04 0.000382 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.002613 0.002373 0.000184 0.000000 0.000000e+00 5.570409e-04 0.000000 0.001367 0.023009 1.074305 0.000000 4.309246e-04 2.267049 0.135528 6.845710 0.000172 4.785010e-02 5.620574e-04 0.015391 0.000000 0.000000 0.008071 0.000000 0.602458 56.772035 4.902713e+01 11.542497 0.175537 0.000000 8.314311 6.899700e-01 0.009341 0.000000e+00 0.118446 0.465433 0.634222 0.008141 4.406345e+01 1.806608e+02 1457.266474 37.572639 16.111153 0.278868 5.828552e-01
AGO 0.000233 0.000000 2.169950e-05 0.000436 0.000021 0.206904 1.850937e-05 0.001081 0.000054 4.163925e-04 0.000437 0.000348 0.000059 1.287730e-04 0.000289 9.425705e-04 0.000140 0.002698 0.000444 0.000116 0.002252 0.000614 0.000295 0.000481 0.000003 0.000008 0.000000e+00 0.000142 0.000742 0.002136 3.700936e-01 1.594887 0.024370 0.000002 0.039695 0.146148 0.000020 0.267286 0.866269 0.036852 0.000384 0.046401 4.496454e-06 0.000000e+00 0.000834 0.016216 0.001110 0.000000 0.000065 7.354149e-04 0.000061 0.004332 0.000039 0.239055 2.501597e-01 3.830196e-04 0.000546 0.008450 0.015806 0.001086 0.002724 0.009187 0.005919 3.321169e-04 0.002146 0.000693 0.001050 0.006553 2.621612e-03 0.017289 8.732829e-05 0.000309 0.000343 0.000303 0.053893 5.683467e-04 6.637084e-05 0.000005 0.000036 0.001527 0.000022 1.886017e-07
ALB 0.004472 0.093826 0.000000e+00 0.000000 0.000000 4.959089 5.096002e-02 0.000000 0.000000 2.111634e-03 0.000487 0.003162 17.984117 1.681137e-02 0.000287 3.287117e-02 0.001309 0.001702 0.000093 0.005981 0.000027 0.004139 0.007258 0.000442 0.000000 0.122049 0.000000e+00 0.028040 1.376963 0.109314 2.201071e+00 0.646953 0.427123 0.055488 37.156633 24.666195 0.000416 0.452249 0.423161 1.855032 9.630443 16.673592 2.321445e-03 5.822343e-03 0.264946 0.000000 0.001616 0.000000 0.067036 1.468721e-03 0.000867 0.000000 0.000000 0.051276 3.280251e-02 1.379145e-02 0.026767 0.000000 0.315969 0.634852 0.004309 0.343613 0.302088 2.262782e+01 4.408535 0.013666 1.185906 1.818876 1.082149e+00 0.031302 5.695562e-03 3.008238 1.286605 0.064267 0.004062 1.028946e+00 5.242426e-02 2.020501 0.595951 1.282575 0.059749 8.487325e-01

Using exactly the example in the docs for df.corr():
def histogram_intersection(a, b):
v = np.minimum(a, b).sum().round(decimals=1)
return v
df = pd.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)],
columns=['dogs', 'cats'])
df.corr(method=histogram_intersection)
# output:
dogs cats
dogs 1.0 0.3
cats 0.3 1.0
So just pass your function in as the method parameter:
df.corr(method=calc_func)

Efficient way to create column referencing its own previous value

I am trying to generate a few columns in a dataframe with datetime index based on a rule which references their own previous values. I have tried a for loop on the length of df as per below but looking for a cleaner solution if possible?
Because what I want to do in the end is get the stats of generated columns (C,D,E in below example) over a large number of A,B....
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(30, 2), columns=list('AB'))
reset_level = 0.5
df['diff'] = df['A'].diff()
df['C'], df['D'], df['E'] = [0.0, 0.0, 0.0]
for i in range(1,len(df)):
if abs(df.iloc[i-1]['C'] + df.iloc[i]['diff']) > (reset_level):
df.iat[i,3] = 0.000
df.iat[i,4] = (df.iloc[i-1]['C'] + df.iloc[i]['diff'])
else:
df.iat[i,3] = (df.iloc[i-1]['C'] + df.iloc[i]['diff'])
df.iat[i,4] = 0.000
df.iat[i,5] = 0.5 * df.iloc[i]['D'] * df.iloc[i]['D']
Edit : Adding expected output below
A B diff C D E
0 -0.352725 1.429037 NaN 0.000000 0.000000 0.000000
1 -1.024418 -0.644302 -0.671693 0.000000 -0.671693 0.225585
2 0.401065 0.419555 1.425483 0.000000 1.425483 1.016001
3 -1.302484 0.724320 -1.703549 0.000000 -1.703549 1.451039
4 0.427035 0.835221 1.729518 0.000000 1.729518 1.495617
5 0.158694 -0.416741 -0.268340 -0.268340 0.000000 0.000000
6 0.921985 -0.490635 0.763291 0.494951 0.000000 0.000000
7 -0.835297 -1.036580 -1.757282 0.000000 -1.262331 0.796740
8 0.752060 -0.279206 1.587356 0.000000 1.587356 1.259850
9 1.795306 -1.554886 1.043246 0.000000 1.043246 0.544181
10 -0.405100 -0.361454 -2.200406 0.000000 -2.200406 2.420893
11 -0.253629 -0.627245 0.151471 0.151471 0.000000 0.000000
12 -0.820573 -0.212886 -0.566944 -0.415473 0.000000 0.000000
13 0.473439 2.532487 1.294012 0.000000 0.878539 0.385916
14 -1.395435 1.016338 -1.868875 0.000000 -1.868875 1.746346
15 -0.244269 -0.337820 1.151166 0.000000 1.151166 0.662592
16 -2.084977 -1.262249 -1.840708 0.000000 -1.840708 1.694103
17 0.666323 -1.696245 2.751300 0.000000 2.751300 3.784825
18 0.235207 -0.513903 -0.431115 -0.431115 0.000000 0.000000
19 1.386456 -0.149153 1.151249 0.000000 0.720134 0.259296
20 0.093456 -0.298154 -1.293000 0.000000 -1.293000 0.835925
21 0.690499 -1.687416 0.597043 0.000000 0.597043 0.178230
22 1.287530 -1.390260 0.597031 0.000000 0.597031 0.178223
23 1.828138 -0.288829 0.540608 0.000000 0.540608 0.146128
24 0.209666 -0.903385 -1.618472 0.000000 -1.618472 1.309727
25 -1.010678 0.615569 -1.220344 0.000000 -1.220344 0.744619
26 -1.799800 1.536332 -0.789122 0.000000 -0.789122 0.311357
27 0.611096 -1.033066 2.410896 0.000000 2.410896 2.906209
28 -0.532675 -0.091541 -1.143770 0.000000 -1.143770 0.654105
29 2.468137 -1.046117 3.000811 0.000000 3.000811 4.502435

I converted your for loop using a numpy array to hold the conditions and then np.where to replace the values according to your condition:
Define condition array
condition = np.abs(df.C.shift() + df["diff"]) > reset_level
Replace the values according to condition
df.iloc[:, 3] = np.where(condition, np.zeros((df.shape[0])), (df['C'].shift() + df['diff']))
df.iloc[:, 4] = np.where(~condition, np.zeros((df.shape[0])), (df['C'].shift() + df['diff']))
df.iloc[:, 5] = 0.5 * df['D'] * df['D']
Output:
A B diff C D E
0 -0.432513 -0.259526 NaN NaN 0.000000 0.000000
1 -1.120872 -1.572850 -0.688360 0.000000 NaN NaN
2 -0.917555 -2.251316 0.203317 0.203317 0.000000 0.000000
3 -1.869781 -1.284524 -0.952225 0.000000 -0.748908 0.280432
4 -2.041950 -0.091837 -0.172169 -0.172169 0.000000 0.000000
5 -0.142499 0.207746 1.899451 0.000000 1.727282 1.491751
6 1.432833 0.085211 1.575332 0.000000 1.575332 1.240835
7 -2.500191 -0.009907 -3.933025 0.000000 -3.933025 7.734341
8 0.154460 -1.859954 2.654651 0.000000 2.654651 3.523587
9 -0.565057 -0.516736 -0.719517 0.000000 -0.719517 0.258853
10 0.329845 0.127978 0.894902 0.000000 0.894902 0.400425
11 -0.920558 1.254617 -1.250402 0.000000 -1.250402 0.781753
12 -1.396913 0.262378 -0.476355 -0.476355 0.000000 0.000000
13 0.117336 -0.439932 1.514249 0.000000 1.037894 0.538612
14 -0.227066 2.565831 -0.344402 -0.344402 0.000000 0.000000
15 0.077750 0.195277 0.304816 0.304816 0.000000 0.000000
16 1.470611 -0.357213 1.392861 0.000000 1.697677 1.441053
17 -0.553844 0.339270 -2.024455 0.000000 -2.024455 2.049209
18 -0.259603 0.212839 0.294242 0.294242 0.000000 0.000000
19 0.605961 0.279599 0.865564 0.000000 1.159805 0.672574
20 -0.326706 -0.774350 -0.932667 0.000000 -0.932667 0.434934
21 -0.927601 -2.360751 -0.600895 0.000000 -0.600895 0.180537
22 -0.372085 0.986228 0.555516 0.000000 0.555516 0.154299
23 -0.687731 -2.966817 -0.315647 -0.315647 0.000000 0.000000
24 -0.041028 -0.328898 0.646703 0.000000 0.331057 0.054799
25 0.099489 0.275983 0.140517 0.140517 0.000000 0.000000
26 0.468274 -0.287097 0.368785 0.368785 0.000000 0.000000
27 0.497417 -0.588481 0.029143 0.029143 0.000000 0.000000
28 0.603178 2.243163 0.105761 0.105761 0.000000 0.000000
29 -0.643283 -1.051491 -1.246461 0.000000 -1.140700 0.650598
Is this what you were looking for, you didn't provide expected output.
Documentation:
np.where

Try this one (but don't iterate over all rows - it will do the whole column at once for you):
df["C_prev"] = df["C"].shift(1)

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(30, 2), columns=list('AB'))
reset_level = 0.5
df['diff'] = df['A'].diff()
df['C'], df['D'], df['E'] = [0.0, 0.0, 0.0]
Then apply a function to each row:
def f(row):
if abs(df.loc[row.name - 1, 'C'] + row['diff']) > reset_level:
C = 0.0
D = df.loc[row.name - 1, 'C'] + row['diff']
else:
C = df.loc[row.name - 1, 'C'] + row['diff']
D = 0.0
E = 0.5 * row['D'] * row['D']
return(pd.Series([C, D, E]))
df.loc[1:, ['C', 'D', 'E']] = df[1:].apply(f, axis=1)

TfIDf Vectorizer weights

Hi I have a lemmatized text in the format as shown by lemma. I want to get TfIdf score for each word this is the function that I wrote:
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
lemma=["'Ah", 'yes', u'say', 'softly', 'Harry',
'Potter', 'Our', 'new', 'celebrity', 'You',
'learn', 'subtle', 'science', 'exact', 'art',
'potion-making', u'begin', 'He', u'speak', 'barely',
'whisper', 'caught', 'every', 'word', 'like',
'Professor', 'McGonagall', 'Snape', 'gift',
u'keep', 'class', 'silent', 'without', 'effort',
'As', 'little', 'foolish', 'wand-waving', 'many',
'hardly', 'believe', 'magic', 'I', 'dont', 'expect', 'really',
'understand', 'beauty']
def Tfidf_Vectorize(lemmas_name):
vect = TfidfVectorizer(stop_words='english',ngram_range=(1,2))
vect_transform = vect.fit_transform(lemmas_name)
# First approach of creating a dataframe of weight & feature names
vect_score = np.asarray(vect_transform.mean(axis=0)).ravel().tolist()
vect_array = pd.DataFrame({'term': vect.get_feature_names(), 'weight': vect_score})
vect_array.sort_values(by='weight',ascending=False,inplace=True)
# Second approach of getting the feature names
vect_fn = np.array(vect.get_feature_names())
sorted_tfidf_index = vect_transform.max(0).toarray()[0].argsort()
print('Largest Tfidf:\n{}\n'.format(vect_fn[sorted_tfidf_index[:-11:-1]]))
return vect_array
tf_dataframe=Tfidf_Vectorize(lemma)
print(tf_dataframe.iloc[:5,:])
The output I am getting by:
print('Largest Tfidf:\n{}\n'.format(vect_fn[sorted_tfidf_index[:-11:-1]]))
is
Largest Tfidf:
[u'yes' u'fools' u'fury' u'gale' u'ghosts' u'gift' u'glory' u'glow' u'good'
u'granger']
The result of tf_dataframe
term weight
261 snape 0.027875
238 say 0.022648
211 potter 0.013937
181 mind 0.010453
123 harry 0.010453
60 dark 0.006969
75 dumbledore 0.006969
311 voice 0.005226
125 head 0.005226
231 ron 0.005226
Shouldn't both approaches lead to the same result of top features? I just want to calculate the tfidf scores and get the top 5 features/weight. What am i doing wrong?

I am not sure what I am looking at here but I have the feeling that you're using TfidfVectorizer incorrectly. However, please correct me in case I got the wrong idea of what you're trying.
So.. what you need is a list of documents which you feed to fit_transform(). From that you can construct a matrix where, for example, each column represents a document and each row a word. One cell in that matrix is the tf-idf score of the word i in document j.
Here's an example:
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"This is a document.",
"This is another document with slightly more text.",
"Whereas this is yet another document with even more text than the other ones.",
"This document is awesome and also rather long.",
"The car he drove was red."
]
document_names = ['Doc {:d}'.format(i) for i in range(len(documents))]
def get_tfidf(docs, ngram_range=(1,1), index=None):
vect = TfidfVectorizer(stop_words='english', ngram_range=ngram_range)
tfidf = vect.fit_transform(documents).todense()
return pd.DataFrame(tfidf, columns=vect.get_feature_names(), index=index).T
print(get_tfidf(documents, ngram_range=(1,2), index=document_names))
Which will give you:
Doc 0 Doc 1 Doc 2 Doc 3 Doc 4
awesome 0.0 0.000000 0.000000 0.481270 0.000000
awesome long 0.0 0.000000 0.000000 0.481270 0.000000
car 0.0 0.000000 0.000000 0.000000 0.447214
car drove 0.0 0.000000 0.000000 0.000000 0.447214
document 1.0 0.282814 0.282814 0.271139 0.000000
document awesome 0.0 0.000000 0.000000 0.481270 0.000000
document slightly 0.0 0.501992 0.000000 0.000000 0.000000
document text 0.0 0.000000 0.501992 0.000000 0.000000
drove 0.0 0.000000 0.000000 0.000000 0.447214
drove red 0.0 0.000000 0.000000 0.000000 0.447214
long 0.0 0.000000 0.000000 0.481270 0.000000
ones 0.0 0.000000 0.501992 0.000000 0.000000
red 0.0 0.000000 0.000000 0.000000 0.447214
slightly 0.0 0.501992 0.000000 0.000000 0.000000
slightly text 0.0 0.501992 0.000000 0.000000 0.000000
text 0.0 0.405004 0.405004 0.000000 0.000000
text ones 0.0 0.000000 0.501992 0.000000 0.000000
The two methods you show to get to words and their respective scores calculate the mean over all documents and fetch the max score of each word respectively.
So let's do this and compare the two methods:
df = get_tfidf(documents, ngram_range=(1,2), index=index)
print(pd.DataFrame([df.mean(1), df.max(1)], index=['score_mean', 'score_max']).T)
We can see that the scores are of course different.
score_mean score_max
awesome 0.096254 0.481270
awesome long 0.096254 0.481270
car 0.089443 0.447214
car drove 0.089443 0.447214
document 0.367353 1.000000
document awesome 0.096254 0.481270
document slightly 0.100398 0.501992
document text 0.100398 0.501992
drove 0.089443 0.447214
drove red 0.089443 0.447214
long 0.096254 0.481270
ones 0.100398 0.501992
red 0.089443 0.447214
slightly 0.100398 0.501992
slightly text 0.100398 0.501992
text 0.162002 0.405004
text ones 0.100398 0.501992
Note:
You can convince yourself that this does the same as calling min/max on the TfidfVectorizer:
vect = TfidfVectorizer(stop_words='english', ngram_range=(1,2))
tfidf = vect.fit_transform(documents)
print(tfidf.max(0))
print(tfidf.mean(0))

Writing XY coordinates from an ASCII file with no feature part ID using Python

I need to read an ASCII file containing X and Y coordinates as well a Z value using Python. These will be written as features in a feature class in ArcMap. Each point makes up a polygon where each feature is seperated by a row containing '999.0 999.0 999.0' as shown in the example. I'm wondering what the best way is to seperate each feature as there is no feature ID column.
329462.713287 8981177.910780 0.000000
331660.441771 8981187.405700 0.000000
331669.945462 8978975.695090 0.000000
329472.340912 8978966.180280 0.000000
329462.713287 8981177.910780 0.000000
999.0 999.0 999.0
297517.590475 8981318.596530 0.000000
299715.649732 8981329.876880 0.000000
299726.953175 8979117.630860 0.000000
297529.017922 8979106.326860 0.000000
297517.590475 8981318.596530 0.000000
999.0 999.0 999.0

Simply iterate the data line by line, and check whether the line contains your magic triplet and when you catch that line increase the feature index.

Is there any nipype interface for avscale (FSL script)?

I am trying to use nipype to analyze transformation matrixes that were created by FSL.
FSL has a script called "avscale" that analyzes those transformation matrixes (*.mat files).
I was wondering whether nipype has any interface that wrap that script and enable to work with its output.
Thanks

Based on the docs and the current source the answer is no. Also, avscale has also not been mentioned on the nipy-devel mailing list since at least last February. It's possible that Nipype already wraps something else that does this (perhaps with a matlab wrapper?) You could try opening an issue or asking the the mailing list.
As long as you're trying to use Python (with nipype and all), maybe the philosophy of the nipype project is that you should just use numpy/scipy for this? Just a guess, I don't know the functions to replicate this output with those tools. It's also possible that no one has gotten around to adding it yet.
For the uninitiated, avscale takes this affine matrix:
1.00614 -8.39414e-06 0 -0.757356
0 1.00511 -0.00317841 -0.412038
0 0.0019063 1.00735 -0.953364
0 0 0 1
and yields this or similar output:
Rotation & Translation Matrix:
1.000000 0.000000 0.000000 -0.757356
0.000000 0.999998 -0.001897 -0.412038
0.000000 0.001897 0.999998 -0.953364
0.000000 0.000000 0.000000 1.000000
Scales (x,y,z) = 1.006140 1.005112 1.007354
Skews (xy,xz,yz) = -0.000008 0.000000 -0.001259
Average scaling = 1.0062
Determinant = 1.01872
Left-Right orientation: preserved
Forward half transform =
1.003065 -0.000004 -0.000000 -0.378099
0.000000 1.002552 -0.001583 -0.206133
0.000000 0.000951 1.003669 -0.475711
0.000000 0.000000 0.000000 1.000000
Backward half transform =
0.996944 0.000004 0.000000 0.376944
0.000000 0.997452 0.001575 0.206357
0.000000 -0.000944 0.996343 0.473777
0.000000 0.000000 0.000000 1.000000