python dask dataframe splitting column of tuples into two columns - python
I am using python 2.7 with dask
I have a dataframe with one column of tuples that I created like this:
table[col] = table.apply(lambda x: (x[col1],x[col2]), axis = 1, meta = pd.Dataframe)
I want to re convert this tuple column into two seperate columns
In pandas I would do it like this:
table[[col1,col2]] = table[col].apply(pd.Series)
The point of doing so, is that dask dataframe does not support multi index and i want to use groupby according to multiple columns, and wish to create a column of tuples that will give me a single index containing all the values I need (please ignore efficiency vs multi index, for there is not yet a full support for this is dask dataframe)
When i try to unpack the tuple columns with dask using this code:
rxTable[["a","b"]] = rxTable["tup"].apply(lambda x: s(x), meta = pd.DataFrame, axis = 1)
I get this error
AttributeError: 'Series' object has no attribute 'columns'
when I try
rxTable[["a","b"]] = rxTable["tup"].apply(dd.Series, axis = 1, meta = pd.DataFrame)
I get the same
How can i take a column of tuples and convert it to two columns like I do in Pandas with no problem?
Thanks
Best i found so for in converting into pandas dataframe and then convert the column, then go back to dask
df1 = df.compute()
df1[["a","b"]] = df1["c"].apply(pd.Series)
df = dd.from_pandas(df1,npartitions=1)
This will work well, if the df is too big for memory, you can either:
1.compute only the wanted column, convert it into two columns and then use merge to get the split results into the original df
2.split the df into chunks, then converting each chunk and adding it into an hd5 file, then using dask to read the entire hd5 file into the dask dataframe
I found this methodology works well and avoids converting the Dask DataFrame to Pandas:
df['a'] = df['tup'].str.partition(sep)[0]
df['b'] = df['tup'].str.partition(sep)[2]
where sep is whatever delimiter you were using in the column to separate the two elements.
Related
Performing a function on dataframe rows
I am trying to winsorize a data set that would contain a few hundred columns of data. I'd like to make a new column to the dataframe and the column would contain the winsorized result from its row's data. How can I do this with a pandas dataframe without having to specify each column (I'd like to use all columns)? Edit: I would want to use the function 'winsorize(list, limits = [0.1,0.1])' but I'm not sure how to format the dataframe rows to work as a list.
Some tips: You may use the pandas function apply with axis=1 to apply a function to every row. The apply function will receive a pandas Series object but you can easily convert it to a list using tolist method For example: df.apply(lambda x: winsorize(x.tolist(), limits=[0.1,0.1]), axis=1)
You can use the numpy version of your dataframe using to_numpy() from scipy.stats.mstats import winsorize ma = winsorize(df.to_numpy(), axis=1, limits=[0.1, 0.1]) out = pd.DataFrame(ma.data, index=df.index, columns=df.columns)
python pandas difference between df_train["x"] and df_train[["x"]]
I have the following dataset and reading it from csv file.
x =[1,2,3,4,5]
with the pandas i can access the array
df_train = pd.read_csv("train.csv")
x = df_train["x"]
And
x = df_train[["x"]]
I could wonder since both producing the same result the former one could make sense but later one not. PLEASE, COULD YOU explain the difference and use?
In pandas, you can slice your data frame in different ways. On a high level, you can choose to select a single column out of a data frame, or many columns. When you select many columns, you have to slice using a list, and the return is a pandas DataFrame. For example df[['col1', 'col2', 'col3']] # returns a data frame When you select only one column, you can pass only the column name, and the return is just a pandas Series df['col1'] # returns a series When you do df[['col1']], you return a DataFrame with only one column. In other words, it's like your telling pandas "give me all the columns from the following list:" and just give it a list with one column on it. It will filter your df, returning all columns in your list (in this case, a data frame with only 1 column) If you want more details on the difference between a Series and a one-column DataFrame, check this thread with very good answers
Merge multiple int columns/rows into one numpy array (pandas dataframe)
I have a pandas dataframe with few columns and rows. I want to merge the columns into one and then merge the rows based on id and date into one. Currently I am doing so by: df['matrix'] = df[[col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,col15,col16,col17,col18,col19,col20,col21,col22,col23,col24,col25,col26,col27,col28,col29,col30,col31,col32,col33,col34,col35,col36,col37,col38,col39,col40,col41,col42,col43,col44,col45,col46,col47,col48]].values.tolist() df = df.groupby(['id','date'])['matrix'].apply(list).reset_index(name='matrix') This gives me the matrix in form of a list. Later I convert it into numpy.ndarray using: df['matrix'] = df['matrix'].apply(np.array) This is a small segment of my dataset for reference: id,date,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,col15,col16,col17,col18,col19,col20,col21,col22,col23,col24,col25,col26,col27,col28,col29,col30,col31,col32,col33,col34,col35,col36,col37,col38,col39,col40,col41,col42,col43,col44,col45,col46,col47,col48 16,2014-06-22,0,0,0,10,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 16,2014-06-22,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 16,2014-06-22,2,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 16,2014-06-22,3,0,0,0,0,0,0,0,0,0,0,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0 16,2014-06-22,4,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,22,0,0,0,0 Though the above piece of code works fine for small datasets, but sometimes crashes for larger ones. Specifically df['matrix'].apply(np.array) statement. Is there a way by which I can perform the merging to fetch me a numpy.array? This would save a lot of time.
No need to merge the columns at first. Split DataFrame using groupby and then flatten the result matrix=df.set_index(['id','date']).groupby(['id','date']).apply(lambda x: x.values.flatten())
Why recast a pandas groupby object as a dataframe to write to excel?
If I read a csv file into a pandas dataframe, followed by using a groupby (pd.groupby([column1,...])), why is that I cannot call a to_excel attribute on the new grouped object.
import pandas as pd
data = pd.read_csv("some file.csv")
data2 = data.groupby(['column1', 'column2'])
data2.to_excel("some file.xlsx") #spits out an error about series lacking the attribute 'to_excel'
data3 = pd.DataFrame(data=data2)
data3.to_excel("some file.xlsx") #works just perfectly!
Can someone explain why pandas needs to go through the whole process of converting from a dataframe to a series to group the rows?
I believe I was unclear in my question.
Re-framed question: Why does pandas convert the dataframe into a different kind of object (groupby object) when you use pd.groupby()? Clearly, you can cast this object as a dataframe, where the grouped columns become the (multi-level) indices.
Why not do this by default (without the user having to manually cast it as a dataframe)?
To answer your reframed question about why groupby gives you a groupby object and not a DataFrame: it does this for efficiency. The groupby object doesn't duplicate all the info about the original data; it essentially stores indices into the original DataFrame, indicating which group each row is in. This allows you to use a single groupby object for multiple aggregating group operations, each of which may use different columns, (e.g., you can do g = df.groupby('Blah') and then separately do g.SomeColumn.sum() and g.OtherColumn.mean()).
In short, the main point of groupby is to let you do aggregating computations on the groups. Simply pivoting the values of a single column out to an index level isn't what most people do with groupby. If you want to do that, you have to do it yourself.
Adding DataFrame columns in Python pandas
I have pandas DataFrame that has a number of columns (about 20) containing string objects. I'm looking for a simple method to add all of the columns together into one new column, but have so far been unsuccessful e.g.: for i in df.columns: df[‘newcolumn’] = df[‘newcolumn’] + ‘/‘ + df.ix[:,i] This results in an empty DataFrame column ‘newcolumn’ instead of the concatenated column. I’m new to pandas, so any help would be much appreciated.
df['newcolumn'] = df.apply(''.join, axis=1)