Related
I need the index to start at 1 rather than 0 when writing a Pandas DataFrame to CSV.
Here's an example:
In [1]: import pandas as pd
In [2]: result = pd.DataFrame({'Count': [83, 19, 20]})
In [3]: result.to_csv('result.csv', index_label='Event_id')
Which produces the following output:
In [4]: !cat result.csv
Event_id,Count
0,83
1,19
2,20
But my desired output is this:
In [5]: !cat result2.csv
Event_id,Count
1,83
2,19
3,20
I realize that this could be done by adding a sequence of integers shifted by 1 as a column to my data frame, but I'm new to Pandas and I'm wondering if a cleaner way exists.
Index is an object, and default index starts from 0:
>>> result.index
Int64Index([0, 1, 2], dtype=int64)
You can shift this index by 1 with
>>> result.index += 1
>>> result.index
Int64Index([1, 2, 3], dtype=int64)
Just set the index before writing to CSV.
df.index = np.arange(1, len(df) + 1)
And then write it normally.
source: In Python pandas, start row index from 1 instead of zero without creating additional column
Working example:
import pandas as pdas
dframe = pdas.read_csv(open(input_file))
dframe.index = dframe.index + 1
Another way in one line:
df.shift()[1:]
In my opinion best practice is to set the index with a RangeIndex
import pandas as pd
result = pd.DataFrame(
{'Count': [83, 19, 20]},
index=pd.RangeIndex(start=1, stop=4, name='index')
)
>>> result
Count
index
1 83
2 19
3 20
I prefer this, because you can define the range and a possible step and a name for the index in one line.
This worked for me
df.index = np.arange(1, len(df)+1)
You can use this one:
import pandas as pd
result = pd.DataFrame({'Count': [83, 19, 20]})
result.index += 1
print(result)
or this one, by getting the help of numpy library like this:
import pandas as pd
import numpy as np
result = pd.DataFrame({'Count': [83, 19, 20]})
result.index = np.arange(1, len(result)+1)
print(result)
np.arange will create a numpy array and return values within a given interval which is (1, len(result)+1) and finally you will assign that array to result.index.
use this
df.index = np.arange(1, len(df)+1)
Add ".shift()[1:]" while creating a data frame
data = pd.read_csv(r"C:\Users\user\path\data.csv").shift()[1:]
Fork from the original answer, giving some cents:
if I'm not mistaken, starting from version 0.23, index object is RangeIndex type
From the official doc:
RangeIndex is a memory-saving special case of Int64Index limited to representing monotonic ranges. Using RangeIndex may in some instances improve computing speed.
In case of a huge index range, that makes sense, using the representation of the index, instead of defining the whole index at once (saving memory).
Therefore, an example (using Series, but it applies to DataFrame also):
>>> import pandas as pd
>>>
>>> countries = ['China', 'India', 'USA']
>>> ds = pd.Series(countries)
>>>
>>>
>>> type(ds.index)
<class 'pandas.core.indexes.range.RangeIndex'>
>>> ds.index
RangeIndex(start=0, stop=3, step=1)
>>>
>>> ds.index += 1
>>>
>>> ds.index
RangeIndex(start=1, stop=4, step=1)
>>>
>>> ds
1 China
2 India
3 USA
dtype: object
>>>
As you can see, the increment of the index object, changes the start and stop parameters.
This adds a column that accomplishes what you want
df.insert(0,"Column Name", np.arange(1,len(df)+1))
Following on from TomAugspurger's answer, we could use list comprehension rather than np.arrange(), which removes the requirement for importing the module: numpy. You can use the following instead:
df.index = [i+1 for i in range(len(df))]
I am trying to apply a Bar Style to all of the data in the dataframe, except the last row, which is supposed to be the Total row.
import pandas as pd
import numpy as np
data = pd.DataFrame(np.random.randn(5, 2), columns=list('AB'))
data.loc['Total'] = data.sum()
A B
0 -1.224620 -0.373898
1 0.75568 0.997875
2 -1.284663 -0.211903
3 -0.274813 -0.871816
4 1.256267 -0.742521
Total -0.772143 -1.202263
It was explained in the docs, that
A tuple is treated as (row_indexer, column_indexer)
You just need to twist a bit the subset option.
On your data
import pandas as pd
import numpy as np
data = pd.DataFrame(np.random.randn(5, 2), columns=list('AB'))
data.loc['Total'] = data.sum()
data.style.bar(subset = ([0,1,2,3,4], ['A', 'B']))
it gives
I have a Numpy array consisting of a list of lists, representing a two-dimensional array with row labels and column names as shown below:
data = array([['','Col1','Col2'],['Row1',1,2],['Row2',3,4]])
I'd like the resulting DataFrame to have Row1 and Row2 as index values, and Col1, Col2 as header values
I can specify the index as follows:
df = pd.DataFrame(data,index=data[:,0]),
however I am unsure how to best assign column headers.
You need to specify data, index and columns to DataFrame constructor, as in:
>>> pd.DataFrame(data=data[1:,1:], # values
... index=data[1:,0], # 1st column as index
... columns=data[0,1:]) # 1st row as the column names
edit: as in the #joris comment, you may need to change above to np.int_(data[1:,1:]) to have correct data type.
Here is an easy to understand solution
import numpy as np
import pandas as pd
# Creating a 2 dimensional numpy array
>>> data = np.array([[5.8, 2.8], [6.0, 2.2]])
>>> print(data)
>>> data
array([[5.8, 2.8],
[6. , 2.2]])
# Creating pandas dataframe from numpy array
>>> dataset = pd.DataFrame({'Column1': data[:, 0], 'Column2': data[:, 1]})
>>> print(dataset)
Column1 Column2
0 5.8 2.8
1 6.0 2.2
I agree with Joris; it seems like you should be doing this differently, like with numpy record arrays. Modifying "option 2" from this great answer, you could do it like this:
import pandas
import numpy
dtype = [('Col1','int32'), ('Col2','float32'), ('Col3','float32')]
values = numpy.zeros(20, dtype=dtype)
index = ['Row'+str(i) for i in range(1, len(values)+1)]
df = pandas.DataFrame(values, index=index)
This can be done simply by using from_records of pandas DataFrame
import numpy as np
import pandas as pd
# Creating a numpy array
x = np.arange(1,10,1).reshape(-1,1)
dataframe = pd.DataFrame.from_records(x)
>>import pandas as pd
>>import numpy as np
>>data.shape
(480,193)
>>type(data)
numpy.ndarray
>>df=pd.DataFrame(data=data[0:,0:],
... index=[i for i in range(data.shape[0])],
... columns=['f'+str(i) for i in range(data.shape[1])])
>>df.head()
[![array to dataframe][1]][1]
Here simple example to create pandas dataframe by using numpy array.
import numpy as np
import pandas as pd
# create an array
var1 = np.arange(start=1, stop=21, step=1).reshape(-1)
var2 = np.random.rand(20,1).reshape(-1)
print(var1.shape)
print(var2.shape)
dataset = pd.DataFrame()
dataset['col1'] = var1
dataset['col2'] = var2
dataset.head()
Adding to #behzad.nouri 's answer - we can create a helper routine to handle this common scenario:
def csvDf(dat,**kwargs):
from numpy import array
data = array(dat)
if data is None or len(data)==0 or len(data[0])==0:
return None
else:
return pd.DataFrame(data[1:,1:],index=data[1:,0],columns=data[0,1:],**kwargs)
Let's try it out:
data = [['','a','b','c'],['row1','row1cola','row1colb','row1colc'],
['row2','row2cola','row2colb','row2colc'],['row3','row3cola','row3colb','row3colc']]
csvDf(data)
In [61]: csvDf(data)
Out[61]:
a b c
row1 row1cola row1colb row1colc
row2 row2cola row2colb row2colc
row3 row3cola row3colb row3colc
I think this is a simple and intuitive method:
data = np.array([[0, 0], [0, 1] , [1, 0] , [1, 1]])
reward = np.array([1,0,1,0])
dataset = pd.DataFrame()
dataset['StateAttributes'] = data.tolist()
dataset['reward'] = reward.tolist()
dataset
returns:
But there are performance implications detailed here:
How to set the value of a pandas column as list
It's not so short, but maybe can help you.
Creating Array
import numpy as np
import pandas as pd
data = np.array([['col1', 'col2'], [4.8, 2.8], [7.0, 1.2]])
>>> data
array([['col1', 'col2'],
['4.8', '2.8'],
['7.0', '1.2']], dtype='<U4')
Creating data frame
df = pd.DataFrame(i for i in data).transpose()
df.drop(0, axis=1, inplace=True)
df.columns = data[0]
df
>>> df
col1 col2
0 4.8 7.0
1 2.8 1.2
I have a dataframe and am trying to get the closest matches using mahalanobis distance across three categories, like:
from io import StringIO
from sklearn import metrics
import pandas as pd
stringdata = StringIO(u"""pid,ratio1,pct1,rsp
0,2.9,26.7,95.073615
1,11.6,29.6,96.963660
2,0.7,37.9,97.750412
3,2.7,27.9,102.750412
4,1.2,19.9,93.750412
5,0.2,22.1,96.750412
""")
stats = ['ratio1','pct1','rsp']
df = pd.read_csv(stringdata)
d = metrics.pairwise.pairwise_distances(df[stats].as_matrix(),
metric='mahalanobis')
print(df)
print(d)
Where that pid column is a unique identifier.
What I need to do is take that ndarray returned by the pairwise_distances call and update the original dataframe so each row has some kind of list of its closest N matches (so pid 0 might have an ordered list by distance of like 2, 1, 5, 3, 4 (or whatever it actually is), but I'm totally stumped how this is done in python.
from io import StringIO
from sklearn import metrics
stringdata = StringIO(u"""pid,ratio1,pct1,rsp
0,2.9,26.7,95.073615
1,11.6,29.6,96.963660
2,0.7,37.9,97.750412
3,2.7,27.9,102.750412
4,1.2,19.9,93.750412
5,0.2,22.1,96.750412
""")
stats = ['ratio1','pct1','rsp']
df = pd.read_csv(stringdata)
dist = metrics.pairwise.pairwise_distances(df[stats].as_matrix(),
metric='mahalanobis')
dist = pd.DataFrame(dist)
ranks = np.argsort(dist, axis=1)
df["rankcol"] = ranks.apply(lambda row: ','.join(map(str, row)), axis=1)
df
I need the index to start at 1 rather than 0 when writing a Pandas DataFrame to CSV.
Here's an example:
In [1]: import pandas as pd
In [2]: result = pd.DataFrame({'Count': [83, 19, 20]})
In [3]: result.to_csv('result.csv', index_label='Event_id')
Which produces the following output:
In [4]: !cat result.csv
Event_id,Count
0,83
1,19
2,20
But my desired output is this:
In [5]: !cat result2.csv
Event_id,Count
1,83
2,19
3,20
I realize that this could be done by adding a sequence of integers shifted by 1 as a column to my data frame, but I'm new to Pandas and I'm wondering if a cleaner way exists.
Index is an object, and default index starts from 0:
>>> result.index
Int64Index([0, 1, 2], dtype=int64)
You can shift this index by 1 with
>>> result.index += 1
>>> result.index
Int64Index([1, 2, 3], dtype=int64)
Just set the index before writing to CSV.
df.index = np.arange(1, len(df) + 1)
And then write it normally.
source: In Python pandas, start row index from 1 instead of zero without creating additional column
Working example:
import pandas as pdas
dframe = pdas.read_csv(open(input_file))
dframe.index = dframe.index + 1
Another way in one line:
df.shift()[1:]
In my opinion best practice is to set the index with a RangeIndex
import pandas as pd
result = pd.DataFrame(
{'Count': [83, 19, 20]},
index=pd.RangeIndex(start=1, stop=4, name='index')
)
>>> result
Count
index
1 83
2 19
3 20
I prefer this, because you can define the range and a possible step and a name for the index in one line.
This worked for me
df.index = np.arange(1, len(df)+1)
You can use this one:
import pandas as pd
result = pd.DataFrame({'Count': [83, 19, 20]})
result.index += 1
print(result)
or this one, by getting the help of numpy library like this:
import pandas as pd
import numpy as np
result = pd.DataFrame({'Count': [83, 19, 20]})
result.index = np.arange(1, len(result)+1)
print(result)
np.arange will create a numpy array and return values within a given interval which is (1, len(result)+1) and finally you will assign that array to result.index.
use this
df.index = np.arange(1, len(df)+1)
Add ".shift()[1:]" while creating a data frame
data = pd.read_csv(r"C:\Users\user\path\data.csv").shift()[1:]
Fork from the original answer, giving some cents:
if I'm not mistaken, starting from version 0.23, index object is RangeIndex type
From the official doc:
RangeIndex is a memory-saving special case of Int64Index limited to representing monotonic ranges. Using RangeIndex may in some instances improve computing speed.
In case of a huge index range, that makes sense, using the representation of the index, instead of defining the whole index at once (saving memory).
Therefore, an example (using Series, but it applies to DataFrame also):
>>> import pandas as pd
>>>
>>> countries = ['China', 'India', 'USA']
>>> ds = pd.Series(countries)
>>>
>>>
>>> type(ds.index)
<class 'pandas.core.indexes.range.RangeIndex'>
>>> ds.index
RangeIndex(start=0, stop=3, step=1)
>>>
>>> ds.index += 1
>>>
>>> ds.index
RangeIndex(start=1, stop=4, step=1)
>>>
>>> ds
1 China
2 India
3 USA
dtype: object
>>>
As you can see, the increment of the index object, changes the start and stop parameters.
This adds a column that accomplishes what you want
df.insert(0,"Column Name", np.arange(1,len(df)+1))
Following on from TomAugspurger's answer, we could use list comprehension rather than np.arrange(), which removes the requirement for importing the module: numpy. You can use the following instead:
df.index = [i+1 for i in range(len(df))]