Develop Reference

Pandas – ValueError: Cannot setitem on a Categorical with a new category, set the categories first

I've been searching for a solution to this for the past few hours now. Relevant pandas documentation is unhelpful and this solution gives me the same error.
I am trying to order my dataframe using a categorical in the following manner:
metabolites_order = CategoricalDtype(['Header', 'Metabolite', 'Unknown'], ordered=True)
df2['Feature type'] = df2['Feature type'].astype(metabolites_order)
df2 = df2.sort_values('Feature type')
The "Feature type" column is populated with the categories correctly. This code runs perfectly in Jupyter Notebooks, but when I run it in Pycharm, I get the following error:
Traceback (most recent call last):
File "/Users/wasim.sandhu/Documents/MSDIALPostProcessor/postprocessor.py", line 138, in process_alignment_file
df2.loc[4] = list(df2.columns)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/indexing.py", line 692, in __setitem__
iloc._setitem_with_indexer(indexer, value, self.name)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/indexing.py", line 1635, in _setitem_with_indexer
self._setitem_with_indexer_split_path(indexer, value, name)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/indexing.py", line 1700, in _setitem_with_indexer_split_path
self._setitem_single_column(loc, v, pi)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/indexing.py", line 1813, in _setitem_single_column
ser._mgr = ser._mgr.setitem(indexer=(pi,), value=value)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 568, in setitem
return self.apply("setitem", indexer=indexer, value=value)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 427, in apply
applied = getattr(b, f)(**kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 1846, in setitem
self.values[indexer] = value
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py", line 211, in __setitem__
value = self._validate_setitem_value(value)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/arrays/categorical.py", line 1898, in _validate_setitem_value
raise ValueError(
ValueError: Cannot setitem on a Categorical with a new category, set the categories first
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
What could be causing this? I believe that I've set the categories correctly...

I'd suggest just mapping these categories to integers then sorting on that column instead.
categories = ['Header', 'Metabolite', 'Unknown']
feature_map = {categories[i]:i for i in range(len(categories))}
df['Feature Order'] = df['Feature Type'].map(feature_map)
df.sort_values('Feature Order')

Figured it out literally minutes after I posted the question. The header column in this dataset is in the 5th row. I checked the "Feature type" column and "Feature type" is one of its values, which threw this error.
Solved by adding the column header name into the categories.
metabolites_order = CategoricalDtype(['Header', 'Feature type', 'Metabolite', 'Unknown'], ordered=True)

pandas.DataFrame.agg does not work with np.std?

I am trying to use the pandas.DataFrame.agg function on the first column of a dataframe with the agg function is numpy.std.
I dont know why it works with numpy.mean but not numpy.std?
Can someone tell me in what circumstance that happens.
This is very strange ##
The following describes what I am facing.
My source is like this:
print(type(dataframe))
print(dataframe.head(5))
first_col = dataframe.columns.values[0]
agg_df = dataframe.agg({first_col: [np.mean]})
print(agg_df)
then it shows the result like this
<class 'pandas.core.frame.DataFrame'>
ax
0 -98.06
1 -97.81
2 -96.00
3 -93.44
4 -92.94
ax
mean -98.06
now I change the function from np.mean into np.std (without changing anything else)
print(type(dataframe))
print(dataframe.head(5))
first_col = dataframe.columns.values[0]
agg_df = dataframe.agg({first_col: [np.std]})
print(agg_df)
it shows the errors
Traceback (most recent call last):
File "C:\prediction_framework_django\predictions\predictor.py", line 112, in pre_aggregated_unseen_data
agg_df = dataframe.agg({axis: [np.std]})
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\frame.py", line 7578, in aggregate
result, how = self._aggregate(func, axis, *args, **kwargs)
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\frame.py", line 7609, in _aggregate
return aggregate(self, arg, *args, **kwargs)
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\aggregation.py", line 582, in aggregate
return agg_dict_like(obj, arg, _axis), True
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\aggregation.py", line 768, in agg_dict_like
results = {key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()}
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\aggregation.py", line 768, in <dictcomp>
results = {key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()}
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\series.py", line 3974, in aggregate
result, how = aggregate(self, func, *args, **kwargs)
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\aggregation.py", line 586, in aggregate
return agg_list_like(obj, arg, _axis=_axis), None
File "C:\prediction_framework_django\env\lib\site-packages\pandas\core\aggregation.py", line 672, in agg_list_like
raise ValueError("no results")
ValueError: no results
So the error is
in agg_list_like raise ValueError("no results") ValueError: no results
Thank you for your time and help.

Simply use the pandas builtin:
# Note the use of string to denote the function here
df.agg({first_col: ['mean', 'std']})
# You can also simply use the following
df[first_col].mean()
df[first_col].std()
[EDIT]: The error that you are getting is probably resulting from mixed types. You can check that all dtypes are float by looking at df.dtypes. If you have one that is object, then convert the problematic ones (probably empty strings) into whatever you need and np.std and pandas' builtin std should work

KeyError: 'None of [['col label 1', 'col label 2']] are in the [columns]'

I am attempting to slice a pandas dataframe by column labels using .loc. Based on Pandas documentation, https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html, .loc seems like the right indexer for the use case.
Original pandas DataFrame and confirmation the columns w/ labels exists:
The column labels as dynamically constructed and passed as list to slice the dataframe.
# Create dictionaries
prop_dict = dict(zip(df_list.id, df_list.Company))
city_dict = dict(zip(df_list.id, df_list.city))
# Lookup keys (property ids) from prop_dict
propKeys = getKeysByValue(prop_dict, landlord)
cityKeys = getKeysByValue(city_dict, market)
prop_list = list(set(propKeys) & set(cityKeys))
print(prop_list)
[19, 27]
# Slice dataframe
df_temp = df_t.loc[:, prop_list]
However, this throws an error KeyError: 'None of [[19, 27]] are in the [columns]'
Full traceback here:
Traceback (most recent call last):
File "/Platform/Deploy/tabs/market.py", line 279, in render_table
result = top_leads(company, market)
File "/Platform/Deploy/return_leads.py", line 86, in top_leads
df_temp = df_matrix.loc[:, prop_list]
File "/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py", line 1472, in __getitem__
return self._getitem_tuple(key)
File "/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py", line 890, in _getitem_tuple
retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
File "/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py", line 1901, in _getitem_axis
return self._getitem_iterable(key, axis=axis)
File "/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py", line 1143, in _getitem_iterable
self._validate_read_indexer(key, indexer, axis)
File "/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py", line 1206, in _validate_read_indexer
key=key, axis=self.obj._get_axis_name(axis)))
KeyError: 'None of [[19, 27]] are in the [columns]'

Is it possible the columns '19' and '27' are located as the 19th and 27th column and that is why the first time it gives you the appropriate result because of the integer value of the 'names' 19 and 27. If you want to pass it as a list there need to be ''s around the names of the column, meaning it should be ['19','27'] instead of [19,27]

Can use dataframe ix for assignment, but not retrieval

I am looping through rows of a pandas df, loop index i.
I am able to assign several columns using the ix function with the loop index as first parameter, column name as second.
However, when I try to retrieve/print using this method,
print(df.ix[i,"Run"])
I get a the following Typerror: str object cannot be interpreted as an integer.
somehow related to Keyerror: 'Run'
Not quite sure why this is occurring, as Run is indeed a column in the dataframe.
Any suggestions?
Traceback (most recent call last):
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexes\base.py\!", line 3124, in get_value
return libindex.get_value_box(s, key)
File \!"pandas\_libs\index.pyx\!", line 55, in pandas._libs.index.get_value_box
File \!"pandas\_libs\index.pyx\!", line 63, in pandas._libs.index.get_value_box
TypeError: 'str' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File \!"C:\...", line 365, in <module>
print(df.ix[i,\!"Run\!"])
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexing.py\!", line 116, in __getitem__
return self._getitem_tuple(key)
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexing.py\!", line 870, in _getitem_tuple
return self._getitem_lowerdim(tup)
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexing.py\!", line 1027, in _getitem_lowerdim
return getattr(section, self.name)[new_key]
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexing.py\!", line 122, in __getitem__
return self._getitem_axis(key, axis=axis)
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexing.py\!", line 1116, in _getitem_axis
return self._get_label(key, axis=axis)
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexing.py\!", line 136, in _get_label
return self.obj[label]
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\series.py\!", line 767, in __getitem__
result = self.index.get_value(self, key)
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexes\base.py\!", line 3132, in get_value
raise e1
File \!"C:\WPy-3670\python-3.6.7.amd64\lib\site-packages\pandas\core\indexes\base.py\!", line 3118, in get_value
tz=getattr(series.dtype, 'tz', None))
File \!"pandas\_libs\index.pyx\!", line 106, in pandas._libs.index.IndexEngine.get_value
File \!"pandas\_libs\index.pyx\!", line 114, in pandas._libs.index.IndexEngine.get_value
File \!"pandas\_libs\index.pyx\!", line 162, in pandas._libs.index.IndexEngine.get_loc
File \!"pandas\_libs\hashtable_class_helper.pxi\!", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
File \!"pandas\_libs\hashtable_class_helper.pxi\!", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Run'
"
Upon changing the name of the column I print to any other column, it does work correctly. Earlier in the code, I "compressed" the rows, which had multiple rows per unique string in 'Run' column, using the following.
df=df.groupby('Run').max()
Did this last line somehow remove the column/column name from the table?

ix has been deprecated. ix has always been ambiguous: does ix[10] refer to the row with the label 10, or the row at position 10?
Use loc or iloc instead:
df.loc[i,"Run"] = ... # by label
df.iloc[i]["Run"] = ... # by position
As for the groupby removing Run: it moves Run to the index of the data frame. To get it back as a column, call reset_index:
df=df.groupby('Run').max().reset_index()
Differences between indexing by label and position:
Suppose you have a series like this:
s = pd.Series(['a', 'b', 'c', 'd', 'e'], index=np.arange(0,9,2))
0 a
2 b
4 c
6 d
8 e
The first column is the labels (aka the index). The second column is the values of the series.
Label based indexing:
s.loc[2] --> b
s.loc[3] --> error. The label doesn't exist
Position based indexing:
s.iloc[2] --> c. since `a` has position 0, `b` has position 1, and so on
s.iloc[3] --> d
According to the documentation, s.ix[3] would have returned d since it first searches for the label 3. When that fails, it falls back to the position 3. On my machine (Pandas 0.24.2), it returns an error, along with a deprecation warning, so I guess the developers changed it to behave like loc.
If you want to use mixed indexing, you have to be explicit about that:
s.loc[3] if 3 in s.index else s.iloc[3]

Pandas hierarchical indexing - not working for dataframe?

I'm having trouble addressing values in a DataFrame, but I don't seem to have any problems with the Series object.
>>> df=DataFrame([0.5,1.5,2.5,3.5,4.5], index=[['a','a','b','b','b'],[1,2,1,2,3]])
>>> series=Series([0.5,1.5,2.5,3.5,4.5], index=[['a','a','b','b','b'],[1,2,1,2,3]])
>>> series['a']
1 0.5
2 1.5
dtype: float64
>>> df['a']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Anaconda\lib\site-packages\pandas\core\frame.py", line 2003, in __getitem__
return self._get_item_cache(key)
File "C:\Anaconda\lib\site-packages\pandas\core\generic.py", line 667, in _get_item_cache
values = self._data.get(item)
File "C:\Anaconda\lib\site-packages\pandas\core\internals.py", line 1655, in get
_, block = self._find_block(item)
File "C:\Anaconda\lib\site-packages\pandas\core\internals.py", line 1935, in _find_block
self._check_have(item)
File "C:\Anaconda\lib\site-packages\pandas\core\internals.py", line 1942, in _check_have
raise KeyError('no item named %s' % com.pprint_thing(item))
KeyError: u'no item named a'
I'm definitely misunderstanding something, if someone could help me out it would be very much appreciated!

You are trying to select a column, and there is indeed no column named 'a'. Try df.loc['a'] instead.
I recommend to look at the basic indexing docs: http://pandas.pydata.org/pandas-docs/stable/indexing.html#basics
In summary:
series[label] selects element in series at index label
dataframe[label] selects column with name label