Develop Reference

Pandas get single warning message per line

I have some code that calls Pandas functions millions of times and many of these calls are issuing warning notices. This is a background process, so I wanted to convert them to logs using logging.captureWarnings(True). The logging works fine, but no matter what filter options I seem to set, the warnings are repeated, millions of times. I have set the logger to record each row to a database table, which works fine with thousands of log records, but is slowed down by millions of warning messages as logs.
So, my question, is there any way to make sure the number of warnings is one per call site, which is the default AFAIK. I think this bug may have something to do with it: https://github.com/python/cpython/issues/73858. I did look at the pandas source code but couldn't find an obvious place where the catch_warnings exception manager was being used.
Does anyone have any potential solutions to return the warnings emitted to what should be the default?
A simple test script is instructive:
import pandas as pd
import warnings
warnings.simplefilter(action='default', category=FutureWarning)
d = {'col1': [1, 2], 'col2': [3, 4], 'col3': [5, 6]}
df = pd.DataFrame(data=d)
a = ['col1','col2']
df[a]
b = set(a)
for i in range(3):
df[b]
Running it with python3 -Wd::FutureWarning test.py
Yields:
/home/stephen/test.py:16: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
df[b]
/home/stephen/test.py:16: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
df[b]
/home/stephen/test.py:16: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
df[b]
Not the behaviour I expected

constant pandas warning in pycharm console "FutureWarning: iteritems is deprecated

Why does this code give me a warning message?
import pandas as pd
import numpy as np
test = pd.DataFrame({'a':np.array([0,1,2]), 'b': np.array([3,4,5])})
It seems anything I do in pandas throws these long error messages, I'm not sure if this is a problem with pycharm or if I'm doing something wrong in pandas.

You can ignore the warning, it is a known bug and is being fixed.

Configuring python warnings control to show first occurrence only

I am trying to use the python warnings module to show only the first occurrence of a given warning. According to Warning control documentation, this looks to be as simple as setting the action flag to 'once' but that doesn't work for me.
Example code.
# Tried with python 3.9.13 for all the pandas versions below,
# and python 3.8.13 and 3.10.6 with pandas 1.5.0.
# Recent pandas versions 1.4.4 and 1.5.0 result in a deprecated warning
# for DataFrame.append, pandas version 1.3.4 does not.
import pandas as pd
import warnings
# Set action parameter. Various options include 'default', 'once' and 'ignore'
warnings.filterwarnings(action='default')
data = [1]
appendDF = pd.DataFrame(data)
for i in range(5):
print('Trial:', i)
appendDF = appendDF.append(data)
For action='default' and action='once', I get several copies of the same deprecated warning. For action='ignore', I get none (as desired).
I can see a potential workaround in How to print only first occurrence of python warning?, but that doesn't go into the 'once' option for the action parameter and also involves additional bespoke coding.
Any suggestions as to why the 'once' option above does not seem to work?

xarray firing a Future Warning

Just upgraded to python3.8, I have updated xarray to v.0.16, but now I am always getting this warning:
/usr/local/lib/python3.8/dist-packages/xarray/core/common.py:1123: FutureWarning: 'base' in .resample() and in Grouper() is deprecated.
The new arguments that you should use are 'offset' or 'origin'.
>>> df.resample(freq="3s", base=2)
becomes:
>>> df.resample(freq="3s", offset="2s")
grouper = pd.Grouper(
The only point in my script in which I am using .resample is this:
mydata = xr.open_dataset(ncfile).resample(time='3H').reduce(np.mean)
but I don't know how to change it to avoid the warning.

Until this is updated in xarray, warnings can be ignored with a call to warnings.filterwarnings.
You're welcome to open an issue on GitHub for xarray (or even a PR!)

code is working but with a 'DeprecationWarning' -scipy.stats

I wrote this code to calculate the mode and standard deviation for a large sample:
import numpy as np
import csv
import scipy.stats as sp
import math
r=open('stats.txt', 'w') #file with results
r.write('Data File'+'\t'+ 'Mode'+'\t'+'Std Dev'+'\n')
f=open('data.ls', 'rb') #file with the data files
for line in f:
dataf=line.strip()
data=csv.reader(open(dataf, 'rb'))
data.next()
data_list=[]
datacol=[]
data_list.extend(data)
for rows in data_list:
datacol.append(math.log10(float(rows[73])))
m=sp.mode(datacol)
s=sp.std(datacol)
r.write(dataf+'\t'+str(m)+'\t'+str(s)+'\n')
del(datacol)
del(data_list)
Which is working well -I think! However after I run the code there is an error message on my terminal and I am wondering if anybody can tell me what it means?
/usr/lib/python2.6/dist-packages/scipy/stats/stats.py:1328: DeprecationWarning: scipy.stats.std is deprecated; please update your code to use numpy.std.
Please note that:
- numpy.std axis argument defaults to None, not 0
- numpy.std has a ddof argument to replace bias in a more general manner.
scipy.stats.std(a, bias=True) can be replaced by numpy.std(x,
axis=0, ddof=0), scipy.stats.std(a, bias=False) by numpy.std(x, axis=0,
ddof=1).
ddof=1).""", DeprecationWarning)
/usr/lib/python2.6/dist-packages/scipy/stats/stats.py:1304: DeprecationWarning: scipy.stats.var is deprecated; please update your code to use numpy.var.
Please note that:
- numpy.var axis argument defaults to None, not 0
- numpy.var has a ddof argument to replace bias in a more general manner.
scipy.stats.var(a, bias=True) can be replaced by numpy.var(x,
axis=0, ddof=0), scipy.stats.var(a, bias=False) by var(x, axis=0,
ddof=1).
ddof=1).""", DeprecationWarning)
/usr/lib/python2.6/dist-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
- numpy.mean axis argument defaults to None, not 0
- numpy.mean has a ddof argument to replace bias in a more general manner.
scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
axis=0, ddof=1).""", DeprecationWarning)

Those are deprecation warnings, which usually mean that your code will work, but may stop working in a future release.
Currently you have this line s=sp.std(datacol). It looks like the warning suggests using numpy.std() instead of scipy.stats.std() Making this change could make this warning go away.
If you don't care about the deprecation warning and want to use your code as is, you can suppress it with the warnings module. For example, if you have a function fxn() that generates a DeprecationWarning, you can wrap it like this:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
fxn() #this function generates DeprecationWarnings

The DeprecationWarnings don't prevent your code to run properly, they are just warnings that the code you are using will soon be deprecated and that you should update it to the proper syntax.
In this particular case, it stems from inconstencies between NumPy and SciPy on the default arguments for the var, std... functions/methods. In order to clean things up, it was decided to drop the functions from scipy.stats and use their NumPy counterparts instead.
Of course, just dropping the functions would upset some users whose code would suddenly fail to work. So, the SciPy devs decided to include a DeprecationWarning for a couple of releases, which should leave enough time for everybody to update their code.
In your case, you should use check the docstring of scipy.stats.std on your system to see what defaults they're using, and follow the warning instructions on how to modify your code accordingly.