Develop Reference

Using Pylint with PyModule generated using PyO3 and maturin

Pylint will not recognize any of the functions from a PyModule that I created using PyO3 and maturin. All of the functions import and run fine in the python code base, but for some reason Pylint is throwing E1011: no-member warnings.
Below is a (likely) incomplete dummy example, but is provided in order to show the way I am decorating using pymodule and pyfunction:
#[pyfunction]
fn add_nums(
_py: Python<'_>,
a: f32,
b: f32,
) -> PyResult<f32> {
let res:f32 = a+b;
Ok(res)
}
#[pymodule]
fn my_module(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(add_nums, m)?)?;
Ok(())
}
Then if I build that using maturin build --release and install the module, from the resulting wheelfile, into my python environment and import into a script:
import my_module
my_module.add_nums(5, 6) # ignore that these are not f32 - irrelevant this is a dummy example
If I then run pylint on that file (from terminal - VS Code pylint extension actually does not complain about this...), I end up with something like: E1101: Module 'my_module' has no 'add-nums' member (no-member) even though the code (not this code - but the real code which I cannot include here) runs just fine.
Has anyone successfully built wheelfiles using maturin, used them in another project, then had Pylint play nicely with that project and recognize that the methods do actually exist?

Pylint has a extension-pkg-allow-list setting which you can use to inspect non-python modules. It will need to load the extension into pylint's interpreter though, which is why it's not enabled by default.
There's also requests to support (and lint) pyi, but AFAIK that's not supported yet, cf #2873 and #4987.
Before Pylint 2.8, the setting is extension-pkg-whitelist.

Similar to the answer by #Masklinn except it looks like the term 'extension-pkg-whitelist' exists in older versions and later the 'extension-pkg-allow-list' does not (though it was introduced for obvious societal reasons).
add the following into the [MASTER] section of your .pylintrc:
[MASTER]
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code.
extension-pkg-allow-list=
my_module
for versions where this is not supported (someone please update which version it changed here) use the extension-pkg-whitelist instead.

how to fix the errors for gensim on window 10?

import gensim
Warning (from warnings module): File "E:\Program
Files\Python\Python35-32\lib\site-packages\gensim\utils.py", line 1212
warnings.warn("detected Windows; aliasing chunkize to chunkize_serial") UserWarning: detected Windows; aliasing chunkize to
chunkize_serial
I installed genism by using pip and then I tried to import genism.
but, there are some troubles as above.
How can I fix these problems?

You've faced with the warning message. Library you are using alerting that it would change default behavior due to the Window's peculiarities.
After you acknowledged the message, you could suppress it by adding this to the head of your program:
import warnings
warnings.filterwarnings(
'ignore',
'detected Windows; aliasing chunkize to chunkize_serial',
)

Python warning configuration not working

So, I get the following output while running the tests for my Django project:
WARNING 2013-04-25 17:55:29,977 18482 29789392 py.warnings:3 <module>() /home/sean/dev/project/testenv/local/lib/python2.7/site-packages/django/conf/urls/defaults.py:3: DeprecationWarning: django.conf.urls.defaults is deprecated; use django.conf.urls instead
DeprecationWarning)
WARNING 2013-04-25 17:55:29,986 18482 29789392 py.warnings:10 <module>() /home/sean/dev/project/testenv/local/lib/python2.7/site-packages/django/utils/copycompat.py:10: DeprecationWarning: django.utils.copycompat is deprecated; use the native copy module instead
DeprecationWarning)
I'd like to figure out where in my code, or in what dependencies, these are originating.
http://docs.python.org/2/library/warnings.html
According to this, I can do the following to make Warnings be thrown as Exception and get a traceback:
At the top of my manage.py, before any other code is called:
import warnings
warnings.filterwarnings(
'error', r".*",
Warning, r'.*')
This causes "PendingDeprecationWarning: django.utils.simplejson is deprecated; use json instead." to be thrown, so I used:
import warnings
warnings.filterwarnings(
'error', r".*",
Warning, r'.*')
warnings.filterwarnings(
'ignore', r".*",
Warning, r'django\.utils\.simplejson')
But it's still printing out the previous DeprecationWarnings and not raising them as errors like it's supposed to.
Edit:
Also, I tried putting print and pdb statements at the beginning of warnings.warn_explicit(), the place where the filter logic happens, and they never get called.

You could run the tests or the local django server with the -W error option:
$ python -W error manage.py runserver
or
$ python -W error manage.py test
That will treat warnings as errors.

Turn off sqlalchemy warnings in nosetests

I'm trying to suppress all sqlalchemy warnings while running my test suite with nosetests. I read Turn off a warning in sqlalchemy
.............................../Users/ca/.pythonbrew/venvs/Python-2.7.3/api/lib/python2.7/site-packages/SQLAlchemy-0.7.5-py2.7-macosx-10.7-x86_64.egg/sqlalchemy/engine/default.py:330: Warning: Field 'random_id' doesn't have a default value
cursor.execute(statement, parameters)
I included this in my package's __init__.py file:
def setup_package():
"""Setup the test during the whole session.
Run by nosetests
"""
# Suppress all SQLAlchemy warnings
warnings.filterwarnings("ignore", category=sa_exc.SAWarning)
With the proper imports. I know it is run by nosetests because I tried some other stuff which raised error. The only thing is that it has no effect whatsoever. Warnings are still displayed.
Any idea?
Thanks!

It seems that nose overwrites whatever you set with:
warnings.filterwarnings("ignore")
However you can filter warnings during nose test with a command line option to nose. e.g.:
$ nosetests --logging-filter=SAWarning
I found that this still may not work under all circumstances. If this is the case you can try:
$ python -W ignore `which nosetests`

How can I check for unused import in many Python files?

I remember when I was developing in C++ or Java, the compiler usually complains for unused methods, functions or imports. In my Django project, I have a bunch of Python files which have gone through a number of iterations. Some of those files have a few lines of import statement at the top of the page and some of those imports are not used anymore. Is there a way to locate those unused imports besides eyeballing each one of them in each file?
All my imports are explicit, I don't usually write from blah import *

PyFlakes (similar to Lint) will give you this information.
pyflakes python_archive.py
Example output:
python_archive.py:1: 'python_archive2.SomeClass' imported but unused

Use a tool like pylint which will signal these code defects (among a lot of others).
Doing these kinds of 'pre-runtime' checks is hard in a language with dynamic typing, but pylint does a terrific job at catching these typos / leftovers from refactoring etc ...

You can also consider vulture as one of several options.
Installation
pip install vulture # from PyPI
Usage
vulture myscript.py
For all python files under your project.
find . -name "*.py" | xargs vulture | grep "unused import"
Example
Applies to the code below.
import numpy as np
import pandas as pd
df = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
The results are as follows.
➜ vulture myscript.py
myscript.py:1: unused import 'np' (90% confidence)
myscript.py:4: unused variable 'df' (60% confidence)

I use flake8 to check the style, and then isort+autoflake to auto remove the unused imports.
Check: See more at flake8 vs pyflake
pip install flake8 --user
flake8 .
Reformat: see more at why isort+autoflake
pip install isort autoflake --user
isort -sl .
autoflake --remove-all-unused-imports -i -r .
isort -m 3 .
Recently(since 2023), I use ruff to replace autoflake/flake8
pip install --upgrade --user ruff
ruff --fix /path/to/file_or_folder

Have a look at PyChecker. It is a debugging tool and able to find unused variables and modules.

I have been using pyflakes successfully and wished to auto-remove the unused imports.
I recently found autoflake:
Uses pyflakes for checking.
Defaults to removing unused standard library imports and redundant pass statements.
Has options for removing other unused imports and unused variables.

If you use the eclipse IDE with pydev and mylyn, it provides automatic checking and highlighting for unused imports, among other things. It integrates with pylint as well.

autoflake is an improved version of pyflakes with additional options.
It uses pyflakes under the hood, I would recommend to use it instead of using pyflakes directly.

I agree with using PyFlakes. It's like linting, but it excludes styling errors.
UPDATE
How to run: pyflakes <your python file> or pyflakes <your folder containing python files>
BE CAREFUL!!!
If you run it just with command pyflakes, it takes a really long time like it is never-ending. My hypothesis is it is trying to check every python file in your machine/folder when you call it that way.

Importchecker is a commandline utility to find unused imports in Python modules.

You can use the following user setting:
"python.linting.pylintEnabled": true,
"python.linting.pylintArgs": [
"--enable=W0614"
]
But I think, you'll need to go through all files yourself and hit "save".

You can easily use pycln to do that, just do:
pip3 install pycln
pycln path_of_your_file.py -a
And then all the unused imports are going to be removed!