Develop Reference

mypy can't find type hints for black

I have a file test.py which only contains the line import black. When I run mypy test.py, I get the following error:
test.py:1: error: Skipping analyzing 'black': found module but no type hints or library stubs
test.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
However, when I look at the source for black, it seems like everything has type hints. What do I need to do differently in mypy to get it to use the type hints when I import black?
Environment setup:
conda create -n test python=3.7 -y
conda activate test
pip install black mypy
EDIT - From the link mentioned in the error message:
Mypy will not try inferring the types of any 3rd party libraries you have installed unless they either have declared themselves to be PEP 561 compliant stub package or have registered themselves on typeshed
I had assumed black have satisfied one of these requirements, but it does not seem to be registered on typeshed. From here, it seems that a package must have a py.typed file to be PEP 561-compliant with inline annotations, and black doesn't seem to have that either.
My question still stands - given that the typing information is already there in black, how can I (without needing a PR to be approved somewhere) get mypy to use that?
I don't see any details about the py.typed file - if that is just an empty file and only its existence matters, I assume I can create it somewhere and then typing will work?

EDIT - this has been fixed in black now (PR here, merged 5/8/2020), so in new versions no workaround is necessary.
Creating an empty file named py.typed in the install directory is sufficient:
cd ~/anaconda/envs/test/lib/python3.7/site-packages # or wherever your packages are installed
# black.py is directly in site-packages, but py.typed needs to be in a directory for the module
mkdir black
touch black/py.typed

How does pylint use easy_install at runtime?

Sorry this is a long question. See the sentence in bold at the bottom for the TL;DR version.
I've spent many hours trying to track down a problem where pylint sometimes doesn't report all the errors in a module. Note that it does find some errors (e.g. long lines), just not all of them (e.g. missing docstrings).
I'm running pylint 1.7.2 on Ubuntu 16.04. (The version available from apt was 1.5.2 but installing via pip gives 1.7.2.)
We typically run pylint from tox, with a tox.ini that looks something like this (this is a cut-down version):
[tox]
envlist = py35
[testenv]
setenv =
MODULE_NAME=our_module
ignore_errors = True
deps =
-r../requirements.txt
whitelist_externals = bash
commands =
pip install --editable=file:///{toxinidir}/../our_other_module
pip install -e .
bash -c \'set -o pipefail; pylint --rcfile=../linting/pylint.cfg our_module | tee pylint.log\'
Amongst other things, the ../requirements.txt file contains a line for pylint==1.7.2.
The behaviour is like this:
[wrong] When the line that imports our_other_module is present, pylint appears to complete successfully and not report any warnings, even though there are errors in the our_module code that it should pick up.
[correct] When that line is commented out, pylint generates the expected warnings.
As part of tracking this down I took two copies of the .tox folder with and without the module import, naming them .tox-no-errors-reported and .tox-with-errors-reported respectively.
So now, even without sourcing their respective tox virtualenvs, I can do the following:
$ .tox-no-errors-reported/py35/bin/pylint --rcfile=../linting/pylint.cfg our_module -- reports no linting warnings
$ .tox-with-errors-reported/py35/bin/pylint --rcfile=../linting/pylint.cfg our_module -- reports the expected linting warnings
(where I just changed the pylint script's #! line in each case to reference the python3.5 inside that specific .tox directory instead of the unrenamed .tox)
By diffing .tox-no-errors-reported and .tox-with-errors-reported, I've found that they are very similar. But I can make the "no errors" version start to report errors by removing the path to our_other_module from .tox-no-errors-reported/py35/lib/python3.5/site-packages/easy-install.pth.
So my question is why is pylint using easy_install at runtime, and what is it picking up from our other component that is causing it to fail to report some errors.
As I understand it, pylint has dependencies on astroid and logilab-common, but including these in the requirements.txt doesn't make any difference.

One possible reason for the surprising pylint behavior is the --editable option.
it creates a special .egg-link file in the deployment directory, that links to your project’s source code. And, ..., it will also update the easy-install.pth file to include your project’s source code
The pth file will then affect the sys.path which affects the module import logic of astroid and it is deeply buried in the call stack of pylint.expand_files via pylint.utils.expand_modules. Also pylint identifies the module part and function names in the AST using astroid.modutils.get_module_part.
To test the theory, you can try calling some of the affected astroid functions manually:
import sys, astroid
print(sys.path)
print(astroid.modutils.get_module_part('your_package.sub_package.module'))
astroid.modutils.file_from_modpath(['your_package', 'sub_package', 'module'])

flake8/pylint fails in Tox testing environment, raises InvocationError

I've been learning about how to do testing in tox for my python project.
I have (what should be) a fairly standard tox initialization file that looks like the following:
[tox]
envlist=py27,flake8
...
[testenv:flake8]
deps=flake8
commands=flake8 library # 'library' is temp. name of project
Everything looks normal, all the test works, and even the flake8 output comes through (output below). However, tox raises an InvocationError (it does the same for testing using pylint)
flake8 recreate: /Users/shostakovich/projects/project_templates/library/.tox/flake8
flake8 installdeps: flake8
flake8 inst: /Users/shostakovich/projects/project_templates/library/.tox/dist/library-0.1.0.zip
flake8 installed: flake8==2.4.1,library==0.1.0,mccabe==0.3,pep8==1.5.7,pyflakes==0.8.1,wheel==0.24.0
library/__main__.py:12:1: F401 'os' imported but unused
library/__main__.py:13:1: F401 're' imported but unused
...
ERROR: InvocationError: '/Users/shostakovich/projects/project_templates/library/.tox/flake8/bin/flake8 library'
I am running tox 2.0.2 on MaxOSX 10.9.5. The problem goes away if I just call flake8 or pylint directly (the version of flake8 is shown above).

tox doesn't fail, it works!
Your flake8 source code check has findings and therefore tox exits with failures, that's your test result. Fix the findings and your done!
You may configure the flake8 run to ignore specific codes with a section in your tox.ini. From the flake8 docs:
[flake8]
ignore = E226,E302,E41
There are more options you may be interested in, e.g. select = ... for whitelisting enabled code checks.

You can also tell flake8 to exit without failure, even if the test results are not perfect. This will suppress the misleading InvocationError. Just add to your command --exit-zero, so for example:
commands=flake8 library --exit-zero

Why does nosetests say --with-coverage is not an option?

I am using nosetests to automatically discover and run my unittests. I would also like to have it generate coverage reports.
When I run nosetests with the following command everything works just fine
nosetests .
I looked up online that to generate the coverage, nosetests has a command line argument --with-coverage. I also double checked that this command exists using nosetests --help. However, whenever I run the following command I get the following output
nosetests --with-coverage .
Usage: nosetests [options]
nosetests: error: no such option: --with-coverage
I double checked that the coverage plugin is installed by running
nosetests --plugins
coverage shows up in the list along with a bunch of other plugins.
I also know I have coverage installed because I can manually run the coverage data collection using something along the lines of:
coverage run test.py
Am I misusing the --with-coverage option? Or is there something else I am missing?
Thanks in advance.

I never got the command line options working. I did what Janne Karila suggested and created a setup.cfg file in my projects main directory. Once I had that file I could just run nosetests with no arguments and everything would run.
One problem I had when trying to create my document was that I couldn't figure out what parameters were allowed in the config. But it turns out that any of the commands listed here https://nose.readthedocs.org/en/latest/usage.html#options can be used. Just leave off the double dashes before the command.
For reference my current config file is
[nosetests]
verbosity=1
detailed-errors=1
with-coverage=1
cover-erase=1
cover-package=application
cover-html=1
cover-html-dir=htmlcov
where=tests
This config file says to use coverage, to erase the previous run's coverage, to only report on the files in the application package, and to output an html report to the htmlcov directory.
Hopefully this will help someone else in the future.

Your syntax is correct. It maybe an issue with your environment, double check your python environment and where your have nose and coverage installed. As a sanity check, you can quickly setup a new virtualenv, install nose, and run the command with the coverage option.

As of nose 1.3.7 - the most recent version available on Pypy - that command doesn't exist:
https://github.com/nose-devs/nose/blob/release_1.3.7/nose/plugins/cover.py
It looks like the documentation is being generated from the master branch of the project that does have those options available:
https://github.com/nose-devs/nose/blob/master/nose/plugins/cover.py
What you can do is install nose from the master branch like this:
pip install git+https://github.com/nose-devs/nose#master --upgrade
It will say it just installed 1.3.7 but that's only because the version hasn't been bumped in the project setup.py file yet: https://github.com/nose-devs/nose/blob/master/setup.py#L4
Remember you have just installed an unreleased version of nose, there may be other bugs.

How can I check for unused import in many Python files?

I remember when I was developing in C++ or Java, the compiler usually complains for unused methods, functions or imports. In my Django project, I have a bunch of Python files which have gone through a number of iterations. Some of those files have a few lines of import statement at the top of the page and some of those imports are not used anymore. Is there a way to locate those unused imports besides eyeballing each one of them in each file?
All my imports are explicit, I don't usually write from blah import *

PyFlakes (similar to Lint) will give you this information.
pyflakes python_archive.py
Example output:
python_archive.py:1: 'python_archive2.SomeClass' imported but unused

Use a tool like pylint which will signal these code defects (among a lot of others).
Doing these kinds of 'pre-runtime' checks is hard in a language with dynamic typing, but pylint does a terrific job at catching these typos / leftovers from refactoring etc ...

You can also consider vulture as one of several options.
Installation
pip install vulture # from PyPI
Usage
vulture myscript.py
For all python files under your project.
find . -name "*.py" | xargs vulture | grep "unused import"
Example
Applies to the code below.
import numpy as np
import pandas as pd
df = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
The results are as follows.
➜ vulture myscript.py
myscript.py:1: unused import 'np' (90% confidence)
myscript.py:4: unused variable 'df' (60% confidence)

I use flake8 to check the style, and then isort+autoflake to auto remove the unused imports.
Check: See more at flake8 vs pyflake
pip install flake8 --user
flake8 .
Reformat: see more at why isort+autoflake
pip install isort autoflake --user
isort -sl .
autoflake --remove-all-unused-imports -i -r .
isort -m 3 .
Recently(since 2023), I use ruff to replace autoflake/flake8
pip install --upgrade --user ruff
ruff --fix /path/to/file_or_folder

Have a look at PyChecker. It is a debugging tool and able to find unused variables and modules.

I have been using pyflakes successfully and wished to auto-remove the unused imports.
I recently found autoflake:
Uses pyflakes for checking.
Defaults to removing unused standard library imports and redundant pass statements.
Has options for removing other unused imports and unused variables.

If you use the eclipse IDE with pydev and mylyn, it provides automatic checking and highlighting for unused imports, among other things. It integrates with pylint as well.

autoflake is an improved version of pyflakes with additional options.
It uses pyflakes under the hood, I would recommend to use it instead of using pyflakes directly.

I agree with using PyFlakes. It's like linting, but it excludes styling errors.
UPDATE
How to run: pyflakes <your python file> or pyflakes <your folder containing python files>
BE CAREFUL!!!
If you run it just with command pyflakes, it takes a really long time like it is never-ending. My hypothesis is it is trying to check every python file in your machine/folder when you call it that way.

Importchecker is a commandline utility to find unused imports in Python modules.

You can use the following user setting:
"python.linting.pylintEnabled": true,
"python.linting.pylintArgs": [
"--enable=W0614"
]
But I think, you'll need to go through all files yourself and hit "save".

You can easily use pycln to do that, just do:
pip3 install pycln
pycln path_of_your_file.py -a
And then all the unused imports are going to be removed!