I would very much like to integrate pylint into the build process for
my python projects, but I have run into one show-stopper: One of the
error types that I find extremely useful--:E1101: *%s %r has no %r
member*--constantly reports errors when using common django fields,
for example:
E1101:125:get_user_tags: Class 'Tag' has no 'objects' member
which is caused by this code:
def get_user_tags(username):
"""
Gets all the tags that username has used.
Returns a query set.
"""
return Tag.objects.filter( ## This line triggers the error.
tagownership__users__username__exact=username).distinct()
# Here is the Tag class, models.Model is provided by Django:
class Tag(models.Model):
"""
Model for user-defined strings that help categorize Events on
on a per-user basis.
"""
name = models.CharField(max_length=500, null=False, unique=True)
def __unicode__(self):
return self.name
How can I tune Pylint to properly take fields such as objects into account? (I've also looked into the Django source, and I have been unable to find the implementation of objects, so I suspect it is not "just" a class field. On the other hand, I'm fairly new to python, so I may very well have overlooked something.)
Edit: The only way I've found to tell pylint to not warn about these warnings is by blocking all errors of the type (E1101) which is not an acceptable solution, since that is (in my opinion) an extremely useful error. If there is another way, without augmenting the pylint source, please point me to specifics :)
See here for a summary of the problems I've had with pychecker and pyflakes -- they've proven to be far to unstable for general use. (In pychecker's case, the crashes originated in the pychecker code -- not source it was loading/invoking.)
Do not disable or weaken Pylint functionality by adding ignores or generated-members.
Use an actively developed Pylint plugin that understands Django.
This Pylint plugin for Django works quite well:
pip install pylint-django
and when running pylint add the following flag to the command:
--load-plugins pylint_django
Detailed blog post here.
I use the following: pylint --generated-members=objects
If you use Visual Studio Code do this:
pip install pylint-django
And add to VSC config:
"python.linting.pylintArgs": [
"--load-plugins=pylint_django"
],
My ~/.pylintrc contains
[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id
the last two are specifically for Django.
Note that there is a bug in PyLint 0.21.1 which needs patching to make this work.
Edit: After messing around with this a little more, I decided to hack PyLint just a tiny bit to allow me to expand the above into:
[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id,[a-zA-Z]+_set
I simply added:
import re
for pattern in self.config.generated_members:
if re.match(pattern, node.attrname):
return
after the fix mentioned in the bug report (i.e., at line 129).
Happy days!
django-lint is a nice tool which wraps pylint with django specific settings : http://chris-lamb.co.uk/projects/django-lint/
github project: https://github.com/lamby/django-lint
Because of how pylint works (it examines the source itself, without letting Python actually execute it) it's very hard for pylint to figure out how metaclasses and complex baseclasses actually affect a class and its instances. The 'pychecker' tool is a bit better in this regard, because it does actually let Python execute the code; it imports the modules and examines the resulting objects. However, that approach has other problems, because it does actually let Python execute the code :-)
You could extend pylint to teach it about the magic Django uses, or to make it understand metaclasses or complex baseclasses better, or to just ignore such cases after detecting one or more features it doesn't quite understand. I don't think it would be particularly easy. You can also just tell pylint to not warn about these things, through special comments in the source, command-line options or a .pylintrc file.
I resigned from using pylint/pychecker in favor of using pyflakes with Django code - it just tries to import module and reports any problem it finds, like unused imports or uninitialized local names.
This is not a solution, but you can add objects = models.Manager() to your Django models without changing any behavior.
I myself only use pyflakes, primarily due to some dumb defaults in pylint and laziness on my part (not wanting to look up how to change the defaults).
Try running pylint with
pylint --ignored-classes=Tags
If that works, add all the other Django classes - possibly using a script, in say, python :P
The documentation for --ignore-classes is:
--ignored-classes=<members names>
List of classes names for which member
attributes should not be checked
(useful for classes with attributes
dynamicaly set). [current: %default]
I should add this is not a particular elegant solution in my view, but it should work.
For neovim & vim8 use w0rp's ale plugin. If you have installed everything correctly including w0rp's ale, pylint & pylint-django. In your vimrc add the following line & have fun developing web apps using django.
Thanks.
let g:ale_python_pylint_options = '--load-plugins pylint_django'
The solution proposed in this other question it to simply add get_attr to your Tag class. Ugly, but works.
So far I have found no real solution to that but work around:
In our company we require a pylint
score > 8. This allows coding
practices pylint doesn't understand
while ensuring that the code isn't
too "unusual". So far we havn't seen
any instance where E1101 kept us
from reaching a score of 8 or
higher.
Our 'make check' targets
filter out "for has no 'objects'
member" messages to remove most of
the distraction caused by pylint not
understanding Django.
For heroku users, you can also use Tal Weiss's answer to this question using the following syntax to run pylint with the pylint-django plugin (replace timekeeping with your app/package):
# run on the entire timekeeping app/package
heroku local:run pylint --load-plugins pylint_django timekeeping
# run on the module timekeeping/report.py
heroku local:run pylint --load-plugins pylint_django timekeeping/report.py
# With temporary command line disables
heroku local:run pylint --disable=invalid-name,missing-function-docstring --load-plugins pylint_django timekeeping/report.py
Note: I was unable to run without specifying project/package directories.
If you have issues with E5110: Django was not configured., you can also invoke as follows to try to work around that (again, change timekeeping to your app/package):
heroku local:run python manage.py shell -c 'from pylint import lint; lint.Run(args=["--load-plugins", "pylint_django", "timekeeping"])'
# With temporary command line disables, specific module
heroku local:run python manage.py shell -c 'from pylint import lint; lint.Run(args=["--load-plugins", "pylint_django", "--disable=invalid-name,missing-function-docstring", "timekeeping/report.py"])'
Related
Is there any way to add custom rules to flake8?
There are bunch of rules here https://lintlyci.github.io/Flake8Rules/ but I can't find the rules' source code in flake8's git repo.
I want to write a custom rule.
You need to write your own Flake8 plugin.
From the Flake8's docs:
Flake8 is useful on its own but a lot of Flake8’s popularity is due to its extensibility. Our community has developed plugins that augment Flake8’s behaviour. Most of these plugins are uploaded to PyPI. The developers of these plugins often have some style they wish to enforce.
You can't exactly add custom rules to Flake8 itself, apart from submitting a feature-request and hoping it gets accepted. But it allows you to publish a plugin that can be installed in addition to Flake8, and Flake8 can then find it and use it along with its own built-in checks.
They have nice developer documentation on how to write one: Writing Plugins for Flake8. If it's a custom rule that you want checked, then it's probably a Check Plugin. It receives the same info about the code as its built-in checkers does, then you need to write your own code for checking the lines.
You can also check out the source codes of existing Flake8 plugins for inspiration, such as:
flake8-quotes (source)
flake8-return (source)
Looking to find a way, if possible, to disable specific PEP8 warnings for a Python project loaded in PyCharm at the repository level (i.e. saving a repository-committed configuration file which can apply PEP8 configuration hints to any user loading a project in PyCharm).
In a situation where other developers may contribute to a project using PyCharm. I personally do not use PyCharm myself (just a text editor) and building/linting is performed through various tox environments. There are select PEP8 rules that I am particularly not fond of, such as the promotion on injecting two blank lines in specific areas of the code (e.g. E305). While linters can be configured to ignore specific PEP8 rules and developers can invoke a command (like tox) with the same linter configuration, developers using PyCharm will still see these warnings in their environment. For example:
The problem I experience is developers will make (undesired) adjustments to the implementation and submit them for changes in pull requests. While developers can dismiss warnings themselves, I do not want developers to have to assume/interpret which PEP8 rules the project follows (aside from what may be mentioned in a CONTRIBUTING document). In addition, while source files can be modified with a # noqa comment to hint to the IDE to ignore an issue on that line, I am looking for an alternative way to ignore specific PEP8 rules without peppering various # noqa hints throughout the implementation.
For example, looking for a way to disable all E305 warnings in a theoretical .pycharm file such as follows:
[pycharm]
ignore = E305
After some additional investigation, there does not appear to be support (as of 2020-09-23; v2020.2) for repository-specific PEP8 warning customization.
The IDE uses pycodestyle for PEP8 processing. Testing out options from pycodestyle's configuration has shown that user-specific configurations will disable various PEP8 hints in the IDE; however, the goal is avoid having the user to configure anything in this case. While pycodestyle indicates that it will also look at setup.cfg and tox.ini, the IDE does not invoke pycodestyle in a way to use them. Examining how the IDE invokes pycodestyle shows that it will feed a source's contents through stdin for information. pycodestyle only implicitly loads supported project-specific configurations (i.e. setup.cfg or tox.ini) based on a common prefix of provided input files (if provided) on the existing working directory pycodestyle is invoked on -- unfortunately, since PyCharm does not change the working directory when invoking pycodestyle to a project's root, project-specific configurations cannot be implicitly loaded.
An issue has been created on their issue tracker:
PY-44684 | support pycodestyle project-specific configuration settings (pep8)
https://youtrack.jetbrains.com/issue/PY-44684
There is way to do it, but at the moment it requires the corresponding PyCharm's project settings to be kept in VCS. For every reported pycodestyle.py error there is a dedicated quick fix "Ignore errors like this" that includes the respective error code in the settings of "PEP 8 coding style violation" inspection (these codes are then passed directly to pycodestyle.py via its --ignore option).
You can then find and edit these codes in the inspection settings.
If a per-project inspection profile was configured, these settings are saved in .idea/inspectionProfiles/Your_Profile_Name.xml and this way can be shared with the team or external contributors.
Obviously, it won't work if you don't want to expose IDE-specific config files in the repository, especially when there is a better, tool-independent place for such preferences. The reported PY-44684 is a legitimate issue, I agree.
Is it possible to run the PyCharm linter / code style checks from command line and get the warnings/errors?
Extension to that: Is it possible to integrate that in my Travis tests?
To put some of the comments and some further research into an answer:
PyCharm comes with a small command-line utility bin/inspect.sh, which is documented here. This tool is quite limited though, and has some problems, e.g. it cannot run while the PyCharm IDE is running, and it reports somewhat incorrectly / different than the IDE. Related code can be seen e.g. here.
PyCharm does not do PEP8 code style checks by itself but uses the (bundled) pycodestyle tool.
Maybe these shortcomings can be fixed upstream. See e.g. this report, or this, or this.
I'm using this in Travis and GitHub Actions now, via this script.
This also does the necessary setup of a project.
This also compensates for the inspect.sh missing warnings by also using the pycodestyle tool and thus exactly matching the PyCharm IDE warnings.
Alternative, I'm thinking about writing an extended simple utility which basically does this. All the relevant PyCharm code is open source. I created a project page pychar-inspect for this. But this is just in the planing phase currently, and maybe obsolete when this will be addressed upstream.
As some features get deprecated with new versions of Django, is there a way to check for that on an existing project code say on github.
Could a tool do that. Is there a way to detect that through testcases.
Would it be possible to do the same against a python version.
I guess one way could be to run through a specific version of django / python using tox and then check for errors.
I am just looking for something more elegant or direct, something like which says - "Note this feature has been deprecated", something which can be done in strongly typed language like Java.
If one wanted to build such a tool, what could be starting point for that, if possible.
This is how I got tox to run one project of mine against Django 1.6, 1.7 and 1.8 with deprecation warnings on:
[tox]
envlist = {py27,py34}-{django16,django17,django18}
[testenv]
basepython =
py27: python2.7
py34: python3.4
deps =
django16: Django>=1.6,<1.7
django17: Django>=1.7,<1.8
django18: Django>=1.8,<1.9
commands =
python -Wmodule ./manage.py test
The -Wmodule argument causes Python to output each deprecation warning the first time it occurs in a module, which was good enough for me. I was able to deal with instances where I used from django.core.cache import get_cache, which will be gone in Django 1.9.
In cases where -Wmodule outputs too much, you might want to be more selective. Python's documentation gives the lowdown on how to use this argument. I've purposely not used -Werror because this would not just make the individual tests fail but would make my test suite fail to execute any test because the suite uses deprecated features.
I think this would have to be in the unit tests for your project.
If your tests exercise code that will be deprecated in a future version of Django you will get warnings. If you've jumped to a version of Django where the feature is already deprecated you'll get exceptions and failed tests of course.
You can tell the Python interpreter to promote warnings to exceptions, which would cause tests to fail.
Instructions here to apply the same trick to the popular nosetests test framework.
If you know already (from Django docs) that some code you're writing will need to change depending on Django version it is run under (eg you're distributing a reusable Django app) I would suggest a form of feature detection using try ... except
For example, here I wanted to conditionally use the new transaction.atomic feature from Django >= 1.6: .
As you anticipated, I then run the tests against different versions of Django with the help of Tox.
Is there already a way to integrate one of Python lint programs (PyLint, PyChecker, PyFlakes, etc.) with GitHub commit status API? In this way Python lint could be automatically called on pull requests to check the code and provide feedback and code (and style).
You could use something like Travis-CI, and run pylint as part of your tests, along the lines of:
language: python
install: "pip install nose pylint"
script: "nosetests && pylint"
Of course that fails commits for minor stylistic violations - you'd probably want to disable certain messages, or use pylint --errors-only to make it less stringent
I had the same question, and just found this blog post describing a project called pylint-server for doing something similar (though triggered on Travis CI build events, not pulls).
From the README:
A small Flask app to keep keep track of pylint reports and ratings on a per-repository basis.
I haven't tried it yet, so I can't comment on its quality. If anyone tries it, please comment and let us know how you like it.