Related
I'm trying to disable warning C0321 ("more than one statement on a single line" -- I often put if statements with short single-line results on the same line), in Pylint 0.21.1 (if it matters: astng 0.20.1, common 0.50.3, and Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56)).
I've tried adding disable=C0321 in the Pylint configuration file, but Pylint insists on reporting it anyway. Variations on that line (like disable=0321 or disable=C321) are flagged as errors, so Pylint does recognize the option properly. It's just ignoring it.
Is this a Pylint bug, or am I doing something wrong? Is there a way around this?
I'd really like to get rid of some of this noise.
pylint --generate-rcfile shows it like this:
[MESSAGES CONTROL]
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time.
#enable=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once).
#disable=
So it looks like your ~/.pylintrc should have the disable= line/s in it inside a section [MESSAGES CONTROL].
Starting from Pylint v. 0.25.3, you can use the symbolic names for disabling warnings instead of having to remember all those code numbers. E.g.:
# pylint: disable=locally-disabled, multiple-statements, fixme, line-too-long
This style is more instructive than cryptic error codes, and also more practical since newer versions of Pylint only output the symbolic name, not the error code.
The correspondence between symbolic names and codes can be found here.
A disable comment can be inserted on its own line, applying the disable to everything that comes after in the same block. Alternatively, it can be inserted at the end of the line for which it is meant to apply.
If Pylint outputs "Locally disabling" messages, you can get rid of them by including the disable locally-disabled first as in the example above.
I had this problem using Eclipse and solved it as follows:
In the pylint folder (e.g. C:\Python26\Lib\site-packages\pylint), hold Shift, right-click and choose to open the windows command in that folder. Type:
lint.py --generate-rcfile > standard.rc
This creates the standard.rc configuration file. Open it in Notepad and under [MESSAGES CONTROL], uncomment
disable= and add the message ID's you want to disable, e.g.:
disable=W0511, C0321
Save the file, and in Eclipse → Window → Preferences → PyDev → *pylint, in the arguments box, type:
--rcfile=C:\Python26\Lib\site-packages\pylint\standard.rc
Now it should work...
You can also add a comment at the top of your code that will be interpreted by Pylint:
# pylint: disable=C0321
Pylint message codes.
Adding e.g. --disable-ids=C0321 in the arguments box does not work.
All available Pylint messages are stored in the dictionary _messages, an attribute of an instance of the pylint.utils.MessagesHandlerMixIn class. When running Pylint with the argument --disable-ids=... (at least without a configuration file), this dictionary is initially empty, raising a KeyError exception within Pylint (pylint.utils.MessagesHandlerMixIn.check_message_id().
In Eclipse, you can see this error-message in the Pylint Console (windows* → show view → Console, select Pylint console from the console options besides the console icon.)
To disable a warning locally in a block, add
# pylint: disable=C0321
to that block.
There are several ways to disable warnings & errors from Pylint. Which one to use has to do with how globally or locally you want to apply the disablement -- an important design decision.
Multiple Approaches
In one or more pylintrc files.
This involves more than the ~/.pylintrc file (in your $HOME directory) as described by Chris Morgan. Pylint will search for rc files, with a precedence that values "closer" files more highly:
A pylintrc file in the current working directory; or
If the current working directory is in a Python module (i.e. it contains an __init__.py file), searching up the hierarchy of Python modules until a pylintrc file is found; or
The file named by the environment variable PYLINTRC; or
If you have a home directory that isn’t /root:
~/.pylintrc; or
~/.config/pylintrc; or
/etc/pylintrc
Note that most of these files are named pylintrc -- only the file in ~ has a leading dot.
To your pylintrc file, add lines to disable specific pylint messages. For example:
[MESSAGES CONTROL]
disable=locally-disabled
Further disables from the pylint command line, as described by Aboo and Cairnarvon. This looks like pylint --disable=bad-builtin. Repeat --disable to suppress additional items.
Further disables from individual Python code lines, as described by Imolit. These look like some statement # pylint: disable=broad-except (extra comment on the end of the original source line) and apply only to the current line. My approach is to always put these on the end of other lines of code so they won't be confused with the block style, see below.
Further disables defined for larger blocks of Python code, up to complete source files.
These look like # pragma pylint: disable=bad-whitespace (note the pragma key word).
These apply to every line after the pragma. Putting a block of these at the top of a file makes the suppressions apply to the whole file. Putting the same block lower in the file makes them apply only to lines following the block. My approach is to always put these on a line of their own so they won't be confused with the single-line style, see above.
When a suppression should only apply within a span of code, use # pragma pylint: enable=bad-whitespace (now using enable not disable) to stop suppressing.
Note that disabling for a single line uses the # pylint syntax while disabling for this line onward uses the # pragma pylint syntax. These are easy to confuse especially when copying & pasting.
Putting It All Together
I usually use a mix of these approaches.
I use ~/.pylintrc for absolutely global standards -- very few of these.
I use project-level pylintrc at different levels within Python modules when there are project-specific standards. Especially when you're taking in code from another person or team, you may find they use conventions that you don't prefer, but you don't want to rework the code. Keeping the settings at this level helps not spread those practices to other projects.
I use the block style pragmas at the top of single source files. I like to turn the pragmas off (stop suppressing messages) in the heat of development even for Pylint standards I don't agree with (like "too few public methods" -- I always get that warning on custom Exception classes) -- but it's helpful to see more / maybe all Pylint messages while you're developing. That way you can find the cases you want to address with single-line pragmas (see below), or just add comments for the next developer to explain why that warning is OK in this case.
I leave some of the block-style pragmas enabled even when the code is ready to check in. I try to use few of those, but when it makes sense for the module, it's OK to do as documentation. However I try to leave as few on as possible, preferably none.
I use the single-line-comment style to address especially potent errors. For example, if there's a place where it actually makes sense to do except Exception as exc, I put the # pylint: disable=broad-except on that line instead of a more global approach because this is a strange exception and needs to be called out, basically as a form of documentation.
Like everything else in Python, you can act at different levels of indirection. My advice is to think about what belongs at what level so you don't end up with a too-lenient approach to Pylint.
This is a FAQ:
4.1 Is it possible to locally disable a particular message?
Yes, this feature has been added in Pylint 0.11. This may be done by
adding
# pylint: disable=some-message,another-one at the desired
block level or at the end of the desired line of code.
4.2 Is there a way to disable a message for a particular module only?
Yes, you can disable or enable (globally disabled) messages at the
module level by adding the corresponding option in a comment at the
top of the file:
# pylint: disable=wildcard-import, method-hidden
# pylint: enable=too-many-lines
You can disable messages by:
numerical ID: E1101, E1102, etc.
symbolic message: no-member, undefined-variable, etc.
the name of a group of checks. You can grab those with pylint --list-groups.
category of checks: C, R, W, etc.
all the checks with all.
See the documentation (or run pylint --list-msgs in the terminal) for the full list of Pylint's messages. The documentation also provide a nice example of how to use this feature.
You can also use the following command:
pylint --disable=C0321 test.py
My Pylint version is 0.25.1.
You just have to add one line to disable what you want to disable.
E.g.,
#pylint: disable = line-too-long, too-many-lines, no-name-in-module, import-error, multiple-imports, pointless-string-statement, wrong-import-order
Add this at the very beginning of your module.
In case this helps someone, if you're using Visual Studio Code, it expects the file to be in UTF-8 encoding. To generate the file, I ran pylint --generate-rcfile | out-file -encoding utf8 .pylintrc in PowerShell.
As per Pylint documentation, the easiest is to use this chart:
C convention-related checks
R refactoring-related checks
W various warnings
E errors, for probable bugs in the code
F fatal, if an error occurred which prevented Pylint from doing further processing.
So one can use:
pylint -j 0 --disable=I,E,R,W,C,F YOUR_FILES_LOC
Sorry for diverging a bit from the initial question, about poster's general preference, which would be better addressed by a global configuration file.
But, as in many popular answers, I tend to prefer seeing in my code what could trigger warnings, and eventually inform contributors as well.
My comment to answer from #imolit needs to stay short, here are some details.
For multiple-statements message, it's probably better to disable it at block or module level, like this
# pylint: disable=multiple-statements
My use-case being now attribute-defined-outside-init in a unittest setup(), I opted for a line-scoped message disabling, using the message code to avoid the line-too-long issue.
class ParserTest(unittest.TestCase):
def setUp(self):
self.parser = create_parser() # pylint: disable=W0201
The correspondance can be found locally with a command like
$ pylint --list-msgs | grep 'outside-init'
:attribute-defined-outside-init (W0201): *Attribute %r defined outside __init__*
Of course, you would similarly retrieve the symbolic name from the code.
Python syntax does permit more than one statement on a line, separated by semicolon (;). However, limiting each line to one statement makes it easier for a human to follow a program's logic when reading through it.
So, another way of solving this issue, is to understand why the lint message is there and not put more than one statement on a line.
Yes, you may find it easier to write multiple statements per line, however, Pylint is for every other reader of your code not just you.
My pylint kept ignoring the disable list in my .pylintrc. Finally, I realized that I was executing:
pylint --disable=all --enable=F,E,W
which was overriding the disable list in my .pylintrc.
The correct command to show only Fatal, Errors, Warnings, is:
pylint --disable=C,R
Edit "C:\Users\Your User\AppData\Roaming\Code\User\settings.json"
and add 'python.linting.pylintArgs' with its lines at the end as shown below:
{
"team.showWelcomeMessage": false,
"python.dataScience.sendSelectionToInteractiveWindow": true,
"git.enableSmartCommit": true,
"powershell.codeFormatting.useCorrectCasing": true,
"files.autoSave": "onWindowChange",
"python.linting.pylintArgs": [
"--load-plugins=pylint_django",
"--errors-only"
],
}
I need to use pyfits (http://www.stsci.edu/institute/software_hardware/pyfits) to open/write some spectra for the work I am doing. Problem is, everytime I use the "writeto" function to write a .fits file and it overwrites it, I get a "Overwrite existing file: XXX.fits" message on the screen. Is it possible to tell the program to not show this partiular message?
I already checked and could not find a keyword for the "writeto" function that would deactivate this message, so I was thinking if there was anyway to tell python to redirect all output (except if it is an error) of a particular function to something like /dev/null or similar.
Worst case scenario, I thought that maybe using "logging" and redirect all output to a file and thats it..
Any ideas?
The library uses the Python warnings module to emit a warning when you 'clobber' an existing file.
You can use that same module to suppress the warning:
import warnings
with warnings.catch_warnings():
warnings.simplefilter("ignore")
pyfits.writeto(...)
Using the catch_warnings() context manager suppresses all warnings that pyfits.writeto() might raise. You can also configure filters for specific messages to be ignored:
import warnings
warnings.filterwarnings('ignore', message='Overwriting existing file .*', module='pyfits\.hdu.*')
would ignore messages that start with Overwriting existing file raised by modules that start with pyfits.hdu for example.
You can use warnings.filterwarnings to suppress the messages. Simply add
warnings.filterwarnings('ignore', message='Overwriting existing file')
We are using Pylint within our build system.
We have a Python package within our code base that has throwaway code, and I'd like to disable all warnings for a module temporarily so I can stop bugging the other devs with these superfluous messages. Is there an easy way to pylint: disable all warnings for a module?
From the Pylint FAQ:
With Pylint < 0.25, add
# pylint: disable-all
at the beginning of the module.
Pylint 0.26.1 and up have renamed that directive to
# pylint: skip-file
(but the first version will be kept for backward compatibility).
In order to ease finding which modules are ignored a information-level message I0013 is emitted. With recent versions of Pylint, if you use the old syntax, an additional I0014 message is emitted.
Pylint has five "categories" for messages (of which I'm aware).
These categories were very obvious in the past, but the numbered Pylint messages have now been replaced by names. For example, C0302 is now too-many-lines. But the 'C' tells us that too-many-lines is a Convention message. This is confusing, because Convention messages frequently just show up as a warning, since many systems (such as Syntastic) seem to classify everything as either a warning or an error. However, the Pylint report still breaks things down into these categories, so it's still definitely supported.
Your question specifically refers to Warnings, and all Pylint Warning message names start with 'W'.
It was a bit difficult for me to track this down, but this answer eventually led me to the answer. Pylint still supports disabling entire categories of messages. So, to disable all Warnings, you would do:
disable=W
This can be used at the command-line:
$ pylint --disable=W myfile.py
Or, you can put it in your pylintrc file:
[MESSAGES CONTROL]
disable=W
Note: you may already have the disable option in your rc file, in which case you should append the 'W' to this list.
Or, you can put it inline in your code, where it will work for the scope into which it is placed:
# pylint: disable=W
To disable it for an entire file, it's best to put it at the very top of the file. However, even at the very top of the file, I found I was still getting the trailing-newlines warning message (technically a convention warning, but I'm getting to that).
In my case, I had a library written by someone from long ago. It worked well, so there was really no need to worry about modern Python convention, etc. All I really cared about were the errors that would likely break my code.
My solution was to disable all the Warning, Convention, and Refactoring messages for this one file only by placing the following Pylint command on the first line:
# pylint: disable=W,C,R
Aside from the aforementioned message for trailing newlines, this did exactly what I needed.
Yes, you can specify # pylint: skip-file in the file, but it is bad practice to disable all warnings for a file.
If you want to disable specific warnings only, this can be done by adding a comment such as # pylint: disable=message-name to disable the specified message for the remainder of the file, or at least until # pylint: enable=message-name.
Example:
# pylint: disable=no-member
class C123:
def __init__(self):
self.foo_x = self.bar_x
# pylint: enable=no-member
class C456:
def __init__(self):
self.foo_x = self.bar_x
Another option is to use the --ignore command line option to skip analysis for some files.
My use case is to run pylint *.py to process all files in a directory, except that I want to skip one particular file.
Adding # pylint: skip-file caused Pylint to fail with I: 8, 0: Ignoring entire file (file-ignored). Adding # pylint: disable=file-ignored does not fix that. Presumably, it's a global error rather than a file-specific one.
The solution was to include --disable=file-ignored in the Pylint command options. It took way too long to figure this out; there shouldn't be a file-ignored error when you explicitly ignore a file.
I just did this for pytest tests organised in to classes and it may be useful to someone. These were throwing up lots of errors but I like the class based structure when I'm running in the cli.
I add this below imports on all my test files and it seems to work. Might be nicer to do it once for the test directory if anyone knows the solution for that, but I don't like to ignore for the whole repo if I can avoid it
# Disable pylint errors for tests organised into classes
# pylint: disable=no-self-use too-few-public-methods
# Disable pylint errors for pytest fixtures
# pylint: disable=unused-argument invalid-name
Using it twice didn't break anything, I checked to make sure. Initially I tried brackets for a 2 line disable but that didn't work. I didn't need commas so removed them.
If anyone knows of a decorator that can mark organisational test classes equivalent to #dataclass let me know.
I'm trying to disable warning C0321 ("more than one statement on a single line" -- I often put if statements with short single-line results on the same line), in Pylint 0.21.1 (if it matters: astng 0.20.1, common 0.50.3, and Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56)).
I've tried adding disable=C0321 in the Pylint configuration file, but Pylint insists on reporting it anyway. Variations on that line (like disable=0321 or disable=C321) are flagged as errors, so Pylint does recognize the option properly. It's just ignoring it.
Is this a Pylint bug, or am I doing something wrong? Is there a way around this?
I'd really like to get rid of some of this noise.
pylint --generate-rcfile shows it like this:
[MESSAGES CONTROL]
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time.
#enable=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once).
#disable=
So it looks like your ~/.pylintrc should have the disable= line/s in it inside a section [MESSAGES CONTROL].
Starting from Pylint v. 0.25.3, you can use the symbolic names for disabling warnings instead of having to remember all those code numbers. E.g.:
# pylint: disable=locally-disabled, multiple-statements, fixme, line-too-long
This style is more instructive than cryptic error codes, and also more practical since newer versions of Pylint only output the symbolic name, not the error code.
The correspondence between symbolic names and codes can be found here.
A disable comment can be inserted on its own line, applying the disable to everything that comes after in the same block. Alternatively, it can be inserted at the end of the line for which it is meant to apply.
If Pylint outputs "Locally disabling" messages, you can get rid of them by including the disable locally-disabled first as in the example above.
I had this problem using Eclipse and solved it as follows:
In the pylint folder (e.g. C:\Python26\Lib\site-packages\pylint), hold Shift, right-click and choose to open the windows command in that folder. Type:
lint.py --generate-rcfile > standard.rc
This creates the standard.rc configuration file. Open it in Notepad and under [MESSAGES CONTROL], uncomment
disable= and add the message ID's you want to disable, e.g.:
disable=W0511, C0321
Save the file, and in Eclipse → Window → Preferences → PyDev → *pylint, in the arguments box, type:
--rcfile=C:\Python26\Lib\site-packages\pylint\standard.rc
Now it should work...
You can also add a comment at the top of your code that will be interpreted by Pylint:
# pylint: disable=C0321
Pylint message codes.
Adding e.g. --disable-ids=C0321 in the arguments box does not work.
All available Pylint messages are stored in the dictionary _messages, an attribute of an instance of the pylint.utils.MessagesHandlerMixIn class. When running Pylint with the argument --disable-ids=... (at least without a configuration file), this dictionary is initially empty, raising a KeyError exception within Pylint (pylint.utils.MessagesHandlerMixIn.check_message_id().
In Eclipse, you can see this error-message in the Pylint Console (windows* → show view → Console, select Pylint console from the console options besides the console icon.)
To disable a warning locally in a block, add
# pylint: disable=C0321
to that block.
There are several ways to disable warnings & errors from Pylint. Which one to use has to do with how globally or locally you want to apply the disablement -- an important design decision.
Multiple Approaches
In one or more pylintrc files.
This involves more than the ~/.pylintrc file (in your $HOME directory) as described by Chris Morgan. Pylint will search for rc files, with a precedence that values "closer" files more highly:
A pylintrc file in the current working directory; or
If the current working directory is in a Python module (i.e. it contains an __init__.py file), searching up the hierarchy of Python modules until a pylintrc file is found; or
The file named by the environment variable PYLINTRC; or
If you have a home directory that isn’t /root:
~/.pylintrc; or
~/.config/pylintrc; or
/etc/pylintrc
Note that most of these files are named pylintrc -- only the file in ~ has a leading dot.
To your pylintrc file, add lines to disable specific pylint messages. For example:
[MESSAGES CONTROL]
disable=locally-disabled
Further disables from the pylint command line, as described by Aboo and Cairnarvon. This looks like pylint --disable=bad-builtin. Repeat --disable to suppress additional items.
Further disables from individual Python code lines, as described by Imolit. These look like some statement # pylint: disable=broad-except (extra comment on the end of the original source line) and apply only to the current line. My approach is to always put these on the end of other lines of code so they won't be confused with the block style, see below.
Further disables defined for larger blocks of Python code, up to complete source files.
These look like # pragma pylint: disable=bad-whitespace (note the pragma key word).
These apply to every line after the pragma. Putting a block of these at the top of a file makes the suppressions apply to the whole file. Putting the same block lower in the file makes them apply only to lines following the block. My approach is to always put these on a line of their own so they won't be confused with the single-line style, see above.
When a suppression should only apply within a span of code, use # pragma pylint: enable=bad-whitespace (now using enable not disable) to stop suppressing.
Note that disabling for a single line uses the # pylint syntax while disabling for this line onward uses the # pragma pylint syntax. These are easy to confuse especially when copying & pasting.
Putting It All Together
I usually use a mix of these approaches.
I use ~/.pylintrc for absolutely global standards -- very few of these.
I use project-level pylintrc at different levels within Python modules when there are project-specific standards. Especially when you're taking in code from another person or team, you may find they use conventions that you don't prefer, but you don't want to rework the code. Keeping the settings at this level helps not spread those practices to other projects.
I use the block style pragmas at the top of single source files. I like to turn the pragmas off (stop suppressing messages) in the heat of development even for Pylint standards I don't agree with (like "too few public methods" -- I always get that warning on custom Exception classes) -- but it's helpful to see more / maybe all Pylint messages while you're developing. That way you can find the cases you want to address with single-line pragmas (see below), or just add comments for the next developer to explain why that warning is OK in this case.
I leave some of the block-style pragmas enabled even when the code is ready to check in. I try to use few of those, but when it makes sense for the module, it's OK to do as documentation. However I try to leave as few on as possible, preferably none.
I use the single-line-comment style to address especially potent errors. For example, if there's a place where it actually makes sense to do except Exception as exc, I put the # pylint: disable=broad-except on that line instead of a more global approach because this is a strange exception and needs to be called out, basically as a form of documentation.
Like everything else in Python, you can act at different levels of indirection. My advice is to think about what belongs at what level so you don't end up with a too-lenient approach to Pylint.
This is a FAQ:
4.1 Is it possible to locally disable a particular message?
Yes, this feature has been added in Pylint 0.11. This may be done by
adding
# pylint: disable=some-message,another-one at the desired
block level or at the end of the desired line of code.
4.2 Is there a way to disable a message for a particular module only?
Yes, you can disable or enable (globally disabled) messages at the
module level by adding the corresponding option in a comment at the
top of the file:
# pylint: disable=wildcard-import, method-hidden
# pylint: enable=too-many-lines
You can disable messages by:
numerical ID: E1101, E1102, etc.
symbolic message: no-member, undefined-variable, etc.
the name of a group of checks. You can grab those with pylint --list-groups.
category of checks: C, R, W, etc.
all the checks with all.
See the documentation (or run pylint --list-msgs in the terminal) for the full list of Pylint's messages. The documentation also provide a nice example of how to use this feature.
You can also use the following command:
pylint --disable=C0321 test.py
My Pylint version is 0.25.1.
You just have to add one line to disable what you want to disable.
E.g.,
#pylint: disable = line-too-long, too-many-lines, no-name-in-module, import-error, multiple-imports, pointless-string-statement, wrong-import-order
Add this at the very beginning of your module.
In case this helps someone, if you're using Visual Studio Code, it expects the file to be in UTF-8 encoding. To generate the file, I ran pylint --generate-rcfile | out-file -encoding utf8 .pylintrc in PowerShell.
As per Pylint documentation, the easiest is to use this chart:
C convention-related checks
R refactoring-related checks
W various warnings
E errors, for probable bugs in the code
F fatal, if an error occurred which prevented Pylint from doing further processing.
So one can use:
pylint -j 0 --disable=I,E,R,W,C,F YOUR_FILES_LOC
Sorry for diverging a bit from the initial question, about poster's general preference, which would be better addressed by a global configuration file.
But, as in many popular answers, I tend to prefer seeing in my code what could trigger warnings, and eventually inform contributors as well.
My comment to answer from #imolit needs to stay short, here are some details.
For multiple-statements message, it's probably better to disable it at block or module level, like this
# pylint: disable=multiple-statements
My use-case being now attribute-defined-outside-init in a unittest setup(), I opted for a line-scoped message disabling, using the message code to avoid the line-too-long issue.
class ParserTest(unittest.TestCase):
def setUp(self):
self.parser = create_parser() # pylint: disable=W0201
The correspondance can be found locally with a command like
$ pylint --list-msgs | grep 'outside-init'
:attribute-defined-outside-init (W0201): *Attribute %r defined outside __init__*
Of course, you would similarly retrieve the symbolic name from the code.
Python syntax does permit more than one statement on a line, separated by semicolon (;). However, limiting each line to one statement makes it easier for a human to follow a program's logic when reading through it.
So, another way of solving this issue, is to understand why the lint message is there and not put more than one statement on a line.
Yes, you may find it easier to write multiple statements per line, however, Pylint is for every other reader of your code not just you.
My pylint kept ignoring the disable list in my .pylintrc. Finally, I realized that I was executing:
pylint --disable=all --enable=F,E,W
which was overriding the disable list in my .pylintrc.
The correct command to show only Fatal, Errors, Warnings, is:
pylint --disable=C,R
Edit "C:\Users\Your User\AppData\Roaming\Code\User\settings.json"
and add 'python.linting.pylintArgs' with its lines at the end as shown below:
{
"team.showWelcomeMessage": false,
"python.dataScience.sendSelectionToInteractiveWindow": true,
"git.enableSmartCommit": true,
"powershell.codeFormatting.useCorrectCasing": true,
"files.autoSave": "onWindowChange",
"python.linting.pylintArgs": [
"--load-plugins=pylint_django",
"--errors-only"
],
}
Today I was thinking about a Python project I wrote about a year back where I used logging pretty extensively. I remember having to comment out a lot of logging calls in inner-loop-like scenarios (the 90% code) because of the overhead (hotshot indicated it was one of my biggest bottlenecks).
I wonder now if there's some canonical way to programmatically strip out logging calls in Python applications without commenting and uncommenting all the time. I'd think you could use inspection/recompilation or bytecode manipulation to do something like this and target only the code objects that are causing bottlenecks. This way, you could add a manipulator as a post-compilation step and use a centralized configuration file, like so:
[Leave ERROR and above]
my_module.SomeClass.method_with_lots_of_warn_calls
[Leave WARN and above]
my_module.SomeOtherClass.method_with_lots_of_info_calls
[Leave INFO and above]
my_module.SomeWeirdClass.method_with_lots_of_debug_calls
Of course, you'd want to use it sparingly and probably with per-function granularity -- only for code objects that have shown logging to be a bottleneck. Anybody know of anything like this?
Note: There are a few things that make this more difficult to do in a performant manner because of dynamic typing and late binding. For example, any calls to a method named debug may have to be wrapped with an if not isinstance(log, Logger). In any case, I'm assuming all of the minor details can be overcome, either by a gentleman's agreement or some run-time checking. :-)
What about using logging.disable?
I've also found I had to use logging.isEnabledFor if the logging message is expensive to create.
Use pypreprocessor
Which can also be found on PYPI (Python Package Index) and be fetched using pip.
Here's a basic usage example:
from pypreprocessor import pypreprocessor
pypreprocessor.parse()
#define nologging
#ifdef nologging
...logging code you'd usually comment out manually...
#endif
Essentially, the preprocessor comments out code the way you were doing it manually before. It just does it on the fly conditionally depending on what you define.
You can also remove all of the preprocessor directives and commented out code from the postprocessed code by adding 'pypreprocessor.removeMeta = True' between the import and
parse() statements.
The bytecode output (.pyc) file will contain the optimized output.
SideNote: pypreprocessor is compatible with python2x and python3k.
Disclaimer: I'm the author of pypreprocessor.
I've also seen assert used in this fashion.
assert logging.warn('disable me with the -O option') is None
(I'm guessing that warn always returns none.. if not, you'll get an AssertionError
But really that's just a funny way of doing this:
if __debug__: logging.warn('disable me with the -O option')
When you run a script with that line in it with the -O option, the line will be removed from the optimized .pyo code. If, instead, you had your own variable, like in the following, you will have a conditional that is always executed (no matter what value the variable is), although a conditional should execute quicker than a function call:
my_debug = True
...
if my_debug: logging.warn('disable me by setting my_debug = False')
so if my understanding of debug is correct, it seems like a nice way to get rid of unnecessary logging calls. The flipside is that it also disables all of your asserts, so it is a problem if you need the asserts.
As an imperfect shortcut, how about mocking out logging in specific modules using something like MiniMock?
For example, if my_module.py was:
import logging
class C(object):
def __init__(self, *args, **kw):
logging.info("Instantiating")
You would replace your use of my_module with:
from minimock import Mock
import my_module
my_module.logging = Mock('logging')
c = my_module.C()
You'd only have to do this once, before the initial import of the module.
Getting the level specific behaviour would be simple enough by mocking specific methods, or having logging.getLogger return a mock object with some methods impotent and others delegating to the real logging module.
In practice, you'd probably want to replace MiniMock with something simpler and faster; at the very least something which doesn't print usage to stdout! Of course, this doesn't handle the problem of module A importing logging from module B (and hence A also importing the log granularity of B)...
This will never be as fast as not running the log statements at all, but should be much faster than going all the way into the depths of the logging module only to discover this record shouldn't be logged after all.
You could try something like this:
# Create something that accepts anything
class Fake(object):
def __getattr__(self, key):
return self
def __call__(self, *args, **kwargs):
return True
# Replace the logging module
import sys
sys.modules["logging"] = Fake()
It essentially replaces (or initially fills in) the space for the logging module with an instance of Fake which simply takes in anything. You must run the above code (just once!) before the logging module is attempted to be used anywhere. Here is a test:
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)-8s %(message)s',
datefmt='%a, %d %b %Y %H:%M:%S',
filename='/temp/myapp.log',
filemode='w')
logging.debug('A debug message')
logging.info('Some information')
logging.warning('A shot across the bows')
With the above, nothing at all was logged, as was to be expected.
I'd use some fancy logging decorator, or a bunch of them:
def doLogging(logTreshold):
def logFunction(aFunc):
def innerFunc(*args, **kwargs):
if LOGLEVEL >= logTreshold:
print ">>Called %s at %s"%(aFunc.__name__, time.strftime("%H:%M:%S"))
print ">>Parameters: ", args, kwargs if kwargs else ""
try:
return aFunc(*args, **kwargs)
finally:
print ">>%s took %s"%(aFunc.__name__, time.strftime("%H:%M:%S"))
return innerFunc
return logFunction
All you need is to declare LOGLEVEL constant in each module (or just globally and just import it in all modules) and then you can use it like this:
#doLogging(2.5)
def myPreciousFunction(one, two, three=4):
print "I'm doing some fancy computations :-)"
return
And if LOGLEVEL is no less than 2.5 you'll get output like this:
>>Called myPreciousFunction at 18:49:13
>>Parameters: (1, 2)
I'm doing some fancy computations :-)
>>myPreciousFunction took 18:49:13
As you can see, some work is needed for better handling of kwargs, so the default values will be printed if they are present, but that's another question.
You should probably use some logger module instead of raw print statements, but I wanted to focus on the decorator idea and avoid making code too long.
Anyway - with such decorator you get function-level logging, arbitrarily many log levels, ease of application to new function, and to disable logging you only need to set LOGLEVEL. And you can define different output streams/files for each function if you wish. You can write doLogging as:
def doLogging(logThreshold, outStream=sys.stdout):
.....
print >>outStream, ">>Called %s at %s" etc.
And utilize log files defined on a per-function basis.
This is an issue in my project as well--logging ends up on profiler reports pretty consistently.
I've used the _ast module before in a fork of PyFlakes (http://github.com/kevinw/pyflakes) ... and it is definitely possible to do what you suggest in your question--to inspect and inject guards before calls to logging methods (with your acknowledged caveat that you'd have to do some runtime type checking). See http://pyside.blogspot.com/2008/03/ast-compilation-from-python.html for a simple example.
Edit: I just noticed MetaPython on my planetpython.org feed--the example use case is removing log statements at import time.
Maybe the best solution would be for someone to reimplement logging as a C module, but I wouldn't be the first to jump at such an...opportunity :p
:-) We used to call that a preprocessor and although C's preprocessor had some of those capablities, the "king of the hill" was the preprocessor for IBM mainframe PL/I. It provided extensive language support in the preprocessor (full assignments, conditionals, looping, etc.) and it was possible to write "programs that wrote programs" using just the PL/I PP.
I wrote many applications with full-blown sophisticated program and data tracing (we didn't have a decent debugger for a back-end process at that time) for use in development and testing which then, when compiled with the appropriate "runtime flag" simply stripped all the tracing code out cleanly without any performance impact.
I think the decorator idea is a good one. You can write a decorator to wrap the functions that need logging. Then, for runtime distribution, the decorator is turned into a "no-op" which eliminates the debugging statements.
Jon R
I am doing a project currently that uses extensive logging for testing logic and execution times for a data analysis API using the Pandas library.
I found this string with a similar concern - e.g. what is the overhead on the logging.debug statements even if the logging.basicConfig level is set to level=logging.WARNING
I have resorted to writing the following script to comment out or uncomment the debug logging prior to deployment:
import os
import fileinput
comment = True
# exclude files or directories matching string
fil_dir_exclude = ["__","_archive",".pyc"]
if comment :
## Variables to comment
source_str = 'logging.debug'
replace_str = '#logging.debug'
else :
## Variables to uncomment
source_str = '#logging.debug'
replace_str = 'logging.debug'
# walk through directories
for root, dirs, files in os.walk('root/directory') :
# where files exist
if files:
# for each file
for file_single in files :
# build full file name
file_name = os.path.join(root,file_single)
# exclude files with matching string
if not any(exclude_str in file_name for exclude_str in fil_dir_exclude) :
# replace string in line
for line in fileinput.input(file_name, inplace=True):
print "%s" % (line.replace(source_str, replace_str)),
This is a file recursion that excludes files based on a list of criteria and performs an in place replace based on an answer found here: Search and replace a line in a file in Python
I like the 'if __debug_' solution except that putting it in front of every call is a bit distracting and ugly. I had this same problem and overcame it by writing a script which automatically parses your source files and replaces logging statements with pass statements (and commented out copies of the logging statements). It can also undo this conversion.
I use it when I deploy new code to a production environment when there are lots of logging statements which I don't need in a production setting and they are affecting performance.
You can find the script here: http://dound.com/2010/02/python-logging-performance/