I wonder what is the advantage of using warnings.warn over using just print and why should I use it.
Not only the code is a bit more messy, but also the warnings.warn's output:
/path/to/script/script.py:42: UserWarning: Warning message.
warn("Warning message.", stacklevel=1)
I just don't see a need to print a script's path or a code fragment with the desired message.
Are my thoughts relevant, or there are some qualities of warnings.warn I'm blind to?
Or maybe, there are some other better ways to handle warnings?
warnings.warn is different from print.
It could show different kind of Warnings: Categories
These warnings could be filtered: enter link description here
So Warnings are very configurable, could be printed (stderr), do nothing, or thrown an Exception.
Logging is another thing completely different, I would say is more useful for a common application, to create logs in a file or just show them: Logging
The trouble with the documentation at pytest, I don't understand where you can see the possible values + I need to see ALL the output in the log at ALL. How to do it?
Pytest uses the builtin logging module in the background. When you check the Logging Documentation, you can find your desired logging levels.
Each higher level includes all the lower ones aswell. So set it as high as you want, then all of the other should be logged aswell.
I have a python script with an error handling using the logging module. Although this python script works when imported to google colab, it doesn't log the errors in the log file.
As an experiment, I tried this following script in google colab just to see if it writes log at all
import logging
logging.basicConfig(filename="log_file_test.log",
filemode='a',
format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
datefmt='%H:%M:%S',
level=logging.DEBUG)
logging.info("This is a test log ..")
To my dismay, it didn't even create a log file named log_file_test.log. I tried running the same script locally and it did produce a file log_file_test.log with the following text
13:20:53,441 root INFO This is a test log ..
What is it that I am missing here?
For the time being, I am replacing the error logs with print statements, but I assume that there must be a workaround to this.
Perhaps you've reconfigured your environment somehow? (Try Runtime menu -> Reset all runtimes...) Your snippets works exactly as written for me --
logging.basicConfig can be run just once*
Any subsequent call to basicConfig is ignored.
* unless you are in Python 3.8 and use the flag force=True
logging.basicConfig(filename='app.log',
level=logging.DEBUG,
force=True, # Resets any previous configuration
)
Workarounds (2)
(1) You can easily reset the Colab workspace with this command
exit
Wait for it to come back and try your commands again.
(2) But, if you plan to do the reset more than once and/or are learning to use logging, maybe it is better to use %%python magic to run the entire cell in a subprocess. See photo below.
What is it that I am missing here?
Deeper understanding of how logging works. It is a bit tricky, but there are many good webs explaining the gotchas.
In Colab
https://realpython.com/python-logging
[This answer][1] cover the issue.
You have to:
Clear your log handlers from the environment with logging.root.removeHandler
Set log level with logging.getLogger('RootLogger').setLevel(logging.DEBUG).
Setting level with logging.basicConfig only did not work for me.
I am printing some set of values in console and this is a program which runs for than an hour. There were run time warning in red during the run. However, when I scroll up to see them, they don't appear, as the program is running very fast displaying new values as it runs.
Is there any way for me to display only run time warnings alone or to see the entire values printed previously (as they are warnings, it does not stop the program from running)?
I don't know how you are "printing some set of values" in your console but if you are using the python logging module and your "warnings" are set with the WARN level (all your other stuff is INFO, or DEBUG etc) you can set the logger to only output for WARN and above (ERROR and CRITICAL).
import logging
logger = logging.getLogger('spam_application')
logger.setLevel(logging.WARN)
See more examples in the logging cookbook.
Another option is to set the history of your terminal (I don't know what you're using so can't give exact instructions) to store more lines of your command line.
Finally you could pipe the output to grep (if using a *NIX system) looking for warning or similar:
python your_script.py | grep warning
For simple debugging in a complex project is there a reason to use the python logger instead of print? What about other use-cases? Is there an accepted best use-case for each (especially when you're only looking for stdout)?
I've always heard that this is a "best practice" but I haven't been able to figure out why.
The logging package has a lot of useful features:
Easy to see where and when (even what line no.) a logging call is being made from.
You can log to files, sockets, pretty much anything, all at the same time.
You can differentiate your logging based on severity.
Print doesn't have any of these.
Also, if your project is meant to be imported by other python tools, it's bad practice for your package to print things to stdout, since the user likely won't know where the print messages are coming from. With logging, users of your package can choose whether or not they want to propogate logging messages from your tool or not.
One of the biggest advantages of proper logging is that you can categorize messages and turn them on or off depending on what you need. For example, it might be useful to turn on debugging level messages for a certain part of the project, but tone it down for other parts, so as not to be taken over by information overload and to easily concentrate on the task for which you need logging.
Also, logs are configurable. You can easily filter them, send them to files, format them, add timestamps, and any other things you might need on a global basis. Print statements are not easily managed.
Print statements are sort of the worst of both worlds, combining the negative aspects of an online debugger with diagnostic instrumentation. You have to modify the program but you don't get more, useful code from it.
An online debugger allows you to inspect the state of a running program; But the nice thing about a real debugger is that you don't have to modify the source; neither before nor after the debugging session; You just load the program into the debugger, tell the debugger where you want to look, and you're all set.
Instrumenting the application might take some work up front, modifying the source code in some way, but the resulting diagnostic output can have enormous amounts of detail, and can be turned on or off to a very specific degree. The python logging module can show not just the message logged, but also the file and function that called it, a traceback if there was one, the actual time that the message was emitted, and so on. More than that; diagnostic instrumentation need never be removed; It's just as valid and useful when the program is finished and in production as it was the day it was added; but it can have it's output stuck in a log file where it's not likely to annoy anyone, or the log level can be turned down to keep all but the most urgent messages out.
anticipating the need or use for a debugger is really no harder than using ipython while you're testing, and becoming familiar with the commands it uses to control the built in pdb debugger.
When you find yourself thinking that a print statement might be easier than using pdb (as it often is), You'll find that using a logger pulls your program in a much easier to work on state than if you use and later remove print statements.
I have my editor configured to highlight print statements as syntax errors, and logging statements as comments, since that's about how I regard them.
In brief, the advantages of using logging libraries do outweigh print as below reasons:
Control what’s emitted
Define what types of information you want to include in your logs
Configure how it looks when it’s emitted
Most importantly, set the destination for your logs
In detail, segmenting log events by severity level is a good way to sift through which log messages may be most relevant at a given time. A log event’s severity level also gives you an indication of how worried you should be when you see a particular message. For instance, dividing logging type to debug, info, warning, critical, and error. Timing can be everything when you’re trying to understand what went wrong with an application. You want to know the answers to questions like:
“Was this happening before or after my database connection died?”
“Exactly when did that request come in?”
Furthermore, it is easy to see where a log has occurred through line number and filename or method name even in which thread.
Here's a functional logging library for Python named loguru.
If you use logging then the person responsible for deployment can configure the logger to send it to a custom location, with custom information. If you only print, then that's all they get.
Logging essentially creates a searchable plain text database of print outputs with other meta data (timestamp, loglevel, line number, process etc.).
This is pure gold, I can run egrep over the log file after the python script has run.
I can tune my egrep pattern search to pick exactly what I am interested in and ignore the rest. This reduction of cognitive load and freedom to pick my egrep pattern later on by trial and error is the key benefit for me.
tail -f mylogfile.log | egrep "key_word1|key_word2"
Now throw in other cool things that print can't do (sending to socket, setting debug levels, logrotate, adding meta data etc.), you have every reason to prefer logging over plain print statements.
I tend to use print statements because it's lazy and easy, adding logging needs some boiler plate code, hey we have yasnippets (emacs) and ultisnips (vim) and other templating tools, so why give up logging for plain print statements!?
I would add to all other mentionned advantages that the print function in standard configuration is buffered. The flush may occure only at the end of the current block (the one where the print is).
This is true for any program launched in a non interactive shell (codebuild, gitlab-ci for instance) or whose output is redirected.
If for any reason the program is killed (kill -9, hard reset of the computer, …), you may be missing some line of logs if you used print for the same.
However, the logging library will ensure to flush the logs printed to stderr and stdout immediately at any call.