Should I log before or after an operation? - python

I'm thinking about where to write the log record around an operation. Here are two different styles. The first one, write log before the operation.
Before:
log.info("Perform operation XXX")
operation()
And here is a different style, write the log after the operation.
After:
operation()
log.info("Operation XXX is done.")
With the before-style, the logging records say what is going to do now. The pro of this style is that when something goes wrong, developer can detect it easily, because they know what is the program doing now. But the con is that you are not sure is the operation finished correctly, if something wrong is inside the operation, for example, a function call gets blocked there and never return, you can't never know it by reading the logging records. With the after-style, you are sure the operation is done.
Of course, we can mix those two style together
Both:
log.info("Perform operation XXX")
operation()
log.info("Operation XXX is done.")
But I feel that is kinda verbose, it makes double logging records. So, here is my question - what is the good logging style? I would like to know how do you think.

I'd typically use two different log levels.
The first one I put on a "debug" level, and the second one on an "info" level. That way typical production machines would only log what's being done, but I can turn on the debug logging and see what it tries to do before it errors out.

It all depends what you want to log. If you're interested in the code getting to the point where it's about to do an operation. If you want to make sure the operation succeeded, do it after. If you want both, do both.

Maybe you could use something like a try catch ? Here 's a naive python example :
try :
operation()
log.info("Operation XXX is done.")
except Exception:
log.info("Operation xxx Failed")
raise Exception() # optional : if you want to propagate failure to another try catch statement and/or crash eventually.
Operation will be launched.
If it doesn't fail (no exception raised) you get a success statement in the logs.
If it fails (by raising an exception. Like disc full or whatever you are trying to do), Exception is caught and you get a failure statement.
Log is more meaning full. You get to keep the verbosity to a oneliner and get to know if operation succeeded. Best of all choices.
Oh and you get a hook point where you can add some code to be executed in case of failure.
I hope it help.

There's another style that I've seen used in Linux boot scripts and in strace. It's got the advantages of your combined style with less verbosity, but you've got to make sure that your logging facility isn't doing any buffering. I don't know log.info, so here's a rough example with print:
print "Doing XXX... ", # Note lack of newline :)
operation()
print "Done."
(Since in most cases print uses buffering, using this example verbatim won't work properly. You won't see "Doing XXX" until you see the "Done". But you get the general idea.)
The other disadvantage of this style is that things can get mixed up if you have multiple threads writing to the same log.

Related

Python ignore all exceptions and procede

Hello this question is about python.
Is there a way to ignore all kinds off exceptions. I know i could just put the whole code in an huge try cahtch but i want it to continue even if one part fails and in result of this some other parts fail too. One way to achieve this would be to put every single line in a try except statement. But is there an other more elegant way to do this?
Well , you can
1 - Put every separate part in a try catch
try:
#something
except:
pass
2 - Put everything in a bigger try catch
try:
#do something
#so something else
#do something else
except:
pass
or you can use contextlib.suppress (https://docs.python.org/3/library/contextlib.html#contextlib.suppress) as Random Davis suggested to ignore certain types of exceptions
But ignoring all exception is a really bad idea , instead you should do
try:
#something
except:
#something else
As far as I know , there is no other "elegant" way to ignore exceptions
(This could have been a comment but I lack rep to post comments)
Besides using try, my python professor will generally comment out a function that he isn't testing so he could test a specific function. If part of your code isn't working try adding a breakpoint before debugging, then you can run your code up until the breakpoint to see if a certain line is doing what you want it to. In visual studio code you can do this by clicking in the empty space just to the left of the line numbers, and you will see a red dot on the line if done correctly.
If you are referencing that a function isn't doing what you want it to do, employing these methods will help you find the error in your ways, starting with tracking the input all the way until you get the output you want, going function by function.
If you're referencing that your code is simply too broken to function correctly, your code will always run until it returns an error, and if the error is early in that process between converting an input to an output, you can come across a multitude of reasons behind this that will stop functions that are supposed to run afterwards from working correctly. If that is the case comment out the later functions until your first one is working, and keep working chronologically to debug those errors. In the future, I highly recommend posting your exact code when asking a coding question, otherwise it can be hard for others to extract the necessary information to answer your question effectively. Good luck!
In reference to Global-Occult's answer, although you can basically try something and except extraneous information, you really don't want to be coding like that, because to develop higher level programs that extra information will no longer be extraneous, in fact it could be very important data that will allow you to develop a program much further.

Pyhon: A better way to run a function when an error occurs in the program?

I created a big program that does a lot of different stuff. In this program, I added some error management but I would like to add management for critical errors which should start the critical_error_function().
So basically, I've used :
try :
//some fabulous code
except :
critical error(error_type)
But I am here to ask if a better way to do this...
In Python exceptions are the intended way of error handling. Assuming you wrap your whole program in one try-except block, a better way would be to
only try-except-wrap the lines that can generate exceptions instead of your complete program
catch them with a specific exception such as ValueError or even your own custom exception instead of the blank except statement
handle them appropriately. Handling could mean skipping this value, logging the error or calling your critical_error_function.

Pythonic Way To Warn Developers For Broken Code

Is there a pythonic way to throw an exception to other developers to warn them about using a piece of code with bugs in it?
For example:
def times_two(x):
raise BrokenException("Attn.This code is unreliable. Only works on positive numbers")
x = abs(x) * 2
return x
I understand that I can raise a generic exception with a message, or even derive my own exception classes, but i just want to know if there is a built-in, pythonic way to do something like this.
And also, I understand that why the actual times_two function doesn't work. That was just an example function.
This is not something to validate input parameters, or even returned values.
This is simply to mark a function as potentially unreliable.
The code must be used in some areas under very specific circumstances, but when devs are writing code and run across this function should be warned of the limitations.
Your example is pretty flawed for any use case in which alerting the developers would be needed. This would need to alert the user not to input a negative number.
def times_two(x):
if x < 0:
raise BrokenException("Attn user. Don't give me negitive numbers.")
return x * 2
Although, I think if your example more accurately described an actual error needing developer attention then you should just fix that and not put it into production knowing there is an error in it.
sentry.io on the other hand can help find errors and help developers fix errors while in production. You may want to look into that if warnings isn't for you. From their README.me:
Sentry fundamentally is a service that helps you monitor and fix
crashes in realtime. The server is in Python, but it contains a full
API for sending events from any language, in any application.
Builtin Exception 'ValueError' is the one that should be used.
def times_two(x):
if x < 0:
raise ValueError('{} is not a positive number.'.format(x))
return x * 2
This seems like an XY problem. The original problem is that you have some code which is incomplete or otherwise known to not work. If it is something you are currently working on, then the correct tool to use here is your version control. With Git, you would create a new branch which only be merged into master and prepared for release to production after you have completed the work. You shouldn't release a partial implementation.
Do you want to stop execution when the function is called? If so, then some sort of exception, like the BrokenException in your example is a good way of doing this.
But if you want to warn the caller, and then continue on anyway, then you want a Warning instead of an exception. You can still create your own:
class BrokenCodeWarning(Warning)
pass
When you raise BrokenCodeWarning, execution will not be halted by default, but a warning will be printed to stderr.
The warnings filter controls whether warnings are ignored, displayed, or turned into errors (raising an exception).
https://docs.python.org/3.7/library/warnings.html#the-warnings-filter

Causing an event in Python when any error happens ever?

I'm not sure exactly how to ask what I'm asking, and I don't know if any sample code would really be relevant here, so if clarification is necessary just ask.
I have a non-trivial program in Python that does a couple of things. It reads from some SQL Server database tables, executes some DDL in SQL Server (both of these with pyodbc), analyzes some data, and has a GUI to orchestrate everything so users in the future besides me can automate the process.
Everything is functioning as it should, but obviously I don't expect future users to always play by the rules. I can explicitly indicate what input is wrong (i.e. fields left empty), but there are quite a bit of things that can go wrong. Try-catch structures are out of the question because they cause a few issues in the web of things happening in my program, some of which are embedded in the resources I'm using, not to mention I feel like it's probably not good form to have them everywhere.
That being said, I'm wondering if there's a way to cause an event (likely just a dialog box saying that an error occurred), so that a user without a view of the output window would know something had gone wrong. It would be nice if I could also grab the error message itself, but that isn't necessary. I'm alright with the error still occurring so long as the program can keep going if the issue is corrected.
I'm not sure if this is possible, or if it is possible what form it would take, like something that monitors the output or listens for errors, but I'm open to all information about this.
Thank you
You can wrap your code with a try/except block and name the Exception to print it in the dialog, for example:
try:
# your code
except Exception as e:
print(e) # change this to be in your dialog
This way you will not use try/except many times in different places and you will catch basically any Exception. The other way is to raise custom exceptions for each error and catch them with except.
If you still don't want to use try/except at all, maybe start a thread to keep checking for certain variables (through a loop) and whenever you want to start an error event you just set that variable to True and the thread will start the error dialog. For instance:
import time
import threading
test1 = False
test2 = False
def errorCheck():
while True:
if test1:
# start error dialog for error1
if test2:
# start error dialog for error2
time.sleep(0.1)
if __name__ == '__main__':
t = threading.Thread(target=errorCheck)
t.daemon = True
t.start()
# start your app
However, I recommend using try/except instead.

In python, why use logging instead of print?

For simple debugging in a complex project is there a reason to use the python logger instead of print? What about other use-cases? Is there an accepted best use-case for each (especially when you're only looking for stdout)?
I've always heard that this is a "best practice" but I haven't been able to figure out why.
The logging package has a lot of useful features:
Easy to see where and when (even what line no.) a logging call is being made from.
You can log to files, sockets, pretty much anything, all at the same time.
You can differentiate your logging based on severity.
Print doesn't have any of these.
Also, if your project is meant to be imported by other python tools, it's bad practice for your package to print things to stdout, since the user likely won't know where the print messages are coming from. With logging, users of your package can choose whether or not they want to propogate logging messages from your tool or not.
One of the biggest advantages of proper logging is that you can categorize messages and turn them on or off depending on what you need. For example, it might be useful to turn on debugging level messages for a certain part of the project, but tone it down for other parts, so as not to be taken over by information overload and to easily concentrate on the task for which you need logging.
Also, logs are configurable. You can easily filter them, send them to files, format them, add timestamps, and any other things you might need on a global basis. Print statements are not easily managed.
Print statements are sort of the worst of both worlds, combining the negative aspects of an online debugger with diagnostic instrumentation. You have to modify the program but you don't get more, useful code from it.
An online debugger allows you to inspect the state of a running program; But the nice thing about a real debugger is that you don't have to modify the source; neither before nor after the debugging session; You just load the program into the debugger, tell the debugger where you want to look, and you're all set.
Instrumenting the application might take some work up front, modifying the source code in some way, but the resulting diagnostic output can have enormous amounts of detail, and can be turned on or off to a very specific degree. The python logging module can show not just the message logged, but also the file and function that called it, a traceback if there was one, the actual time that the message was emitted, and so on. More than that; diagnostic instrumentation need never be removed; It's just as valid and useful when the program is finished and in production as it was the day it was added; but it can have it's output stuck in a log file where it's not likely to annoy anyone, or the log level can be turned down to keep all but the most urgent messages out.
anticipating the need or use for a debugger is really no harder than using ipython while you're testing, and becoming familiar with the commands it uses to control the built in pdb debugger.
When you find yourself thinking that a print statement might be easier than using pdb (as it often is), You'll find that using a logger pulls your program in a much easier to work on state than if you use and later remove print statements.
I have my editor configured to highlight print statements as syntax errors, and logging statements as comments, since that's about how I regard them.
In brief, the advantages of using logging libraries do outweigh print as below reasons:
Control what’s emitted
Define what types of information you want to include in your logs
Configure how it looks when it’s emitted
Most importantly, set the destination for your logs
In detail, segmenting log events by severity level is a good way to sift through which log messages may be most relevant at a given time. A log event’s severity level also gives you an indication of how worried you should be when you see a particular message. For instance, dividing logging type to debug, info, warning, critical, and error. Timing can be everything when you’re trying to understand what went wrong with an application. You want to know the answers to questions like:
“Was this happening before or after my database connection died?”
“Exactly when did that request come in?”
Furthermore, it is easy to see where a log has occurred through line number and filename or method name even in which thread.
Here's a functional logging library for Python named loguru.
If you use logging then the person responsible for deployment can configure the logger to send it to a custom location, with custom information. If you only print, then that's all they get.
Logging essentially creates a searchable plain text database of print outputs with other meta data (timestamp, loglevel, line number, process etc.).
This is pure gold, I can run egrep over the log file after the python script has run.
I can tune my egrep pattern search to pick exactly what I am interested in and ignore the rest. This reduction of cognitive load and freedom to pick my egrep pattern later on by trial and error is the key benefit for me.
tail -f mylogfile.log | egrep "key_word1|key_word2"
Now throw in other cool things that print can't do (sending to socket, setting debug levels, logrotate, adding meta data etc.), you have every reason to prefer logging over plain print statements.
I tend to use print statements because it's lazy and easy, adding logging needs some boiler plate code, hey we have yasnippets (emacs) and ultisnips (vim) and other templating tools, so why give up logging for plain print statements!?
I would add to all other mentionned advantages that the print function in standard configuration is buffered. The flush may occure only at the end of the current block (the one where the print is).
This is true for any program launched in a non interactive shell (codebuild, gitlab-ci for instance) or whose output is redirected.
If for any reason the program is killed (kill -9, hard reset of the computer, …), you may be missing some line of logs if you used print for the same.
However, the logging library will ensure to flush the logs printed to stderr and stdout immediately at any call.

Categories

Resources