I'm trying to put a simple log into my script. This log should tell me where is the error and as much as possible info needed to repair the script.
I've put print to file str(e) into each except but it provides a very few info to know what is going wrong.
How could I make it elaborated? For example the whole not catched exception text which I can see in the console?
try:
#code
except Exception as e:
print_to_file(log.txt,str(e))
try this,
import traceback
try:
1/0
except Exception as e:
with open('log.txt', 'a') as f:
f.write(str(e))
f.write(traceback.format_exc())
If you want a better solution should use Logger that manage timestamps, file size, and rotation for syou (doing a logger handler)
this is an example with logger, timestamp and rotation
import logging
from logging.handlers import RotatingFileHandler
import traceback
logger = logging.getLogger("Rotating Log")
logger.setLevel(logging.ERROR)
handler = RotatingFileHandler("log.txt", maxBytes=10000, backupCount=5)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
try:
1/0
except Exception as e:
logger.error(str(e))
logger.error(traceback.format_exc())
Related
I have a file named helper.py
import logging
import os
from json import load
def get_config(value):
with open('config.json','r') as f:
result=load(f)[value]
return result
def get_logger(name,level):
logpath=get_config("log_path")
if not os.path.exists(logpath):
os.mkdir(logpath)
logger = logging.getLogger(name)
if not bool(logger.handlers):
formatter = logging.Formatter('%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',datefmt='%Y-%m-%d %H:%M:%S')
fh = logging.FileHandler(os.path.join(logpath,f'{get_config("log_file_name")}.log'),mode="w",encoding='utf-8')
fh.setFormatter(formatter)
logger.addHandler(fh)
ch = logging.StreamHandler()
ch.setFormatter(formatter)
ch.setLevel(level)
logger.addHandler(ch)
return logger
LOGGER=get_logger("MyLogger",logging.INFO)
This is config.json:
{
"save_path" : "results/",
"log_path" : "logs/",
"log_file_name" : "MyLog"
}
let's say I am using LOGGER from helper it in x.py
from helper import LOGGER
logger=LOGGER
def div(x,y):
try:
logger.info("inside div")
return x/y
except Exception as e:
logger.error(f"div failed due to {e.message if 'message' in dir(e) else e}")
I am using this LOGGER in other files by importing helper.LOGGER for logging purposes but it's not printing anything on the console nor writing in a log file
My attempt:
I tried adding sys.stdout in StreamHandler() It doesn't worked
Then I tried setting the level of fh but nothing works
I tried adding basicConfig() instead of fileHandler() but then printing to console using print() and the output of logs is not coming in the correct order
Kindly let me know where I go wrong
Any help is appreciated :)
Thanks :)
You are not setting the level on the LOGGER, which by default is warning. This is why your info level log is not appearing. The Python documentation has a flow chart illustrating when a log will be logged: https://docs.python.org/3/howto/logging.html#logging-flow
The first thing it does, is that before a logger sends a log to their handlers it checks if the level is enabled for the logger. You should add logger.setLevel(level) in your get_logger().
I am trying to build a logger logger_script.py for my python scripts that:
outputs a log file with customizable log level.
outputs a console output with customizable log level (not necessarily equal to the log file's one)
logs unhandled exceptions both to the log file and to the console
I achieved the first two points by following the answer to "https://stackoverflow.com/questions/29087297/is-there-a-way-to-change-the-filemode-for-a-logger-object-that-is-not-configured/29087645 ". I adapted it a to my needs and it now looks like:
import sys
import logging
def create_logger(log_filename, logfile_level, console_level):
# create logger and file handler
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
fh = logging.FileHandler(log_filename, mode='w')
fh.setLevel(logfile_level)
# create console handler with independent log level
ch = logging.StreamHandler(stream=sys.stdout)
ch.setLevel(console_level)
formatter = logging.Formatter('[%(asctime)s] %(levelname)8s: %(message)s' +
' (%(filename)s:%(lineno)s)',
datefmt='%m-%d, %H:%M:%S')
fh.setFormatter(formatter)
ch.setFormatter(formatter)
if (logger.hasHandlers()): #clear pre-existing logs (?)
logger.handlers.clear()
logger.addHandler(ch)
logger.addHandler(fh)
return logger
#create the logger: file log + console output
logger = create_logger("LogFile.log", logging.INFO, logging.WARNING)
##### piece of code with handled exceptions: #####
beta = 3
while beta > -3:
try:
2/beta
logger.info("division successful".rjust(20))
except ZeroDivisionError:
logger.exception("ZeroDivisionError".rjust(20))
beta -= 1
##### piece of code with unhandled exception #####
gamma = 1/0
However, when running a code with an unhandled exception (see last line), these do not get passed to the log file, only to the console.
I followed advices from Logging uncaught exceptions in Python and added the following snippet;
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
either before or after the logger creation lines, but it does not work in my case.
How can I make unhandled exception appear, together with their traceback message?
I would like to avoid encapsulating the whole code into a try: except statement.
I would like to avoid encapsulating the whole code into a try: except statement.
I'd recommend doing the whole code in def main() then - it would be already useful if you decide to reuse parts of your code so that no code gets executed when importing the file. :)
Then you can do a catch-all thing in if __name__== "__main__"::
if __name__ == "__main__":
try:
main()
except Exception as e:
logger.exception("Program stopped because of a critical error.")
raise
[That's literally how I do it in my own code. It doesn't need any tricks, I literally added it for unexpected exceptions happened so that I could debug later.]
raise re-raises the same error that was caught - so it's uncaught to the outside world. And .exception does the same thing as .error but includes the exception info on its own.
I accidentally called logging.info() in a wrong way using Python3, and the library itself threw an error to the console, but the FileHandler failed to catch such error. So is there a way to catch all the errors no matter where they are thrown.
The error message looks like:
--- Logging error ---
...
TypeError: not all arguments converted during string formatting
Call stack:
File "<ipython-input-12-5ba547bc4aeb>", line 1, in <module>
logging.info(1,1)
Message: 1
Arguments: (1,)
Copying the following code can reproduce my question. The log file can catch logging.info() and the error of ZeroDivisionError, but it failed to catch error messages thrown by the logging library.
import logging
logger_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s: %(message)s')
logger_handler = logging.FileHandler('/Users/your_name/Desktop/logging.log')
logger_handler.setLevel(logging.DEBUG)
logger_handler.setFormatter(logger_formatter)
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(logger_handler)
try:
logging.info('test')
logging.info(1,1)
1/0
except:
logging.exception("")
Log file output:
2021-03-10 18:07:32,315 - root - INFO: test
2021-03-10 18:07:32,324 - root - ERROR:
Traceback (most recent call last):
File "<ipython-input-1-6a4f609a80ca>", line 17, in <module>
1/0
ZeroDivisionError: division by zero
Logging all errors that can happen during logging is impossible, because the error may be such that it breaks logging and if that would trigger a logging call it would lead to an infinite loop. You can however implement custom error handling by overriding handleError for your Handlers, and if you are feeling particularly brave, attempt to write a log within that error handler. Based on your code it would look something like this:
import logging
class MyFileHandler(logging.FileHandler):
def handleError(self, record):
logging.error('There was an error when writing a log. Msg was: '+str(record.msg))
logger_handler = MyFileHandler('custom.log')
logger_handler.setLevel(logging.DEBUG)
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(logger_handler)
try:
logging.info('test')
logging.info(1,1)
1/0
except:
logging.exception("")
Of course if you would rather have an exception that bubbles up instead you could just raise from handleError.
When I logging an error in a decorator, the logging pathname is not what I want.
logging.conf:
[loggers]
keys=root
[handlers]
keys=console
[formatters]
keys=console
[logger_root]
...
[handler_console]
...
[formatter_console]
format=%(levelname)s - File "%(pathname)s", line %(lineno)s, %(funcName)s: %(message)s
Nomally, logging an error in file /home/lizs/test/app.py:
def app():
try:
a # error, on line 12
except Exception, err:
logging.getLogger().error(str(err))
Debug message on console:
ERROR - File "/home/lizs/test/app.py", line 12, app: global name 'a' is not defined
The above logging pathname(/home/lizs/test/app.py) is what I want. But when I use a decorator:
/home/lizs/test/app.py:
from decorators import logging_decorator
#logging_decorator
def app():
a
/home/lizs/test/decorators.py:
def logging_decorator(func):
def error_log():
try:
func() # on line 10
except Exception, err:
logging.getLogger().error(str(err))
return error_log
The debug message:
ERROR - File "/home/lizs/test/decorators.py", line 10, error_log: global name 'a' is not defined
Now, the logging pathname is a pathname of the decorator (/home/lizs/test/decorators.py).
How to set the logging pathname to /home/lizs/test/app.py when I use decorator.
Solution:
Try this:
app.py:
from decorators import logging_decorator
#logging_decorator
def app():
a
app()
decorators.py:
import logging
import inspect
# init logger
logger = logging.getLogger()
formatter = logging.Formatter('%(levelname)s - File %(real_pathname)s,'
' line %(real_lineno)s, %(real_funcName)s: %(message)s')
console_handle = logging.StreamHandler()
console_handle.setFormatter(formatter)
logger.addHandler(console_handle)
def logging_decorator(func):
def error_log():
try:
func()
except Exception as err:
logger.error(err, extra={'real_pathname': inspect.getsourcefile(func), # path to source file
'real_lineno': inspect.trace()[-1][2], # line number from trace
'real_funcName': func.__name__}) # function name
return error_log
Explanation:
According to docs here you can pass a dictionary as extra argument to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like.
Thus, because we can't modify pathname directly, this approach with real_pathname is most straight possible.
Links:
inspect.getsourcefile
inspect.trace
logging.message
Your problem is that your exception handler is one level upper than where the exception was initially raised, so you will have to examine the stacktrace and manually build a LogRecord with the correct file/line information:
def logging_decorator(func):
def error_log():
try:
func() # on line 10
except Exception, err:
tb = sys.exc_info()[2] # extract the current exception info
exc_tup = traceback.extract_tb(tb)[-1] # extract the deeper stack frame
logger = logging.getLogger()
# manually build a LogRecord from that stack frame
lr = logger.makeRecord(logger.name,
logging.ERROR, exc_tup[0], exc_tup[1],
str(err), {}, None, exc_tup[2])
logger.handle(lr) # and ask the logging system to process it
return error_log
when I run my python script in command line I get an error output when trying to retrieve an xml file. I would like to write this error message to a logfile.
import logging
logger = logging.getLogger('fileChecker')
hdlr = logging.FileHandler('./fileChecker.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.WARNING)
try:
fileXML = mylibrary.getXML(enf, name)
logger.info('got xml file ok')
except TypeError:
errorCount += 1
logger.error('we have a problem')
mylibrary.getXML() is throwing an error here that I can see in commandline output.
how do I get that to be written to my log file?
You can catch the exception object that was raised:
try:
fileXML = myLibrary.getXML(enf, name)
logger.info('got xml file ok')
except TypeError, e:
errorCount += 1
logger.error('We have a problem: ' + str(e))
Note that if you're using the Python logging module, there is also the Logger.exception method.