Develop Reference

python: logging all errors into single log file

I am coding a tool in python and I want to put all the errors -and only the errors-(computations which didn't go through as they should have) into a single log file. Additionally I would want to have a different text in the error log file for each section of my code in order to make the error log file easy to interpret. How do I code this? Much appreciation for who could help with this!

Check out the python module logging. This is a core module for unifying logging not only in your own project but potentially in third party modules too.
For a minimal logging file example, this is taken directly from the documentation:
import logging
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
Which results in the contents of example.log:
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
However, I personally recommend using the yaml configuration method (requires pyyaml):
#logging_config.yml
version: 1
disable_existing_loggers: False
formatters:
standard:
format: '%(asctime)s [%(levelname)s] %(name)s - %(message)s'
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: standard
stream: ext://sys.stdout
file:
class: logging.FileHandler
level: DEBUG
formatter: standard
filename: output.log
email:
class: logging.handlers.SMTPHandler
level: WARNING
mailhost: smtp.gmail.com
fromaddr: to#address.co.uk
toaddrs: to#address.co.uk
subject: Oh no, something's gone wrong!
credentials: [email, password]
secure: []
root:
level: DEBUG
handlers: [console, file, email]
propagate: True
Then to use, for example:
import logging.config
import yaml
with open('logging_config.yml', 'r') as config:
logging.config.dictConfig(yaml.safe_load(config))
logger = logging.getLogger(__name__)
logger.info("This will go to the console and the file")
logger.debug("This will only go to the file")
logger.error("This will go everywhere")
try:
list = [1, 2, 3]
print(list[10])
except IndexError:
logger.exception("This will also go everywhere")
This prints:
2018-07-18 13:29:21,434 [INFO] __main__ - This will go to the console and the file
2018-07-18 13:29:21,434 [ERROR] __main__ - This will go everywhere
2018-07-18 13:29:21,434 [ERROR] __main__ - This will also go everywhere
Traceback (most recent call last):
File "C:/Users/Chris/Desktop/python_scratchpad/a.py", line 16, in <module>
print(list[10])
IndexError: list index out of range
While the contents of the log file is:
2018-07-18 13:35:55,423 [INFO] __main__ - This will go to the console and the file
2018-07-18 13:35:55,424 [DEBUG] __main__ - This will only go to the file
2018-07-18 13:35:55,424 [ERROR] __main__ - This will go everywhere
2018-07-18 13:35:55,424 [ERROR] __main__ - This will also go everywhere
Traceback (most recent call last):
File "C:/Users/Chris/Desktop/python_scratchpad/a.py", line 15, in <module>
print(list[10])
IndexError: list index out of range
Of course, you can add or remove handlers, formatters, etc, or do all of this in code (see the Python documentation) but this is my starting point whenever I use logging in a project. I find it helpful to have the configuration in a dedicated config file rather than polluting my project with defining logging in code.

If I understand the question correctly, the request was to capture only the errors in a dedicated log file, and I would do that differently.
I would stick to the BKM that all modules in the package define their own logger objects (logger = logging.getLogger(__name__)).
I'd let them be without any handlers and whenever they will emit they will look up the hierarchy tree for handlers to actually take care of the emitted messages.
At the root logger, I would add a dedicated FileHandler(filename='errors.log') and I would set the log level of that handler to logging.ERROR.
That means, whenever a logger from the package will emit something, this dedicated file-handler will discard anything below ERROR and will log into the files only ERROR and CRITICAL messages.
You could still add global StreamHandler and regular FileHandler to your root logger. Since you'll not change their log levels, they will be set to logging.NOTSET and will log everything that is emitted from the loggers in the package.
And to answer the second part of the question, the logger handlers can define their own formatting. So for the handler that handles only the errors, you could set the formatter to something like this: %(name)s::%(funcName)s:%(lineno)d - %(message)s which basically means, it will print:
the logger name (and if you used the convention to define loggers in every *.py file using __name__, then name will actually hold the hierarchical path to your module file (e.g. my_pkg.my_sub_pkg.module))
the funcName will hold the function from where the log was emitted and
lineno is the line number in the module file where the log was emitted.

Python Logging - Only for own imported modules

referring to this question here: LINK
How can I set up a config, that will only log my root script and my own sub-scripts? The question of the link asked for disabling all imported modules, but that is not my intention.
My root setup:
import logging
from exchangehandler import send_mail
log_wp = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s - %(levelname)s [%(filename)s]: %(name)s %(funcName)20s - Message: %(message)s',
datefmt='%d.%m.%Y %H:%M:%S',
filename='C:/log/myapp.log',
filemode='a')
handler = logging.StreamHandler()
log_wp.addHandler(handler)
log_wp.debug('This is from root')
send_mail('address#eg.com', 'Request', 'Hi there')
My sub-module exchangehandler.py:
import logging
log_wp = logging.getLogger(__name__)
def send_mail(mail_to,mail_subject,mail_body, mail_attachment=None):
log_wp.debug('Hey this is from exchangehandler.py!')
m.send_and_save()
myapp.log:
16.07.2018 10:27:40 - DEBUG [test_script.py]: __main__ <module> - Message: This is from root
16.07.2018 10:28:02 - DEBUG [exchangehandler.py]: exchangehandler send_mail - Message: Hey this is from exchangehandler.py!
16.07.2018 10:28:02 - DEBUG [folders.py]: exchangelib.folders get_default_folder - Message: Testing default <class 'exchangelib.folders.SentItems'> folder with GetFolder
16.07.2018 10:28:02 - DEBUG [services.py]: exchangelib.services get_payload - Message: Getting folder ArchiveDeletedItems (archivedeleteditems)
16.07.2018 10:28:02 - DEBUG [services.py]: exchangelib.services get_payload - Message: Getting folder ArchiveInbox (archiveinbox)
My problem is, that the log-file contains also a lot of information of the exchangelib-module, that is imported in exchangehandler.py. Either the imported exchangelib-module is configured incorrectly or I have made a mistake. So how can I reduce the log-output only to my logging messages?
EDIT:
An extract of the folder.py of the exchangelib-module. This is not anything that I have written:
import logging
log = logging.getLogger(__name__)
def get_default_folder(self, folder_cls):
try:
# Get the default folder
log.debug('Testing default %s folder with GetFolder', folder_cls)
# Use cached instance if available
for f in self._folders_map.values():
if isinstance(f, folder_cls) and f.has_distinguished_name:
return f
return folder_cls.get_distinguished(account=self.account)

The imported exchangelib module is not configured at all when it comes to logging. You are configuring it implicitly by calling logging.basicConfig() in your main module.
exchangelib does create loggers and logs to them, but by default these loggers do not have handlers and formatters attached, so they don't do anything visible. What they do, is propagating up to the root logger, which by default also has no handlers and formatters attached.
By calling logging.basicConfig in your main module, you actually attach handlers to the root logger. Your own, desired loggers propagate to the root logger, hence the messages are written to the handlers, but the same is true for the exchangelib loggers from that point onwards.
You have at least two options here. You can explicitly configure "your" named logger(s):
main module
import logging
log_wp = logging.getLogger(__name__) # or pass an explicit name here, e.g. "mylogger"
hdlr = logging.StreamHandler()
fhdlr = logging.FileHandler("myapp.log")
log_wp.addHandler(hdlr)
log_wp.addHandler(fhdlr)
log_wp.setLevel(logging.DEBUG)
The above is very simplified. To explicitly configure multiple named loggers, refer to the logging.config HowTo
If you rather want to stick to just using the root logger (configured via basicConfig()), you can also explicitly disable the undesired loggers after exchangelib has been imported and these loggers have been created:
logging.getLogger("exchangelib.folders").disabled = True
logging.getLogger("exchangelib.services").disabled = True
If you don't know the names of the loggers to disable, logging has a dictionary holding all the known loggers. So you could temporarily do this to see all the loggers your program creates:
# e.g. after the line 'log_wp.addHandler(handler)'
print([k for k in logging.Logger.manager.loggerDict])
Using the dict would also allow you to do sth. like this:
for v in logging.Logger.manager.loggerDict.values():
if v.name.startswith('exchangelib'):
v.disabled = True

universal logger name in python logging yaml config file

Ok, so the situation is I need to use a yaml config file for logging.(don't ask - I just need it :) ). And when writing the 'loggers:' directive I would like to use one logger and to be able to fetch it from multiple modules in my app using getLogger(__name__). I know how to do it if I use a normal python config file for logging, but I can't to find a way to do the same with a yaml file.
using python 2.7
So long story short, that's what I have(this is just a simplified example of my problem, not part of the actual application :) ):
#this is app.py
import logging.config
import yaml
import os
def init_logging():
path = 'logging.yaml'
if os.path.exists(path):
with open(path, 'r') as f:
config = yaml.safe_load(f.read())
logging.config.dictConfig(config['logging'])
main()
def main():
logger = logging.getLogger('app')
logger.debug("done!")
init_logging()
and here's the logging.yaml config file:
logging:
version: 1
formatters:
brief:
format: '%(message)s'
default:
format: '%(asctime)s %(levelname)-8s [%(name)s] %(message)s'
datefmt: '%Y-%m-%d %H:%M:%S'
handlers:
console:
class: logging.StreamHandler
level: DEBUG
loggers:
app:
handlers: [console]
level: DEBUG
So as is it is here - it works. the 'done!' message shows in the console.
But I want to be able to set in the config not some distinct logger (here I called it 'app') but a universal, like if it was in a .py config it'would be
"loggers": {
"": {
"handlers": ["console"],
"level": "DEBUG",
},
}
and then I would be using logging.getLogger(__name__) in different modules and it would always use the one "" logger and show me the messages.
So is there a way to create a universal logger in yaml? like the "": in the python logging config?
I tried (), ~, null - those don't do the work.
Basically I need to be able to call loggers with any names I want and to get one specified logger.
And yep - I can create a root directive in the yaml and call it by using logging.getLogger()

The trick is to use a special logger called root, outside the list of other loggers.
I found this in the Python documentation:
root - this will be the configuration for the root logger. Processing of the configuration will be as for any logger, except that the propagate setting will not be applicable.
Here's the configuration from your question changed to use the root key:
logging:
version: 1
formatters:
brief:
format: '%(message)s'
default:
format: '%(asctime)s %(levelname)-8s [%(name)s] %(message)s'
datefmt: '%Y-%m-%d %H:%M:%S'
handlers:
console:
class: logging.StreamHandler
level: DEBUG
root:
handlers: [console]
level: WARN
loggers:
app:
level: DEBUG
That configures app at the DEBUG level, and everything else at the WARN level.

Okay, so I found an answer (thanks to #flyx!).
I compared the original python dict config and the converted from yaml dict config and found out that logging automatically adds disable_existing_loggers: False to the python config. After that I added this line to the yaml and used '' in the loggers directive.. and it worked!
So the resulting yaml config is like:
..............
disable_existing_loggers: False
loggers:
'':
handlers: [console, sentry]
level: DEBUG
propagate: False
and now it works. Even if I create a logger like logging.getLogger('something') and logger 'something' is not in the config then the app will use the '' logger.
#Don Kirkby's answer won't work if the logger is defined in the begining of the file(before configurating it). But it works with no problem if the logger is defined after configuring logging. So it's a resolve for the code from my question but not an answer to the question - "So is there a way to create a universal logger in yaml? like the "": in the python logging config?" That's why I didn't pick it as an answer. But he's comment is totally legit :)

Logging over multiple modules

How can I log everything using Python 'logging' to 1 text file, over multiple modules?
Main.py:
import logging
logging.basicConfig(format='localhost - - [%(asctime)s] %(message)s', level=logging.DEBUG)
log_handler = logging.handlers.RotatingFileHandler('debug.out', maxBytes=2048576)
log = logging.getLogger('logger')
log.addHandler(log_handler)
import test
Test.py:
import logging
log = logging.getLogger('logger')
log.error('test')
debug.out stays empty. I'm not sure what to try next, even after reading the logging documentation.
Edit: Fixed with the code above.

Set the correct logging level (at least ERROR if you want to get all messages with level ERROR or higher) and add a handler to write all messages into a file. For more details have a look at https://docs.python.org/2/howto/logging-cookbook.html.

Logging in Python with Config File - Using handlers defined in file through code

I am using logging module of python. How can I access the handlers defined in config file from the code. As an example, I have a logger defined and two handlers - one for screen and other for file. I want to use appropriate handler based on user preference (whether they want to log on screen or to a file). How can I dynamically add and remove handlers defined in config file from loggers defined in config file?
[loggers]
keys=root,netmap
[handlers]
keys=fil,screen
[logger_root]
level=NOTSET
handlers=
[logger_netmap]
level=INFO
handlers=fil,screen
qualname=netmap
[formatters]
keys = simple
[formatter_simple]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=
[handler_fil]
class=handlers.RotatingFileHandler
args=('file.log','a','maxBytes=10000','backupCount=5')
formatter=simple
[handler_screen]
class=StreamHandler
args = (sys.stdout,)
formatter=simple
Depending on whether user runs the program with -v or not I need to use one of File or Screen Handler. How can I add or delete fil or screen handlers from netmap logger?

Instead of having to dynamically change the config file, just use Logger.addHandler(handler).
fileHandler = logging.handlers.RotatingFileHandler('file.log', mode='a', maxBytes=10000, backupCount=5)
logger = logging.getLogger('netmap')
if LOG_TO_FILE:
logger.addHandler(fileHandler)
Then to load in formatting from perhaps the same file;
import ConfigParser
configParser = ConfigParser.ConfigParser()
config.read('config.ini')
format = config.get('formatter_simple', 'format')
fileHandler.setFormatter(format)

From the logging module's documentation it looks like logging objects have these two methods:
Logger.addHandler(hdlr)
Adds the specified handler hdlr to this logger.
Logger.removeHandler(hdlr)
Removes the specified handler hdlr from this logger.
So it seems like you ought be able to use them to add or change the netmap logger handler to be whatever you want based on what's in the config file.

Ok So I found an elegant way through a gud soul on the python google group. It works like charm. Here's the code
import logging
import getopt
import sys
import logging.config
def stop(m,handl):
consoleHand=[h for h in m.handlers if h.__class__ is handl]
print consoleHand
if consoleHand:
h1=consoleHand[0]
h1.filter=lambda x: False
logging.config.fileConfig('log.conf')
my_log = logging.getLogger('netmap')
args=sys.argv[1:]
opt,arg = getopt.gnu_getopt(args,'v')
l=''
for o,a in opt:
if o=='-v':
l='verbose'
stop(my_log,logging.handlers.RotatingFileHandler)
if not l:
stop(my_log,logging.StreamHandler)
my_log.debug('Starting.....')
my_log.warning('Unstable')
my_log.error('FIles Missing')
my_log.critical('BlowOut!!')
The config file is still the same. I can now easily activate or deactivate handlers.