Develop Reference

Logging in python with YAML and filter

Want to set up a logger with a filter using YAML.
YAML configuration file config.yaml is as follows:
version: 1
formatters:
simple:
format: "%(asctime)s %(name)s: %(message)s"
extended:
format: "%(asctime)s %(name)s %(levelname)s: %(message)s"
filters:
noConsoleFilter:
class: noConsoleFilter
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: simple
filters: [noConsoleFilter]
file_handler:
class: logging.FileHandler
level: INFO
filename: test.log
formatter: extended
root:
handlers: [console, file_handler]
propagate: true
...and the main program in main.py as follows:
import logging.config
import yaml
class noConsoleFilter(logging.Filter):
def filter(self, record):
print("filtering!")
return not (record.levelname == 'INFO') & ('no-console' in record.msg)
with open('config.yaml', 'r') as f:
log_cfg = yaml.safe_load(f.read())
logging.config.dictConfig(log_cfg)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.info("no-console. Should not be in console, but be in test.log!")
logger.info('This is an info message')
logger.error('This is an error message')
Expected output in console without the "no-console" message:
2020-04-27 18:05:26,936 __main__: This is an info message
2020-04-27 18:05:26,936 __main__: This is an error message
But looks like class: noConsoleFilter is not even being considered, as the print statement is also not working.
Where am I going wrong? How can I fix it?

The syntax is a bit odd, but it's described in the logging docs, under User-defined objects, that you have to use the key (), not class. Like so:
filters:
noConsoleFilter:
(): noConsoleFilter
Next, you need to specify a qualified name for the class. If you're running the script directly, not as a module, you can refer to it under __main__:
filters:
noConsoleFilter:
(): __main__.noConsoleFilter
I would also recommend using the PEP 8 CapWords convention for class names. Here's a slightly tidied up, fully self-contained example:
# logging.yml
version: 1
formatters:
simple_formatter:
format: "%(asctime)s %(name)s: %(message)s"
extended_formatter:
format: "%(asctime)s %(name)s %(levelname)s: %(message)s"
filters:
no_console_filter:
(): __main__.NoConsoleFilter
handlers:
console_handler:
class: logging.StreamHandler
level: INFO
formatter: simple_formatter
filters: [no_console_filter]
file_handler:
class: logging.FileHandler
level: INFO
filename: test.log
formatter: extended_formatter
root:
handlers: [console_handler, file_handler]
propagate: true
# script.py
import logging.config
import yaml
class NoConsoleFilter(logging.Filter):
def filter(self, record):
print('filtering!')
return not (record.levelname == 'INFO') & ('no-console' in record.msg)
with open('logging.yml', 'r') as f:
log_cfg = yaml.safe_load(f.read())
logging.config.dictConfig(log_cfg)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.info('no-console. Should not be in console, but be in test.log!')
logger.info('This is an info message')
logger.error('This is an error message')

Python3 logging

I'm trying to create my own logger in python 3.6.8 to send output both to stdout and a log file (chosen by date, if the log file doesn't exist yet for today's date it gets created, if there already is a file with the same date just append).
from datetime import date
import logging
import logging.handlers
class Log:
def __init__(self):
pass
def getCleanerLogger(self,moduleName, logFolder, format):
filename = logFolder+ str(date.today()) + '-log.log'
handler = logging.FileHandler(filename)
shandler = logging.StreamHandler()
shandler.setLevel(logging.INFO)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(format)
handler.setFormatter(formatter)
shandler.setFormatter(formatter)
logger = logging.getLogger(moduleName)
logger.addHandler(handler)
logger.addHandler(shandler)
print("I've been called")
return logger
import Conf
conf = Conf.configuration()
print(conf['logFolder'] + " " + conf['logFormat'])
logger = Log()
logger = logger.getCleanerLogger("Log", conf['logFolder'], conf['logFormat'])
logger.info('initializing')
logger.debug('initializing debug')
in the json conf file these are the keys I load
"logFolder": "log/",
"logFormat": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
the log file gets created with the correct logic but there is no logging in either the console or the log file, only the prints go to stdout, no error or exception is raised, I really don't understand why this isn't working. I can only log with logging.root.level('msg') after loading a basiconfig.

Every handler has own logging level but logger has also global logging level which has bigger prioritet so you have to change this level to level which doesn't block handlers - ie.
logger.setLevel(logging.DEBUG)
Mininal working code with few smaller changes.
It doesn't use file with settings so everyone can easily copy it and run it.
from datetime import date
import os
import logging
import logging.handlers
class Log:
def get_cleaner_logger(self, module_name, log_folder, format):
if not os.path.exists(log_folder):
os.makedirs(log_folder)
filename = os.path.join(log_folder, date.today().strftime('%Y-%m-%d-log.log'))
print(filename)
logger = logging.getLogger(module_name)
print('before:', logger.level)
logger.setLevel(logging.DEBUG)
print('after:', logger.level)
formatter = logging.Formatter(format)
handler = logging.FileHandler(filename)
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)
logger.addHandler(handler)
shandler = logging.StreamHandler()
shandler.setLevel(logging.INFO)
shandler.setFormatter(formatter)
logger.addHandler(shandler)
print("I've been called")
return logger
conf = {
"logFolder": "log/",
"logFormat": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
}
logger = Log()
logger = logger.get_cleaner_logger("Log", conf['logFolder'], conf['logFormat'])
logger.info('initializing')
logger.debug('initializing debug')
BTW: I changed some names based on PEP 8 -- Style Guide for Python Code

Changing formatter when using logging module with yaml in python

When using logging module in python, we can initialize general setting for each logger using yaml file. My test code looks like
[main.py]
import yaml
import logging, logging.config
def setup_logging(default_level=logging.DEBUG):
with open("./logging.yaml", 'rt') as f:
configdata = yaml.safe_load(f.read())
logging.config.dictConfig(configdata)
setup_logging()
dbg = logging.getLogger(__name__)
dbg.info("Test")
[logging.yaml]
version: 1
disable_existing_loggers: False
formatters:
detail:
format: "%(asctime)s - %(name)s - %(message)s"
onlymessage:
format: "%(message)s"
handlers:
file_handler:
class: logging.FileHandler
level: DEBUG
formatter: detail
filename: ./log
mode: w
loggers:
__main__:
level: DEBUG
handlers: [file_handler]
propagate: No
So for "file_handler", the default formatter is "detail". Then how do I change the formatter of this logger to another one, in this case "onlymessage"?
I know that if we were given formatter object, we can use Handler.setFormatter() to change the formatter of a logger, like
dbg.handlers[0].setFormatter(FORMATTER NAME)
But since I specified all information about formatter in yaml file and used logging.config when initizliaing logger, I have no formatter object. I think if I can retrieve formatter object written in that yaml file, problem can be solved. Or is there any other way to do this?

Yes, "unused" formatters will be lost in the process if you don't bind them to any handler.
You can check the source of what happens, here is the interesting bit.

Understanding Python logger names

I have named my Python loggers following the practice described in Naming Python loggers
Everything works fine if I use basicConfig(). But now I'm trying to use a configuration file and dictConfig() to configure the loggers at runtime.
The docs at http://docs.python.org/2/library/logging.config.html#dictionary-schema-details seem to say that I can have a "root" key in my dictionary that configures the root logger. But if I configure only this logger, I don't get any output.
Here's what I have:
logging_config.yaml
version: 1
formatters:
simple:
format: '%(asctime)s - %(name)s - %(levelname)s - %(pathname)s:%(lineno)s - %(message)s'
datefmt: '%Y%m%d %H:%M:%S'
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: simple
stream: ext://sys.stdout
file:
class: logging.FileHandler
level: DEBUG
formatter: simple
filename: 'test.log'
mode: "w"
# If I explicitly define a logger for __main__, it works
#loggers:
# __main__:
# level: DEBUG
# handlers: [console, file]
root:
level: DEBUG
handlers: [console, file]
test_log.py
import logging
logger = logging.getLogger(__name__)
import logging.config
import yaml
if __name__ == "__main__":
log_config = yaml.load(open("logging_config.yaml", "r"))
logging.config.dictConfig(log_config)
#logging.basicConfig() #This works, but dictConfig doesn't
logger.critical("OH HAI")
logging.shutdown()
Why doesn't this produce any logging output, and what's the proper way to fix it?

The reason is that you haven't specified disable_existing_loggers: false in your YAML, and the __main__ logger already exists at the time dictConfig is called. So that logger is disabled (because it isn't explicitly named in the configuration - if it is named, then it's not disabled).
Just add that line to your YAML:
version: 1
disable_existing_loggers: false
formatters:
simple:
...

Python logging configuration file

I seem to be having some issues while attempting to implement logging into my python project.
I'm simply attempting to mimic the following configuration:
Python Logging to Multiple Destinations
However instead of doing this inside of code, I'd like to have it in a configuration file.
Below is my config file:
[loggers]
keys=root
[logger_root]
handlers=screen,file
[formatters]
keys=simple,complex
[formatter_simple]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
[formatter_complex]
format=%(asctime)s - %(name)s - %(levelname)s - %(module)s : %(lineno)d - %(message)s
[handlers]
keys=file,screen
[handler_file]
class=handlers.TimedRotatingFileHandler
interval=midnight
backupCount=5
formatter=complex
level=DEBUG
args=('logs/testSuite.log',)
[handler_screen]
class=StreamHandler
formatter=simple
level=INFO
args=(sys.stdout,)
The problem is that my screen output looks like:
2010-12-14 11:39:04,066 - root - WARNING - 3
2010-12-14 11:39:04,066 - root - ERROR - 4
2010-12-14 11:39:04,066 - root - CRITICAL - 5
My file is output, but looks the same as above (although with the extra information included). However the debug and info levels are not output to either.
I am on Python 2.7
Here is my simple example showing failure:
import os
import sys
import logging
import logging.config
sys.path.append(os.path.realpath("shared/"))
sys.path.append(os.path.realpath("tests/"))
class Main(object):
#staticmethod
def main():
logging.config.fileConfig("logging.conf")
logging.debug("1")
logging.info("2")
logging.warn("3")
logging.error("4")
logging.critical("5")
if __name__ == "__main__":
Main.main()

It looks like you've set the levels for your handlers, but not your logger. The logger's level filters every message before it can reach its handlers and the default is WARNING and above (as you can see). Setting the root logger's level to NOTSET as you have, as well as setting it to DEBUG (or whatever is the lowest level you wish to log) should solve your issue.

Adding the following line to the root logger took care of my problem:
level=NOTSET

Just add log level in [logger_root]. It is worked.
[logger_root]
level=DEBUG
handlers=screen,file

A simple approach to both write to terminal and file would be as following:
import logging.config
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("log_file.log"),
logging.StreamHandler()
]
)
logger = logging.getLogger(__name__)
And then use it in your code like this:
logger.info('message')
logger.error('message')

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import logging.handlers
from logging.config import dictConfig
logger = logging.getLogger(__name__)
DEFAULT_LOGGING = {
'version': 1,
'disable_existing_loggers': False,
}
def configure_logging(logfile_path):
"""
Initialize logging defaults for Project.
:param logfile_path: logfile used to the logfile
:type logfile_path: string
This function does:
- Assign INFO and DEBUG level to logger file handler and console handler
"""
dictConfig(DEFAULT_LOGGING)
default_formatter = logging.Formatter(
"[%(asctime)s] [%(levelname)s] [%(name)s] [%(funcName)s():%(lineno)s] [PID:%(process)d TID:%(thread)d] %(message)s",
"%d/%m/%Y %H:%M:%S")
file_handler = logging.handlers.RotatingFileHandler(logfile_path, maxBytes=10485760,backupCount=300, encoding='utf-8')
file_handler.setLevel(logging.INFO)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(default_formatter)
console_handler.setFormatter(default_formatter)
logging.root.setLevel(logging.DEBUG)
logging.root.addHandler(file_handler)
logging.root.addHandler(console_handler)
[31/10/2015 22:00:33] [DEBUG] [yourmodulename] [yourfunction_name():9] [PID:61314 TID:140735248744448] this is logger infomation from hello module
I think you should add the disable_existing_loggers to false.