Develop Reference

Is it possible/advisable to put a logger into a single package and use it across all packages in a project?

What I want to do:
To create a Python package, and in that package, create a class that instantiates a logger. I'd create an instance of this class in the main function of my project and whenever I instantiate any package or class, I'd either import the logger class that I created or I'd pass a reference of the instance I created in the main function.
Why I want to do this:
Referential integrity of the logger filename and not having to write
the same lines of code everywhere to instantiate a logger.
To not have the logger as a global.
To be able to perform file operations before instantiating the
logger file. For example, to create a log folder to store the log
files before starting the logger.
To be able to load program parameters from a config file (my config
file manager will be a separate class in a separate package) and to
be able to load logger parameters from the config file.
The code I implemented as of now (which I want to change):
The main file:
import logging
from logging.handlers import RotatingFileHandler
from operatingSystemFunctions import operatingSystemFunc
from diskOperations import fileAndFolderOperations
logFileName = os.path.join('logs', 'logs.log')
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName, maxBytes=2000, backupCount=5)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log.addHandler(handler)
log.setLevel(logging.INFO)
if __name__ == '__main__':
while True:
log.info("Running")
#do something
Similarly, the custom packages I've imported, each have their own global instantiations of logger:
operatingSystemFunc for example, is:
import subprocess
from sys import platform #to check which OS the program is running on
from abc import ABC, abstractmethod
import logging
from logging.handlers import RotatingFileHandler
logFileName = os.path.join('logs', 'logs.log')
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName , maxBytes = 2000 , backupCount = 5)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') #setting this was necessary to get it to write the format to file. Without this, only the message part of the log would get written to the file
log.addHandler(handler)
log.setLevel(logging.INFO)
class AudioNotifier(ABC):#Abstract parent class
#abstractmethod
def __init__(self):
#self.id = "someUniqueNotifierName" #has to be implemented in the child class
pass
Right now the code throws an error FileNotFoundError: [Errno 2] No such file or directory: '/home/nav/prog/logs/logs.log' because I haven't yet created the logs folder.
One concern I have is that if I implement the logger in a separate package and use it by importing that package into all other packages or by passing a reference to an instance of the class to all other classes, then would it be a bad practice?
Update:
Learning from Pablo's answer, I changed the code to ensure that the RotatingFileHandler was simultaneously able to write to a file. So this is the resulting code:
import logging
from logging.handlers import RotatingFileHandler
from operatingSystemFunctions import operatingSystemFunc
from diskOperations import fileAndFolderOperations
logFileName = 'logs.log'
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
#log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName, maxBytes=2000, backupCount=2)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') #setting this was necessary to get it to write the format to file. Without this, only the message part of the log would get written to the file
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)
if __name__ == '__main__':
logging.info("\n\n---------------------------------")
logging.info("program started")
operatingSystemCheck = operatingSystemFunc.OperatingSystemFunc()
logging.info("Monitoring ...")
while True:
pass
and it's used in the packages like this:
import logging
class OperatingSystemFunc:
def __init__(self):
logging.info("Some message")

Few considerations, the python logger is already a singleton that runs in his own thread, so only one instance of the logger is ever made. When you call it, or you try to create it in different places, what you are doing is creating it just the first time and calling the same singleton again and again. So the handlers defined at the first place are accessible everywhere without passing them nor importing them.
If you make a instance of the logger once in your main function using a name, later you will only have to use logging.getLogger("your chosen name"), so in most cases there is no necessity to create a class inside a package etc etc. You can use the handlers, that you have defined elsewhere doing that and without directly importing them.
So I would say that what you are proposing is not a good practice, because it is not necessary, maybe with a few exceptions. If you really want save writing "log = logging.getLogger('...')" which is the only thing you will save, then you can definitely write that line in one package and import the "log" instead the logging module and that will be OK. But then I advice to you not putting the logger inside a class, or if in a class, make the instance in the same file and import the instance, not the class.

its my logger file what i used in my projects:
import logging
class CustomFormatter(logging.Formatter):
grey = "\x1b[38;21m"
green = "\x1b[0;30;42m"
yellow = "\x1b[1;30;43m"
red = "\x1b[0:30;41m"
bold_red = "\x1b[1;30;1m"
reset = "\x1b[0m"
FORMATS = {
logging.DEBUG: f"%(asctime)s - {grey}%(levelname)s{reset} - %(message)s",
logging.INFO: f"%(asctime)s - {green}%(levelname)s{reset} - %(message)s",
logging.WARNING: f"%(asctime)s - {yellow}%(levelname)s{reset} - %(message)s",
logging.ERROR: f"%(asctime)s - {red}%(levelname)s{reset} - %(message)s",
logging.CRITICAL: f"%(asctime)s - {bold_red}%(levelname)s{reset} - %(message)s"
}
def format(self, record):
log_fmt = self.FORMATS.get(record.levelno)
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
logger = logging.getLogger("core")
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(CustomFormatter())
logger.addHandler(ch)
and put it in
_ init _.py
from .logger import logger as LOGGER
and made an ini file like
log.ini
[loggers]
keys=root
[handlers]
keys=logfile,logconsole
[formatters]
keys=logformatter
[logger_root]
level=INFO
handlers=logfile, logconsole
[formatter_logformatter]
format=[%(asctime)s.%(msecs)03d] %(levelname)s [%(thread)d] - %(message)s
[handler_logfile]
class=handlers.RotatingFileHandler
level=INFO
args=('logs.log','a')
formatter=logformatter
[handler_logconsole]
class=handlers.logging.StreamHandler
level=INFO
args=()
formatter=logformatter
these helped me too much, i hope it will help you

Logger not outputting INFO messages after specifying filename in basicConfig

I'm using a logging object to display the process of my Python script. Things were working fine until I specified a log file.
Working Code
from datetime import timedelta
import logging
import time
def main():
start_time = time.monotonic()
config = get_args()
# Logger configuration.
msg_format = '[%(asctime)s - %(levelname)s - %(filename)s: %(lineno)d (%(funcName)s)] %(message)s'
logging.basicConfig(format=msg_format, level=logging.INFO)
logger = logging.getLogger()
logger.info(msg='Starting process...')
# End of process.
end_time = time.monotonic()
process_time = str(timedelta(seconds=(end_time - start_time))).split('.')[0] # Get rid of trailing microseconds.
logger.info(msg=('End. Entire process took approximately %s' % process_time))
if __name__ == '__main__':
main()
Not Working Code
from datetime import datetime, timedelta
import logging
import os
import time
def main():
start_time = time.monotonic()
config = get_args()
# Logger configuration.
timestamp = datetime.now().strftime(format='%Y%m%d-%H%M')
log_filename = '_'.join(['log', timestamp])
log_file_path = os.path.join(config.log_dir, log_filename)
msg_format = '[%(asctime)s - %(levelname)s - %(filename)s: %(lineno)d (%(funcName)s)] %(message)s'
logging.basicConfig(filename=log_file_path, format=msg_format, level=logging.INFO)
logger = logging.getLogger()
logger.info(msg='Starting process...')
# End of process.
end_time = time.monotonic()
process_time = str(timedelta(seconds=(end_time - start_time))).split('.')[0] # Get rid of trailing microseconds.
logger.info(msg=('End. Entire process took approximately %s' % process_time))
if __name__ == '__main__':
main()
Would anyone know what might be the problem? I'm suspecting that specifying the log filename in the logging.basicConfig may have messed up the handlers or something, but I'm not 100% certain. Thanks!

The answer was relatively obvious after reading the documentation a bit more carefully. I wasn't aware that I hadn't included any handlers in my configuration. To fix the issue, all I had to do was change:
logging.basicConfig(filename=log_file_path, format=msg_format, level=logging.INFO)
to this:
logging.basicConfig(format=msg_format, level=logging.INFO, \
handlers=[logging.FileHandler(filename=log_file_path), logging.StreamHandler()])
Hope this helps anyone else running into the same mistake.
This answer helped me: logger configuration to log to file and print ot stdout

Why is my configured logger not being used?

Reading the logging HOWTO (https://docs.python.org/3/howto/logging.html) I came away under the impression that if I configured a logger, then I could subsequently request my logger from the factory via logging.getLogger() and python would know how to get the right logger (the one I configured) and everything would just auto-work, i.e. I wouldn't need to pass the configured logger instance around my code, I could just ask for it wherever I needed it. Instead, I'm observing something different.
File log_tester.py:
from util.logging_custom import SetupLogger
import logging
import datetime
def test():
logger = logging.getLogger()
logger.debug("In test()")
def main():
logger = SetupLogger("logger_test")
logger.setLevel(logging.DEBUG)
logger.info(f"now is {datetime.datetime.now()}", )
logger.debug("In main()")
test()
if __name__ == '__main__':
main()
File util/logging_custom.py:
import os
import time
import logging
from logging.handlers import RotatingFileHandler
def SetupLogger(name_prefix):
if not os.path.exists("log"):
os.makedirs("log")
recfmt = logging.Formatter('%(asctime)s.%(msecs)03d %(levelname)s %(message)s')
handler = RotatingFileHandler(time.strftime(f"log/{name_prefix}.log"),maxBytes=5000000, backupCount=10)
handler.setFormatter(recfmt)
handler.setLevel(logging.DEBUG)
logger = logging.getLogger(f"{name_prefix} {__name__}")
logger.addHandler(handler)
return logger
When I run this code only the debug statement that is in main() ends up in the log file. The debug statement from test() ends up I'm not sure where exactly.
Contents of log/logger_test.log:
2019-02-07 09:14:39,906.906 INFO now is 2019-02-07 09:14:39.906848
2019-02-07 09:14:39,906.906 DEBUG In main()
My expectation was that In test() would also show up in my log file. Have I made some assumptions about how python logging works that are untrue? How do I make it so that all of the logging in my program (which has many classes and modules) goes to the same configured logger? Is that possible without passing around a logger instance everywhere, after it's created in main()?
Thanks.

The getLogger function will return a the logger by its name (kind of a singleton):
if it doesn't exist, it creates it
If it already exist, it returns it
Then what you could do is:
util/logging_custom.py
def SetupLogger(logger_name, level=logging.INFO):
if not os.path.exists("log"):
os.makedirs("log")
recfmt = logging.Formatter('%(asctime)s.%(msecs)03d %(levelname)s %(message)s')
handler = RotatingFileHandler(time.strftime(f"log/{logger_name}.log"),maxBytes=5000000, backupCount=10)
handler.setFormatter(recfmt)
handler.setLevel(level)
logger = logging.getLogger(logger_name)
logger.addHandler(handler)
# no need to return the logger, I would even advice not to do so
log_tester.py
from util.logging_custom import SetupLogger
import logging
import datetime
logger = SetupLogger("logger_test", logging.DEBUG) # you only need to run this once, in your main script.
logger = logging.getLogger("logger_test")
def test():
logger.debug("In test()")
def main():
logger.info(f"now is {datetime.datetime.now()}", )
logger.debug("In main()")
test()
if __name__ == '__main__':
main()
any_other.py
import logging
logger = logging.getLogger("logger_test") # this will return the logger you already instantiate in log_tester.py
logger.info("that works!")
Update
To set the level and the handling of the root logger instead of the one you setted up, use logging.getLogger() without passing any name:
root_logger = logging.getLogger()
root_logger.addHandler(your_handler)
root_logger.setLevel(logging.DEBUG)
root_logger.info("hello world")

From the docs:
Multiple calls to getLogger() with the same name will return a
reference to the same logger object.
Your assumptions are quite correct. The problem here is the way you are calling getLogger() in test(). You should be passing the name you used in SetupLogger()'s getLogger() i.e. logger = logging.getLogger(f"{name_prefix} {__name__}").

selectively setting the console logging level

I am trying to use an FTP server stub during tests. I don't want the console output, but I would like to capture the logging to a file.
I want the FTP server to run in a different process, so I use multiprocessing.
My code as follows sets all logging to level WARNING:
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
import pyftpdlib.log as pyftpdliblog
import os
import logging
import multiprocessing as mp
authorizer = DummyAuthorizer()
authorizer.add_user('user', '12345', '.', perm='elradfmwM')
handler = FTPHandler
handler.authorizer = authorizer
pyftpdliblog.LEVEL = logging.WARNING
logging.basicConfig(filename='pyftpd.log', level=logging.INFO)
server = FTPServer(('', 2121), handler)
def main():
p = mp.Process(target=server.serve_forever)
p.start()
if __name__ == '__main__':
main()
How do I set only the console logging to level WARNING, or even better, completely shutdown without giving up the file logging?

So, after digging inside the code, I found out the following hint:
# This is a method of FTPServer and it is called before
# server.serve_forever
def _log_start(self):
if not logging.getLogger('pyftpdlib').handlers:
# If we get to this point it means the user hasn't
# configured logger. We want to log by default so
# we configure logging ourselves so that it will
# print to stderr.
from pyftpdlib.ioloop import _config_logging
_config_logging()
So, all I had to do is to define my own appropriate handlers:
logger = logging.getLogger('pyftpdlib')
logger.setLevel(logging.INFO)
hdlr = logging.FileHandler('pyftpd.log' )
logger.addHandler(hdlr)
Now, there is file logging, but console logging will not start.

Something like this:
import logging
date_format = "%Y/%m/%d %H:%M:%S"
log_file_path = "my_file.txt"
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
# own_module_logger = logging.getLogger(__name__)
pyftpdlib_logger = logging.getLogger("pyftpdlib")
# Setup logging to file (Only pyftpdlib)
filehandler = logging.FileHandler(filename = log_file_path)
filehandler.setLevel(logging.DEBUG)
fileformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
filehandler.setFormatter(fileformatter)
pyftpdlib_logger.addHandler(filehandler)
pyftpdlib_logger.propagate = False
# Setup logging to console (All other)
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
console.setFormatter(consoleformatter)
root_logger.addHandler(console)
# Do your loggings
a = logging.getLogger()
a.info('root I')
a.debug('root D')
b = logging.getLogger("pyftpdlib")
b.info('P I')
b.debug('P D')
logging.shutdown()
So loggings of pyftpdlib go to file. Everything from your module to console. One of the key thing here is the propagate!

python's logging module is not logging to a log file

I am trying out python's logger module to log info both in console and in a log and it works fine.I wanted that script to get invoked during machine star-tup and hence invoked my script from rc.local.
And my script gets invoked during boot-up, but the problem is, only console logging is working at that time. Logging to a file is not happening. Any clue on what could be the issue.
Is it anything related to syslog daemon?
import time
import logging
import datetime
global logger
global LOG_FILE
def initialize():
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
time = datetime.datetime.now()
LOG_FILE = "sample.log." + time.strftime("%Y%m%d")
log_file_handler = logging.FileHandler(LOG_FILE)
log_file_handler.setLevel(logging.DEBUG)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
log_file_handler = logging.StreamHandler()
log_file_handler.setLevel(logging.INFO)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
if __name__ == '__main__':
initialize()
logger.info("This should go both in console and log")
logger.debug("This should go only to Log")

import time
import logging
import datetime
def initialize():
# because logging is global module, so you can
# always get logger by logging.getLogger(__name__)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
time = datetime.datetime.now()
# if you only use LOG_FILE in function, you don't
# have to declare it outside.
LOG_FILE = "sample.log." + time.strftime("%Y%m%d")
log_file_handler = logging.FileHandler(LOG_FILE)
log_file_handler.setLevel(logging.DEBUG)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
log_file_handler = logging.StreamHandler()
log_file_handler.setLevel(logging.INFO)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
if __name__ == '__main__':
initialize()
logger = logging.getLogger(__name__)
logger.info("This should go both in console and log")
logger.debug("This should go only to Log")