What I want to do:
To create a Python package, and in that package, create a class that instantiates a logger. I'd create an instance of this class in the main function of my project and whenever I instantiate any package or class, I'd either import the logger class that I created or I'd pass a reference of the instance I created in the main function.
Why I want to do this:
Referential integrity of the logger filename and not having to write
the same lines of code everywhere to instantiate a logger.
To not have the logger as a global.
To be able to perform file operations before instantiating the
logger file. For example, to create a log folder to store the log
files before starting the logger.
To be able to load program parameters from a config file (my config
file manager will be a separate class in a separate package) and to
be able to load logger parameters from the config file.
The code I implemented as of now (which I want to change):
The main file:
import logging
from logging.handlers import RotatingFileHandler
from operatingSystemFunctions import operatingSystemFunc
from diskOperations import fileAndFolderOperations
logFileName = os.path.join('logs', 'logs.log')
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName, maxBytes=2000, backupCount=5)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log.addHandler(handler)
log.setLevel(logging.INFO)
if __name__ == '__main__':
while True:
log.info("Running")
#do something
Similarly, the custom packages I've imported, each have their own global instantiations of logger:
operatingSystemFunc for example, is:
import subprocess
from sys import platform #to check which OS the program is running on
from abc import ABC, abstractmethod
import logging
from logging.handlers import RotatingFileHandler
logFileName = os.path.join('logs', 'logs.log')
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName , maxBytes = 2000 , backupCount = 5)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') #setting this was necessary to get it to write the format to file. Without this, only the message part of the log would get written to the file
log.addHandler(handler)
log.setLevel(logging.INFO)
class AudioNotifier(ABC):#Abstract parent class
#abstractmethod
def __init__(self):
#self.id = "someUniqueNotifierName" #has to be implemented in the child class
pass
Right now the code throws an error FileNotFoundError: [Errno 2] No such file or directory: '/home/nav/prog/logs/logs.log' because I haven't yet created the logs folder.
One concern I have is that if I implement the logger in a separate package and use it by importing that package into all other packages or by passing a reference to an instance of the class to all other classes, then would it be a bad practice?
Update:
Learning from Pablo's answer, I changed the code to ensure that the RotatingFileHandler was simultaneously able to write to a file. So this is the resulting code:
import logging
from logging.handlers import RotatingFileHandler
from operatingSystemFunctions import operatingSystemFunc
from diskOperations import fileAndFolderOperations
logFileName = 'logs.log'
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
#log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName, maxBytes=2000, backupCount=2)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') #setting this was necessary to get it to write the format to file. Without this, only the message part of the log would get written to the file
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)
if __name__ == '__main__':
logging.info("\n\n---------------------------------")
logging.info("program started")
operatingSystemCheck = operatingSystemFunc.OperatingSystemFunc()
logging.info("Monitoring ...")
while True:
pass
and it's used in the packages like this:
import logging
class OperatingSystemFunc:
def __init__(self):
logging.info("Some message")
Few considerations, the python logger is already a singleton that runs in his own thread, so only one instance of the logger is ever made. When you call it, or you try to create it in different places, what you are doing is creating it just the first time and calling the same singleton again and again. So the handlers defined at the first place are accessible everywhere without passing them nor importing them.
If you make a instance of the logger once in your main function using a name, later you will only have to use logging.getLogger("your chosen name"), so in most cases there is no necessity to create a class inside a package etc etc. You can use the handlers, that you have defined elsewhere doing that and without directly importing them.
So I would say that what you are proposing is not a good practice, because it is not necessary, maybe with a few exceptions. If you really want save writing "log = logging.getLogger('...')" which is the only thing you will save, then you can definitely write that line in one package and import the "log" instead the logging module and that will be OK. But then I advice to you not putting the logger inside a class, or if in a class, make the instance in the same file and import the instance, not the class.
its my logger file what i used in my projects:
import logging
class CustomFormatter(logging.Formatter):
grey = "\x1b[38;21m"
green = "\x1b[0;30;42m"
yellow = "\x1b[1;30;43m"
red = "\x1b[0:30;41m"
bold_red = "\x1b[1;30;1m"
reset = "\x1b[0m"
FORMATS = {
logging.DEBUG: f"%(asctime)s - {grey}%(levelname)s{reset} - %(message)s",
logging.INFO: f"%(asctime)s - {green}%(levelname)s{reset} - %(message)s",
logging.WARNING: f"%(asctime)s - {yellow}%(levelname)s{reset} - %(message)s",
logging.ERROR: f"%(asctime)s - {red}%(levelname)s{reset} - %(message)s",
logging.CRITICAL: f"%(asctime)s - {bold_red}%(levelname)s{reset} - %(message)s"
}
def format(self, record):
log_fmt = self.FORMATS.get(record.levelno)
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
logger = logging.getLogger("core")
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(CustomFormatter())
logger.addHandler(ch)
and put it in
_ init _.py
from .logger import logger as LOGGER
and made an ini file like
log.ini
[loggers]
keys=root
[handlers]
keys=logfile,logconsole
[formatters]
keys=logformatter
[logger_root]
level=INFO
handlers=logfile, logconsole
[formatter_logformatter]
format=[%(asctime)s.%(msecs)03d] %(levelname)s [%(thread)d] - %(message)s
[handler_logfile]
class=handlers.RotatingFileHandler
level=INFO
args=('logs.log','a')
formatter=logformatter
[handler_logconsole]
class=handlers.logging.StreamHandler
level=INFO
args=()
formatter=logformatter
these helped me too much, i hope it will help you
I have put the following in my config.py:
import time
import logging
#logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S', level=logging.INFO)
logFormatter = logging.Formatter('%(asctime)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
rootLogger = logging.getLogger()
rootLogger.setLevel(logging.INFO)
fileHandler = logging.FileHandler("{0}.log".format(time.strftime('%Y%m%d%H%M%S')))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
and then I am doing
from config import *
in all of my scripts and imported files.
Unfortunately, this causes multiple log files created.
How to fix this? I wan't centralized config.py with logging configured both to console and file.
Case 1: Independent Scripts / Programs
In case we are talking about multiple, independent scripts, that should have logging set up in the same way: I would say, each independent application should have its own log. If you definitively do not want this, you would have to
make sure that all applications have the same log file name (e.g. create a function in config.py, with a parameter "timestamp", which is provided by your script(s)
specify the append filemode for the fileHandler
make sure that config.py is not called twice somewhere, as you would add the log handlers twice, which would result in each log message being printed twice.
Case 2: One big application consisting of modules
In case we are talking about one big application, consisting of modules, you could adopt a structure like the following:
config.py:
def set_up_logging():
# your logging setup code
module example (some_module.py):
import logging
def some_function():
logger = logging.getLogger(__name__)
[...]
logger.info('sample log')
[...]
main example (main.py)
import logging
from config import set_up_logging
from some_module import some_function
def main():
set_up_logging()
logger = logging.getLogger(__name__)
logger.info('Executing some function')
some_function()
logger.info('Finished')
if __name__ == '__main__':
main()
Explanation:
With the call to set_up_logging() in main() you configure your applications root logger
each module is called from main(), and get its logger via logger = logging.getLogger(__name__). As the modules logger are in the hierarchy below the root logger, those loggings get "propagated up" to the root logger and handled by the handlers of the root logger.
For more information see Pythons logging module doc and/or the logging cookbook
In my programs, I generally want to log to both a file and to the screen. If I import a module, I want the calls to logger within that module's functions/classes to also result in a log to a file and the screen. Test_module_A does this, and test_module_B logs to the file but not the screen. Is this because the loggers in the modules propagate up to the root logger which through basicConfig is only setup to log to the file? I just want to make sure I'm using logging properly, as I am going to rewrite all my code to use this type of logging rather than "print." Going forward, I am only going to use logging and not print, as I figure logging is more flexible. Is this a good idea?
main.py
import logging
logger = logging.getLogger(__name__)
task_file='temp.txt'
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
logging.basicConfig(filename=task_file, format=format_str, datefmt=datefmt_str,
level=logging.INFO)
console = logging.StreamHandler()
#console.setLevel(logging.INFO)
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("from main.py")
import test_module_A
import test_module_B
test_module_A.py
import logging
logger = logging.getLogger(__name__)
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
console = logging.StreamHandler()
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("upon import of test_module_A")
test_module_B.py
import logging
logger = logging.getLogger(__name__)
logger.info("upon import of test_module_B")
Upon running main.py:
#screen output
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
# test.txt
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
05/06/2014 12:36:33AM test_module_B INFO: upon import of test_module_B
It's not good practice to configure logging (add handlers, etc.) in each module: the way to do it is to just have
logger = logging.getLogger(__name__)
and actual logging calls in each module as required, and only have the configuration done in one place (called from the main script early on)
if __name__ == '__main__':
logging.basicConfig(level=..., format=..., ...) #or other configuration code
handler = logging.FileHandler(...)
# set formatters, filters etc.
logging.getLogger().addHandler(handler) # add to root logger
main()
The basicConfig() call adds to the root logger a handler which writes to sys.stderr, and together with the FileHandler added explicitly, the logs from all modules should be written to both screen and file.
I'm trying to add logging (to console rather than a file) to my a piece of code I've been working on for a while. Having read around a bit I have a pattern that I think should work, but I'm not quite sure where I'm going wrong.
I have the following three files (simplified, obviously):
controller.py
import my_module
import logging
from setup_log import configure_log
def main():
logger = configure_log(logging.DEBUG, __name__)
logger.info('Started logging')
my_module.main()
if __name__ == "__main__":
main()
setup_log.py
import logging
def configure_log(level=None, name=None):
logger = logging.getLogger(name)
logger.setLevel(level)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
chFormatter = logging.Formatter('%(levelname)s - %(filename)s - Line: %(lineno)d - %(message)s')
console_handler.setFormatter(chFormatter)
logger.addHandler(console_handler)
return logger
my_module.py
import logging
def main():
logger = logging.getLogger(__name__)
logger.info("Starting my_module")
print "Something"
if __name__ == "__main__":
main()
When I run them, only the first call to logging produces an output to console - 'Started logging'. The second call to logging - 'Starting my module' is just passed over.
What have I misunderstood/mangled?
According to the documentation it looks like you might get away with an even simpler setup like so:
If your program consists of multiple modules, here’s an example of how
you could organize logging in it:
# myapp.py
import logging
import mylib
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
mylib.do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
# mylib.py
import logging
def do_something():
logging.info('Doing something')
If you run myapp.py, you should see this in myapp.log:
INFO:root:Started
INFO:root:Doing something
INFO:root:Finished
It looks like your call to logger = logging.getLogger(__name__) inside your module is creating a separate track (with a level of NOTSET but no parent relationship to result in a log entry)
The actual bug can be seen by putting the line:
print '__name__', __name__
at the beginning of both your mains which yields:
$ python controller.py
__name__ __main__
INFO - controller.py - Line: 8 - Started logging
__name__ my_module
Something
So you properly configured a logger called __main__ but the logger named my_module isn't configured.
The deeper problem is that you have two main methods which is probably confusing you (it did me).
I have a small python project that has the following structure -
Project
-- pkg01
-- test01.py
-- pkg02
-- test02.py
-- logging.conf
I plan to use the default logging module to print messages to stdout and a log file.
To use the logging module, some initialization is required -
import logging.config
logging.config.fileConfig('logging.conf')
logger = logging.getLogger('pyApp')
logger.info('testing')
At present, I perform this initialization in every module before I start logging messages. Is it possible to perform this initialization only once in one place such that the same settings are reused by logging all over the project?
Best practice is, in each module, to have a logger defined like this:
import logging
logger = logging.getLogger(__name__)
near the top of the module, and then in other code in the module do e.g.
logger.debug('My message with %s', 'variable data')
If you need to subdivide logging activity inside a module, use e.g.
loggerA = logging.getLogger(__name__ + '.A')
loggerB = logging.getLogger(__name__ + '.B')
and log to loggerA and loggerB as appropriate.
In your main program or programs, do e.g.:
def main():
"your program code"
if __name__ == '__main__':
import logging.config
logging.config.fileConfig('/path/to/logging.conf')
main()
or
def main():
import logging.config
logging.config.fileConfig('/path/to/logging.conf')
# your program code
if __name__ == '__main__':
main()
See here for logging from multiple modules, and here for logging configuration for code which will be used as a library module by other code.
Update: When calling fileConfig(), you may want to specify disable_existing_loggers=False if you're using Python 2.6 or later (see the docs for more information). The default value is True for backward compatibility, which causes all existing loggers to be disabled by fileConfig() unless they or their ancestor are explicitly named in the configuration. With the value set to False, existing loggers are left alone. If using Python 2.7/Python 3.2 or later, you may wish to consider the dictConfig() API which is better than fileConfig() as it gives more control over the configuration.
Actually every logger is a child of the parent's package logger (i.e. package.subpackage.module inherits configuration from package.subpackage), so all you need to do is just to configure the root logger. This can be achieved by logging.config.fileConfig (your own config for loggers) or logging.basicConfig (sets the root logger). Setup logging in your entry module (__main__.py or whatever you want to run, for example main_script.py. __init__.py works as well)
using basicConfig:
# package/__main__.py
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
using fileConfig:
# package/__main__.py
import logging
import logging.config
logging.config.fileConfig('logging.conf')
and then create every logger using:
# package/submodule.py
# or
# package/subpackage/submodule.py
import logging
log = logging.getLogger(__name__)
log.info("Hello logging!")
For more information see Advanced Logging Tutorial.
A simple way of using one instance of logging library in multiple modules for me was following solution:
base_logger.py
import logging
logger = logging
logger.basicConfig(format='%(asctime)s - %(message)s', level=logging.INFO)
Other files
from base_logger import logger
if __name__ == '__main__':
logger.info("This is an info message")
I always do it as below.
Use a single python file to config my log as singleton pattern which named 'log_conf.py'
#-*-coding:utf-8-*-
import logging.config
def singleton(cls):
instances = {}
def get_instance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return get_instance()
#singleton
class Logger():
def __init__(self):
logging.config.fileConfig('logging.conf')
self.logr = logging.getLogger('root')
In another module, just import the config.
from log_conf import Logger
Logger.logr.info("Hello World")
This is a singleton pattern to log, simply and efficiently.
Throwing in another solution.
In my module's init.py I have something like:
# mymodule/__init__.py
import logging
def get_module_logger(mod_name):
logger = logging.getLogger(mod_name)
handler = logging.StreamHandler()
formatter = logging.Formatter(
'%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
return logger
Then in each module I need a logger, I do:
# mymodule/foo.py
from [modname] import get_module_logger
logger = get_module_logger(__name__)
When the logs are missed, you can differentiate their source by the module they came from.
Several of these answers suggest that at the top of a module you you do
import logging
logger = logging.getLogger(__name__)
It is my understanding that this is considered very bad practice. The reason is that the file config will disable all existing loggers by default. E.g.
#my_module
import logging
logger = logging.getLogger(__name__)
def foo():
logger.info('Hi, foo')
class Bar(object):
def bar(self):
logger.info('Hi, bar')
And in your main module :
#main
import logging
# load my module - this now configures the logger
import my_module
# This will now disable the logger in my module by default, [see the docs][1]
logging.config.fileConfig('logging.ini')
my_module.foo()
bar = my_module.Bar()
bar.bar()
Now the log specified in logging.ini will be empty, as the existing logger was disabled by fileconfig call.
While is is certainly possible to get around this (disable_existing_Loggers=False), realistically many clients of your library will not know about this behavior, and will not receive your logs. Make it easy for your clients by always calling logging.getLogger locally. Hat Tip : I learned about this behavior from Victor Lin's Website.
So good practice is instead to always call logging.getLogger locally. E.g.
#my_module
import logging
logger = logging.getLogger(__name__)
def foo():
logging.getLogger(__name__).info('Hi, foo')
class Bar(object):
def bar(self):
logging.getLogger(__name__).info('Hi, bar')
Also, if you use fileconfig in your main, set disable_existing_loggers=False, just in case your library designers use module level logger instances.
I would like to add my solution (which is based on logging cookbook and other articles and suggestions from this thread. However it took me quite a while to figure out, why it wasn't immediately working how I expected. So I created a little test project to learn how logging is working.
Since I have figured it out, I wanted to share my solution, maybe it can be of help to someone.
I know some of my code might not be best practice, but I am still learning. I left the print() functions in there, as I used them, while logging was not working as expected. Those are removed in my other application. Also I welcome any feedback on any parts of the code or structure.
my_log_test project structure (cloned/simplified from another project I work on)
my_log_test
├── __init__.py
├── __main__.py
├── daemon.py
├── common
│ ├── my_logger.py
├── pkg1
│ ├── __init__.py
│ └── mod1.py
└── pkg2
├── __init__.py
└── mod2.py
Requirements
A few things different or that I have not seen explicitly mentioned in the combination I use:
the main module is daemon.pywhich is called by __main__.py
I want to be able to call the modules mod1.py and mod2.py seperately while in development/testing
At this point I did not want to use basicConfig() or FileConfig() but keep it like in the logging cookbook
So basically, that means, I need to initialize the root logger in daemon.py (always) and in the modules mod1.py and mod2.py (only when calling them directly).
To make this init in several modules easier, I created my_logger.py which does, what is described in the cookbook.
My mistakes
Beforehand, my mistake in that module was to init the logger with logger = logging.getLogger(__name__) (module logger) instead of using logger = logging.getLogger() (to get the root logger).
The first problem was, that when called from daemon.py the logger's namespace was set to my_log_test.common.my_logger. The module logger in mod1.py with an "unmatching" namespace my_log_test.pkg1.mod1 could hence not attach to the other logger and I would see no log output from mod1.
The second "problem" was, that my main program is in daemon.py and not in __main__.py. But after all not a real problem for me, but it added to the namespace confusion.
Working solution
This is from the cookbook but in a separate module. I also added a logger_cleanup function that I can call from daemon, to remove logs older than x days.
## my_logger.py
from datetime import datetime
import time
import os
## Init logging start
import logging
import logging.handlers
def logger_init():
print("print in my_logger.logger_init()")
print("print my_logger.py __name__: " +__name__)
path = "log/"
filename = "my_log_test.log"
## get logger
#logger = logging.getLogger(__name__) ## this was my mistake, to init a module logger here
logger = logging.getLogger() ## root logger
logger.setLevel(logging.INFO)
# File handler
logfilename = datetime.now().strftime("%Y%m%d_%H%M%S") + f"_{filename}"
file = logging.handlers.TimedRotatingFileHandler(f"{path}{logfilename}", when="midnight", interval=1)
#fileformat = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s")
fileformat = logging.Formatter("%(asctime)s [%(levelname)s]: %(name)s: %(message)s")
file.setLevel(logging.INFO)
file.setFormatter(fileformat)
# Stream handler
stream = logging.StreamHandler()
#streamformat = logging.Formatter("%(asctime)s [%(levelname)s:%(module)s] %(message)s")
streamformat = logging.Formatter("%(asctime)s [%(levelname)s]: %(name)s: %(message)s")
stream.setLevel(logging.INFO)
stream.setFormatter(streamformat)
# Adding all handlers to the logs
logger.addHandler(file)
logger.addHandler(stream)
def logger_cleanup(path, days_to_keep):
lclogger = logging.getLogger(__name__)
logpath = f"{path}"
now = time.time()
for filename in os.listdir(logpath):
filestamp = os.stat(os.path.join(logpath, filename)).st_mtime
filecompare = now - days_to_keep * 86400
if filestamp < filecompare:
lclogger.info("Delete old log " + filename)
try:
os.remove(os.path.join(logpath, filename))
except Exception as e:
lclogger.exception(e)
continue
to run deamon.py (through __main__.py) use python3 -m my_log_test
## __main__.py
from my_log_test import daemon
if __name__ == '__main__':
print("print in __main__.py")
daemon.run()
to run deamon.py (directly) use python3 -m my_log_test.daemon
## daemon.py
from datetime import datetime
import time
import logging
import my_log_test.pkg1.mod1 as mod1
import my_log_test.pkg2.mod2 as mod2
## init ROOT logger from my_logger.logger_init()
from my_log_test.common.my_logger import logger_init
logger_init() ## init root logger
logger = logging.getLogger(__name__) ## module logger
def run():
print("print in daemon.run()")
print("print daemon.py __name__: " +__name__)
logger.info("Start daemon")
loop_count = 1
while True:
logger.info(f"loop_count: {loop_count}")
logger.info("do stuff from pkg1")
mod1.do1()
logger.info("finished stuff from pkg1")
logger.info("do stuff from pkg2")
mod2.do2()
logger.info("finished stuff from pkg2")
logger.info("Waiting a bit...")
time.sleep(30)
if __name__ == '__main__':
try:
print("print in daemon.py if __name__ == '__main__'")
logger.info("running daemon.py as main")
run()
except KeyboardInterrupt as e:
logger.info("Program aborted by user")
except Exception as e:
logger.info(e)
To run mod1.py (directly) use python3 -m my_log_test.pkg1.mod1
## mod1.py
import logging
# mod1_logger = logging.getLogger(__name__)
mod1_logger = logging.getLogger("my_log_test.daemon.pkg1.mod1") ## for testing, namespace set manually
def do1():
print("print in mod1.do1()")
print("print mod1.py __name__: " +__name__)
mod1_logger.info("Doing someting in pkg1.do1()")
if __name__ == '__main__':
## Also enable this pkg to be run directly while in development with
## python3 -m my_log_test.pkg1.mod1
## init root logger
from my_log_test.common.my_logger import logger_init
logger_init() ## init root logger
print("print in mod1.py if __name__ == '__main__'")
mod1_logger.info("Running mod1.py as main")
do1()
To run mod2.py (directly) use python3 -m my_log_test.pkg2.mod2
## mod2.py
import logging
logger = logging.getLogger(__name__)
def do2():
print("print in pkg2.do2()")
print("print mod2.py __name__: " +__name__) # setting namespace through __name__
logger.info("Doing someting in pkg2.do2()")
if __name__ == '__main__':
## Also enable this pkg to be run directly while in development with
## python3 -m my_log_test.pkg2.mod2
## init root logger
from my_log_test.common.my_logger import logger_init
logger_init() ## init root logger
print("print in mod2.py if __name__ == '__main__'")
logger.info("Running mod2.py as main")
do2()
Happy if it helps. Happy to receive feedback as well!
You could also come up with something like this!
def get_logger(name=None):
default = "__app__"
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
log_map = {"__app__": "app.log", "__basic_log__": "file1.log", "__advance_log__": "file2.log"}
if name:
logger = logging.getLogger(name)
else:
logger = logging.getLogger(default)
fh = logging.FileHandler(log_map[name])
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.setLevel(logging.DEBUG)
return logger
Now you could use multiple loggers in same module and across whole project if the above is defined in a separate module and imported in other modules were logging is required.
a=get_logger("__app___")
b=get_logger("__basic_log__")
a.info("Starting logging!")
b.debug("Debug Mode")
#Yarkee's solution seemed better. I would like to add somemore to it -
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances.keys():
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
class LoggerManager(object):
__metaclass__ = Singleton
_loggers = {}
def __init__(self, *args, **kwargs):
pass
#staticmethod
def getLogger(name=None):
if not name:
logging.basicConfig()
return logging.getLogger()
elif name not in LoggerManager._loggers.keys():
logging.basicConfig()
LoggerManager._loggers[name] = logging.getLogger(str(name))
return LoggerManager._loggers[name]
log=LoggerManager().getLogger("Hello")
log.setLevel(level=logging.DEBUG)
So LoggerManager can be a pluggable to the entire application.
Hope it makes sense and value.
There are several answers. i ended up with a similar yet different solution that makes sense to me, maybe it will make sense to you as well.
My main objective was to be able to pass logs to handlers by their level (debug level logs to the console, warnings and above to files):
from flask import Flask
import logging
from logging.handlers import RotatingFileHandler
app = Flask(__name__)
# make default logger output everything to the console
logging.basicConfig(level=logging.DEBUG)
rotating_file_handler = RotatingFileHandler(filename="logs.log")
rotating_file_handler.setLevel(logging.INFO)
app.logger.addHandler(rotating_file_handler)
created a nice util file named logger.py:
import logging
def get_logger(name):
return logging.getLogger("flask.app." + name)
the flask.app is a hardcoded value in flask. the application logger is always starting with flask.app as its the module's name.
now, in each module, i'm able to use it in the following mode:
from logger import get_logger
logger = get_logger(__name__)
logger.info("new log")
This will create a new log for "app.flask.MODULE_NAME" with minimum effort.
The best practice would be to create a module separately which has only one method whose task we be to give a logger handler to the the calling method. Save this file as m_logger.py
import logger, logging
def getlogger():
# logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
#ch = logging.StreamHandler()
ch = logging.FileHandler(r'log.txt')
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
return logger
Now call the getlogger() method whenever logger handler is needed.
from m_logger import getlogger
logger = getlogger()
logger.info('My mssg')
New to python so I don't know if this is advisable, but it works great for not re-writing boilerplate.
Your project must have an init.py so it can be loaded as a module
# Put this in your module's __init__.py
import logging.config
import sys
# I used this dictionary test, you would put:
# logging.config.fileConfig('logging.conf')
# The "" entry in loggers is the root logger, tutorials always
# use "root" but I can't get that to work
logging.config.dictConfig({
"version": 1,
"formatters": {
"default": {
"format": "%(asctime)s %(levelname)s %(name)s %(message)s"
},
},
"handlers": {
"console": {
"level": 'DEBUG',
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout"
}
},
"loggers": {
"": {
"level": "DEBUG",
"handlers": ["console"]
}
}
})
def logger():
# Get the name from the caller of this function
return logging.getLogger(sys._getframe(1).f_globals['__name__'])
sys._getframe(1) suggestion comes from here
Then to use your logger in any other file:
from [your module name here] import logger
logger().debug("FOOOOOOOOO!!!")
Caveats:
You must run your files as modules, otherwise import [your module] won't work:
python -m [your module name].[your filename without .py]
The name of the logger for the entry point of your program will be __main__, but any solution using __name__ will have that issue.