I want to log to a single log file from main and all sub modules.
The log messages send from a main file, where I define the logger, work as expected. But the ones send from a call to an imported function are missing.
It is working if I use logging.basicConfig as in Example 1 below.
But the second example which allows for more custom settings does not work.
Any ideas why?
# in the submodule I have this code
import logging
logger = logging.getLogger(__name__)
EXAMPLE 1 - Working
Here I create two handlers and just pass them to basicConfig:
# definition of root looger in main module
formatter = logging.Formatter(fmt="%(asctime)s %(name)s.%(levelname)s: %(message)s", datefmt="%Y.%m.%d %H:%M:%S")
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
handler2 = logging.StreamHandler(stream=None)
handler2.setFormatter(formatter)
handler2.setLevel(logging.DEBUG)
logging.basicConfig(handlers=[handler, handler2], level=logging.DEBUG)
logger = logging.getLogger(__name__)
EXAMPLE 2 - Not working
Here I create two handlers and addHandler() them to the root logger:
# definition of root looger in main module
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
#handler.setLevel(logging.ERROR)
logger.addHandler(handler)
handler = logging.StreamHandler(stream=None)
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
You need to configure the (one and only) root logger in the main module of your software. This is done by calling
logger = logging.getLogger() #without arguments
instead of
logger = logging.getLogger(__name__)
(Python doc on logging)
The second example creates a separate, child logger with the name of your script.
If there are no handlers defined in the submodules, the log message is being passed down to the root logger to handle it.
A related question can be found here:
Python Logging - How to inherit root logger level & handler
Related
I have two file entrypoint.py and op_helper.py that I am trying to send each scripts logs to different log files (webhook.log & op.log). I set up my logger.py file with two different log classes.
import logging
from logging.handlers import TimedRotatingFileHandler
class Logger:
def create_timed_rotating_log(self, path):
logger = logging.getLogger("Rotating Log")
logger.setLevel(logging.INFO)
handler = TimedRotatingFileHandler(path,
when="d",
interval=1,
backupCount=7)
formatter = logging.Formatter(fmt='%(asctime)s %(levelname)-8s %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
class WebhookLogger:
def create_timed_rotating_log(self, path):
logger = logging.getLogger("Rotating Log")
logger.setLevel(logging.INFO)
handler = TimedRotatingFileHandler(path,
when="d",
interval=1,
backupCount=7)
formatter = logging.Formatter(fmt='%(asctime)s %(levelname)-8s %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
today = datetime.today()
month = today.strftime("%B")
logger = Logger().create_timed_rotating_log(f'./{month + str(today.year)}Logger.log')
webhook_logger = WebhookLogger().create_timed_rotating_log(f'./{month + str(today.year)}WebhookLogger.log')
In my entrypoint.py script:
from logger import webhook_logger
webhook_logger.info("Something to log")
And in my op_helper.py script:
from logger import logger
logger.info("Something else to log")
But when I run the script, both log statements are logged to both log files.
2021-10-15 14:17:51 INFO Something to log
2021-10-15 14:17:51 INFO Something else to log
Can anyone explain to me what's going on here, and possibly, what I'm doing incorrectly?
Thank you in advance!
Here is an excerpt from the documentation for logging (the bold is mine):
logging.getLogger(name=None)
Return a logger with the specified name or, if name is None, return a logger which is the root logger of the hierarchy. If specified, the name is typically a dot-separated hierarchical name like ‘a’, ‘a.b’ or ‘a.b.c.d’. Choice of these names is entirely up to the developer who is using logging.
All calls to this function with a given name return the same logger instance. This means that logger instances never need to be passed between different parts of an application.
...
The solution, therefore, is to assign a different name to your second logger.
EDIT:
Keep in mind, however, that, as you can see, calling getLogger either creates a new instance, if one under the given name doesn't exist, or returns an already existing instance. Therefore every following instruction will only modify an existing logger. If your intention is to use your classes to create multiple instances of one logger type, that approach will not work. Right now, they both do exactly the same thing, so there's not really a need for two separate classes either. As you can see, logging doesn't lend itself well to being used with an object-oriented approach, because the objects are already instanced elsewhere and can be accessed as "global" objects. But this is all just a side note.
How can I change root logger level in it's submodule
#main.py
logging.basicConfig(filename=filename,format='%(asctime)s %(levelname)s %(message)s',filemode='w')
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
After certain event I want to change all the logging to CRITICAL
sub_logger = logging.getLogger('jdm_health')
#submodule.py
if event:
logger.setLevel(logging.CRITICAL)#root logger
I am having some difficulties using python's logging. I have two files, main.py and mymodule.py. Generally main.py is run, and it will import mymodule.py and use some functions from there. But sometimes, I will run mymodule.py directly.
I tried to make it so that logging is configured in only 1 location, but something seems wrong.
Here is the code.
# main.py
import logging
import mymodule
logger = logging.getLogger(__name__)
def setup_logging():
# only cofnigure logger if script is main module
# configuring logger in multiple places is bad
# only top-level module should configure logger
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s', datefmt = '%Y-%m-%d %H:%M:%S')
ch.setFormatter(formatter)
logger.addHandler(ch)
if __name__ == '__main__':
setup_logging()
logger.info('calling mymodule.myfunc()')
mymodule.myfunc()
and the imported module:
# mymodule.py
import logging
logger = logging.getLogger(__name__)
def myfunc():
msg = 'myfunc is being called'
logger.info(msg)
print('done with myfunc')
if __name__ == '__main__':
# only cofnigure logger if script is main module
# configuring logger in multiple places is bad
# only top-level module should configure logger
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s', datefmt = '%Y-%m-%d %H:%M:%S')
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.info('myfunc was executed directly')
myfunc()
When I run the code, I see this output:
$>python main.py
INFO: 2016-07-14 18:13:04 <module>(22) -- calling mymodule.myfunc()
done with myfunc
But I expect to see this:
$>python main.py
INFO: 2016-07-14 18:13:04 <module>(22) -- calling mymodule.myfunc()
INFO: 2016-07-14 18:15:09 myfunc(8) -- myfunc is being called
done with myfunc
Anybody have any idea why the second logging.info call doesn't print to screen? thanks in advance!
Loggers exist in a hierarchy, with a root logger (retrieved with logging.getLogger(), no arguments) at the top. Each logger inherits configuration from its parent, with any configuration on the logger itself overriding the inherited configuration. In this case, you are never configuring the root logger, only the module-specific logger in main.py. As a result, the module-specific logger in mymodule.py is never configured.
The simplest fix is probably to use logging.basicConfig in main.py to set options you want shared by all loggers.
Chepner is correct. I got absorbed into this problem. The problem is simply in your main script
16 log = logging.getLogger() # use this form to initialize the root logger
17 #log = logging.getLogger(__name__) # never use this one
If you use line 17, then your imported python modules will not log any messages
In you submodule.py
import logging
logger = logging.getLogger()
logger.debug("You will not see this message if you use line 17 in main")
Hope this posting can help someone who got stuck on this problem.
While the logging-package is conceptually arranged in a namespace hierarchy using dots as separators, all loggers implicitly inherit from the root logger (like every class in Python 3 silently inherits from object). Each logger passes log messages on to its parent.
In your case, your loggers are incorrectly chained. Try adding print(logger.name) in your both modules and you'll realize, that your instantiation of logger in main.py is equivalent to
logger = logging.getLogger('__main__')
while in mymodule.py, you effectively produce
logger = logging.getLogger('mymodule')
The call to log INFO-message from myfunc() passes directly the root logger (as the logger in main.py is not among its ancestors), which has no handler set up (in this case the default message dispatch will be triggered, see here)
I have read from python docs https://docs.python.org/3/howto/logging.html#advanced-logging-tutorial that the best way to configure module level logging is to name the logger:
logger = logging.getLogger(__name__)
I the main application logging works fine:
if __name__ == '__main__':
logging.config.fileConfig('logging.conf')
# create logger
logger = logging.getLogger(__name__)
However in another module when I set the logger in the module scope:
logger = logging.getLogger(__name__)
the logger does not log anything. When I create the logger within a method logging works fine:
class TestDialog(QDialog, Ui_TestDialog):
def __init__(self, fileName, parent=None):
super(TestDialog, self).__init__(parent)
self.logger = logging.getLogger(__name__)
logger.debug("--_init_() "+str(fileName))
Then I would need to use the self.logger.. formatting to get a logger in all methods in the class - which I have never seen before. I tried to set the logging.conf to log where the call is comming from:
[loggers]
keys=root,fileLogger
...
[formatter_consoleFormatter]
format=%(asctime)s - %(module)s - %(lineno)d - %(name)s - %(levelname)s - %(message)s
datefmt=
However when the logger is set in the module scope logging still doesn't work even with this configuration.
I also tried:
logger = logging.getLogger('root')
at the start of a module, again no logger. However If I use:
logger = logging.getLogger('fileLogger')
at the start of a module, logging works fine and with config file above I can see which module the call is comming from.
Why is configuring logger using name not inheriting it's config from root?
Why does configuring using root not work while using fileHandler does, when both root and fileHandler are configured in the logging.conf file?
To avoid surprises, use disable_existing_loggers=False in your fileConfig() call, as documented in this section of the docs.
You should never need to use a self.logger instance variable - module level loggers should be fine.
I would like to log my python script that uses elasticsearch-py. In particular, I want to have three logs:
General log: log INFO and above both to the stdout and to a file.
ES log: only ES related messages only to a file.
ES tracing log: Extended ES logging (curl queries and their output for instance) only to a file.
Here is what I have so far:
import logging
import logging.handlers
es_logger = logging.getLogger('elasticsearch')
es_logger.setLevel(logging.INFO)
es_logger_handler=logging.handlers.RotatingFileHandler('top-camps-base.log',
maxBytes=0.5*10**9,
backupCount=3)
es_logger.addHandler(es_logger_handler)
es_tracer = logging.getLogger('elasticsearch.trace')
es_tracer.setLevel(logging.DEBUG)
es_tracer_handler=logging.handlers.RotatingFileHandler('top-camps-full.log',
maxBytes=0.5*10**9,
backupCount=3)
es_tracer.addHandler(es_tracer_handler)
logger = logging.getLogger('mainLog')
logger.setLevel(logging.DEBUG)
# create file handler
fileHandler = logging.handlers.RotatingFileHandler('top-camps.log',
maxBytes=10**6,
backupCount=3)
fileHandler.setLevel(logging.INFO)
# create console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setLevel(logging.INFO)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
consoleHandler.setFormatter(formatter)
fileHandler.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(consoleHandler)
logger.addHandler(fileHandler)
My problem is that INFO messages of es_logger are displayed also on the terminal. As a matter of fact the log messages are saved to the right files!
If I remover the part related to logger, then the ES logging works fine, i.e. only saved to the corresponding files. But then I don't have the other part.... What is it that I'm doing wrong with the last part of the settings?
Edit
Possible hint: In the sources of elasticsearch-py there's a logger named logger. Could it be that it conflicts with mine? I tried to change the name of logger to main_logger in the lines above but it didn't help.
Possible hint 2: If I replace logger = logging.getLogger('mainLog') with logger = logging.getLogger(), then the format of the output to the console of es_logger changes and becomes identical to the one defined in the snippet.
I think you are being hit by the somewhat confusing logger hierarchy propagation. Everything that is logged in "elasticsearch.trace" that passes the loglevel of that logger, will propagate first to the "elasticsearch" logger and then to the root ("") logger. Note that once the message passes the loglevel of the "elasticsearch.trace" logger, the loglevels of the parents ("elasticsearch" and root) are not checked, but all messages will be sent to the handlers. (The handlers themselves have log levels that do apply.)
Consider the following example that illustrates the issue, and a possible solution:
import logging
# The following line will basicConfig() the root handler
logging.info('DUMMY - NOT SEEN')
ll = logging.getLogger('foo')
ll.setLevel('DEBUG')
ll.addHandler(logging.StreamHandler())
ll.debug('msg1')
ll.propagate = False
ll.debug('msg2')
Output:
msg1
DEBUG:foo:msg1
msg2
You see that "msg1" is logged both by the "foo" logger, and its parent, the root logger (as "DEBUG:foo:msg1"). Then, when propagation is turned off ll.propagate = False before "msg2", the root logger no longer logs it. Now, if you were to comment out the first line (logging.info("DUMMY..."), then the behavior would change so that the root logger line would not be shown. This is because the logging module top level functions info(), debug() etc. configure the root logger with a handler when no handler has yet been defined. That is also why you see different behavior in your example when you modify the root handler by doing logger = logging.getLogger().
I can't see in your code that you would be doing anything to the root logger, but as you see, a stray logging.info() or the like in your code or library code would cause a handler to be added.
So, to answer your question, I would set logger.propagate = False to the loggers where it makes sense for you and where you want propagation, check that the log level of the handlers themselves are as you want them.
Here is an attempt:
es_logger = logging.getLogger('elasticsearch')
es_logger.propagate = False
es_logger.setLevel(logging.INFO)
es_logger_handler=logging.handlers.RotatingFileHandler('top-camps-base.log',
maxBytes=0.5*10**9,
backupCount=3)
es_logger.addHandler(es_logger_handler)
es_tracer = logging.getLogger('elasticsearch.trace')
es_tracer.propagate = False
es_tracer.setLevel(logging.DEBUG)
es_tracer_handler=logging.handlers.RotatingFileHandler('top-camps-full.log',
maxBytes=0.5*10**9,
backupCount=3)
es_tracer.addHandler(es_tracer_handler)
logger = logging.getLogger('mainLog')
logger.propagate = False
logger.setLevel(logging.DEBUG)
# create file handler
fileHandler = logging.handlers.RotatingFileHandler('top-camps.log',
maxBytes=10**6,
backupCount=3)
fileHandler.setLevel(logging.INFO)
# create console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setLevel(logging.INFO)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
consoleHandler.setFormatter(formatter)
fileHandler.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(consoleHandler)
logger.addHandler(fileHandler)