How can one change the formatting of output from the logging module in Google App Engine?
I've tried, e.g.:
log_format = "* %(asctime)s %(levelname)-8s %(message)s"
date_format = "%a, %d %b %Y %H:%M:%S"
console = logging.StreamHandler()
fr = logging.Formatter(log_format)
console.setFormatter(fr)
logger = logging.getLogger()
logger.addFilter(SuperfluousFilter())
logger.addHandler(console)
logger.setLevel(logging.DEBUG)
console.setLevel(logging.DEBUG)
logging.error("Reconfiguring logging")
However this results in duplicate logging output: One with the logging handler from google/appengine/tools/dev_appserver.py (or somewhere in the Google code), and one from my new StreamHandler above. The above code outputs:
ERROR 2010-06-23 20:46:18,871 initialize.py:38] Reconfiguring logging
2010-06-23 20:46:18,871 ERROR Reconfiguring logging
Where the top line is clearly from dev_appserver.py, the bottom line from my code.
So I guess the corollary question is: How can change the formatting of Google App Engine, yet avoid the duplicate output?
Thank you for reading.
Brian
Here is one way you can change the logging format without duplicating output:
# directly access the default handler and set its format directly
logging.getLogger().handlers[0].setFormatter(fr)
This is a bit of a hack because you have to directly access the handlers list stored in the root logger. The problem is GAE automatically uses logging before your code is ever run - this creates a default handler. Unfortunately, I don't see how you can get a reference to this handler without directly accessing the handlers list as above.
Related
I have a basic config for the logging module with debug level - now I want to create another logger with error level only. How can I do that?
The problem is that the root handler is called in addition to the error-handler - this is something I want to avoid.
import logging
fmt = '%(asctime)s:%(funcName)s:%(lineno)d:%(levelname)s:%(name)s:%(message)s'
logging.basicConfig(level=logging.DEBUG, format=fmt)
logger = logging.getLogger('Temp')
logger.setLevel(logging.ERROR)
handler = logging.StreamHandler()
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.error('boo')
The above code prints boo twice while I expect once only, and I have no idea what to do with this annoying issue...
In [4]: logger.error('boo')
boo
2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
In [5]: logger.handlers
Out[5]: [<StreamHandler stderr (ERROR)>]
Some basics about the logging module
logger: a person who receives the log string, sorts it by a predefined level, then uses his own handler if any to process the log and, by default passes the log to his superior.
root logger: the superior of superiors, does all the things that a normal logger does but doesn't pass the received log to anyone else.
handler: a private contractor of a logger, who actually does anything with the log, eg. formats the log, writes it to a file or stdout, or sends it through tcp/udp.
formatter: a theme, a design that the handler applies to the log.
basicConfig: a shortcut way to config the root logger. This is useful when you want him to do all the job and all his lower rank loggers would just pass the log to him.
With no argument, basicConfig sets root logger's level to WARNING and add a StreamHandler that output the log to stderr.
What you did
You created a format and used a shortcut basicConfig to config the root logger. You want the root logger to do all the actual logging things
You created a new low-rank logger Temp
You want it to accept logs with only ERROR level and above.
You created another StreamHandler. Which output to stdout by default.
You want it to handle only ERROR level and above
Oh, you assigned it to Temp logger, which made 5. redundant since the level is set in 3.
Oh wait, thought you just want the root logger to do the job since 1.!
You logged an ERROR with your logger.
What happened
Your Temp logger accepted a string boo at ERROR level. Then told its handler to process the string. Since this handler didn't have any formatter assigned to it, it outputted the string as-is to stdout: boo
After that, Temp logger passed the string boo to his superior, the root logger.
The root logger accepted the log since the log level is ERROR > WARNING.
The root logger then told its handler to process the string boo.
This handler applies the format string to boo. Added timestamp, added location, added the name of logger that passed the log, etc.
Finally it outputted the result to stderr: 2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
Make it right
Since your code does exactly what you tell it to do, you have to tell it as much detail as possible.
Only use basicConfig when you are lazy. By removing basicConfig line, your problem solved.
Use logger = logging.getLogger('__name__') so that the logger has the name of the module. Looking at the log and know exactly which import path that it came from.
Decide if a logger should keep the log on its own or pass it up the chain with the propagate property. In your case, logger.propagate = False also solves the problem.
Use a dictConfig file so you don't get messed with the config code.
In practice, you should not add handlers to your logger, and just let the logger pass the log all the way to the root and let the root do the actual logging. Why?
Someone else uses your code as a module, can control the logging. For example, not output to stdout but to tcp/udp, output with a different format, etc.
You can turn off the logging from a specific logger entirely, by propagating=False.
You know exactly all the handlers and formatters in the code if you only added them to the root logger. You have centralized control over the logging.
To set up logging in Python without basicConfig we would go through the steps:
Set up a file handler.
Set the logging level of the file handler.
Set up a formatter.
Point the file handler to the formatter.
Get the logger object.
Set the logging level of the logger object.
Add the file handler as a handler to the logger object.
Use the .info(), .warning(), etc method on the logger.
These steps are executed by the following code:
import logging
file_handler = logging.FileHandler('./out.log', 'a')
file_handler.setLevel(logging.DEBUG)
format_string = '%(asctime)s\t%(levelname)s: %(message)s'
formatter = logging.Formatter(format_string)
file_handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
logger.info('visible info')
logger.debug('invisible debug')
What is the difference between setting the logging level for the file handler and setting the logging level for the logger?
Okay, so here is a small bit of code to work out:
import logging
# Declare a function to log all 5 levels with different information
def log_all_levels(logger):
logger.debug("Debug from logger {}".format(logger.name))
logger.info("Info from logger {}".format(logger.name))
logger.warning("Warning from logger {}".format(logger.name))
logger.error("Error from logger {}".format(logger.name))
logger.critical("Fatal from logger {}".format(logger.name))
# This file handler will track errors from all loggers
all_errors_handler = logging.FileHandler('errors.log')
all_errors_handler.setLevel(logging.ERROR)
# This file handler will only be used in a specific region of code
foo_info_handler = logging.FileHandler('foo_info.log')
foo_info_handler.setLevel(logging.INFO)
foo_info_handler.addFilter(lambda r: r.levelno == logging.INFO)
# The following loggers will be used in the main execution
foo_logger = logging.getLogger("Foo")
nameless_logger = logging.getLogger("nameless")
foo_logger.setLevel(logging.INFO)
nameless_logger.setLevel(logging.DEBUG)
loggers = (foo_logger, nameless_logger)
# Set each logger up to use the file handlers
# Each logger can have many handlers, each handler can be used by many loggers
for logger in loggers:
logger.addHandler(all_errors_handler)
debug_file_handler = logging.FileHandler('{}.log'.format(logger.name))
debug_file_handler.setLevel(logging.DEBUG)
logger.addHandler(debug_file_handler)
if logger.name == "Foo":
logger.addHandler(foo_info_handler)
# Let's run some logging operations
for logger in loggers:
log_all_levels(logger)
There are 2 loggers - foo_logger set to the info level and nameless_logger set to the debug level. Both of them use the errors and debug handlers, however only the foo_logger uses the foo_file_handler. There are now loggers and file handlers with different levels, connected together in a many-to-many relationship.
As you can find out:
errors.log will contain errors from both loggers. Quite self-explanatory for a real life scenario - reading through logs containing just the errors helps debugging the code.
Foo.log and nameless.log will contain everything possible about those loggers, respecting their levels. So the former will contain info and greater, whereas the latter will track debug and greater levels. Logging per object will potentially create a lot of files, but it might be crucial when trying to detect some object-specific errors.
foo_info is a very special file handler and it only allows info level from the associated logger. Such files can be a life saver when you enter a potentially unsafe or untested area of code and would like to see what exactly is happening within that code block, without having to browse through all your program log.
There are many other things you can do with logging - set up your own logging rules, make a logging hierarchy, create a logger factory - possibilities are endless. Logging should allow flexibility - for example by allowing logger objects and file handlers to have different and separate logging levels, and letting the programmer combine them together as needed.
I hope the small code exercise alongside with my explanations cleared any further doubts - but I do recommend to have a look at Logging Cookbook or the docs if you still need more examples.
I set up a basic python logger that writes to a log file and to stdout. When I run my python program locally, log messages with logging.info appear as expected in the file and in the console. However, when I run the same program remotely via ssh -n user#server python main.py neither the console nor the file show any logging.info messages.
This is the code used to set up the logger:
def setup_logging(model_type, dataset):
file_name = dataset + "_" + model_type + time.strftime("_%Y%m%d_%H%M")
logging.basicConfig(
level=logging.INFO,
format="[%(levelname)-5.5s %(asctime)s] %(message)s",
datefmt='%H:%M:%S',
handlers=[
logging.FileHandler("log/{0}.log".format(file_name)),
logging.StreamHandler()
])
I already tried the following things:
Sending a message to logging.warning: Those appear as expected on the root logger. However, even without setting up the logger and falling back to the default logging.info messages do not show up.
The file and folder permissions seem to be alright and an empty file is created on disk.
Using print works as usual as well
If you look into the source code of basicConfig function, you will see that the function is applied only when there are no handlers on the root logger:
_acquireLock()
try:
force = kwargs.pop('force', False)
if force:
for h in root.handlers[:]:
root.removeHandler(h)
h.close()
if len(root.handlers) == 0:
handlers = kwargs.pop("handlers", None)
if handlers is None:
...
I think, one of the libraries you use configures logging on import. And as you see from the sample above, one of the solutions is to use force=True argument.
A possible disadvantage is that several popular data-science libraries keep a reference to the loggers they configure, so that when you reconfigure logging yourselves their old loggers with the handlers are still there and do not see your changes. In which case you will also need to clean the handlers for those loggers as well.
Im trying to use the logging module to print to file.
IT works fine, my file has my logs but It also shows those logs to my console window, which i dont want to happen.
I tried the following:
logger = logging.getLogger('log')
hdlr = logging.handlers.RotatingFileHandler(LOG_FILENAME, maxBytes=200000, backupCount=3)
formatter = logging.Formatter('%(asctime)s -- %(levelname)s -- \n%(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
logger.propagate = False
even tough, it still prints to console.
Any ideas why?
Most likely you have a call to logging.basicConfig() somewhere, or a call to logging.info() or similar (calling a module-level function, rather than a logger method).
The module-level functions are meant for simple usage, and create and add a console handler if no handlers are assigned to the root logger. So ensure that you have no logging.debug(...) style calls and that you're not calling basicConfig() or other logging configuration functions which might be changing your logging configuration.
First time playing around with the logging module. I have two questions really:
All of the examples I have found on creating a FileHandler just use a dummy file name, but I want my log file to have a certain title format.
log = time.strftime('./logs/'+'%H:%M:%S %d %b %Y', time.localtime())+'.log'
logger = logging.getLogger('myproject')
formatter = logging.Formatter('%(asctime)s %(message)s', '%H:%M:%S %d %b %Y')
handler = logging.FileHandler(log)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
This seems to work at first, in that the log file is created, but when I kill the script and examine it, there is nothing in it. And before you suggest using with open(log) as f:, using f in place of log in handler = logging.FileHandler(log) gives me errors about the file not even being created. What am I doing wrong to not have the log file written to? Is it because I'm trying to be too clever with the filename?
The second question may subsume the first somehow. What I would really like is a way to write to both the console and the log file concurrently, since this script takes a long time and I'd like to gauge progress. I have seen that there are answers on how to do this, such as here, but all of these seem to assume that I want different levels of logging for the two different outputs; I don't. Incidentally, I do realize that if my script didn't dynamically generate the name of the log file, I could do something like python myscript.py | tee log.file to avoid making different handlers.
I feel I am having trouble answering my second question because of the variables involved in the first, but I am foggy right now due to illness and subsequent lack of sleep. If someone could explain to me like a five-year-old how to write to both a console and log file with identical output, I would greatly appreciate it.
Many thanks.
Your code is configuring a logger named 'myproject':
logger = logging.getLogger('myproject')
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
If you use methods on that logger, everything will work:
logger.info('foo bar')
logging.getLogger('myproject').info('foo bar')
But if you just use the module-level functions, it will not:
logging.info('foo bar')
Why? Because the module-level functions use the default root logger, which is not the one you've configured. See the docs for details on how this works.
Generally, the way you deal with this is either to create a module-level logger object for each module, like this:
logger = logging.getLogger(__name__)
… or a class or instance attribute logger, something like this:
self.logger = logging.getLogger('{}.{}'.format(__name__, cls.__name__))
And then do everything through that logger or self.logger object, not through the module-level functions.
So, why are the module-level functions even there? For convenience in simple programs, mainly. If that sounds appropriate for you, and you want to use them, you can; you just have to configure the root logger instead of a different one. Change the first line to this:
logger = logging.getLogger()
Now when you configure logger, you're also configuring the module-level functions.