I will start by saying the I found a similar question and none of the suggested solutions worked for me at the end of 2022.
I do not manage to do simple things like changing the severity level of the logging on Colab:
Or writing the logs to a file:
import logging
logging.basicConfig(filename='app.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s', force=True)
logging.warning('This will get logged to a file')
As Colab is very popular and so it is the logging module, I thought it makes sense to ask again.
For the first problem, the debug message does not print because the lowest severity of the root logger is set to WARNING. You can change it to DEBUG with
import logging
logging.getLogger('RootLogger').setLevel(logging.DEBUG)
For the second problem, as Colab is still on Python 3.7 and not 3.8, the force= parameter does not work yet. So you just need to manually delete the default StreamHandler that sends logging output to sys.stderr before carrying out your configuration. (code source: https://stackoverflow.com/a/49202811/9987623)
import logging
# Check current handlers
print(logging.root.handlers)
# [<StreamHandler stderr (NOTSET)>]
# Remove all handlers associated with the root logger object.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
print(logging.root.handlers)
# []
Now, you can run your code. Here is an example:
from pathlib import Path
fmt = '%(name)s - %(levelname)s - %(message)s'
log_dir = Path.cwd().joinpath("logs")
path = Path(log_dir)
path.mkdir(exist_ok=True)
log_fname = "app.log"
log_path = log_dir.joinpath(log_fname)
logging.basicConfig(
level=logging.DEBUG,
filemode="w",
format=fmt,
filename=log_path,
)
logging.warning("this is a warning")
Next cell:
!cat ./logs/app.log
# root - WARNING - this is a warning
Next cell:
logging.debug('this is a debug message')
Next cell:
!cat ./logs/app.log
# root - WARNING - this is a warning
# root - DEBUG - this is a debug message
Related
I have a file named helper.py
import logging
import os
from json import load
def get_config(value):
with open('config.json','r') as f:
result=load(f)[value]
return result
def get_logger(name,level):
logpath=get_config("log_path")
if not os.path.exists(logpath):
os.mkdir(logpath)
logger = logging.getLogger(name)
if not bool(logger.handlers):
formatter = logging.Formatter('%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',datefmt='%Y-%m-%d %H:%M:%S')
fh = logging.FileHandler(os.path.join(logpath,f'{get_config("log_file_name")}.log'),mode="w",encoding='utf-8')
fh.setFormatter(formatter)
logger.addHandler(fh)
ch = logging.StreamHandler()
ch.setFormatter(formatter)
ch.setLevel(level)
logger.addHandler(ch)
return logger
LOGGER=get_logger("MyLogger",logging.INFO)
This is config.json:
{
"save_path" : "results/",
"log_path" : "logs/",
"log_file_name" : "MyLog"
}
let's say I am using LOGGER from helper it in x.py
from helper import LOGGER
logger=LOGGER
def div(x,y):
try:
logger.info("inside div")
return x/y
except Exception as e:
logger.error(f"div failed due to {e.message if 'message' in dir(e) else e}")
I am using this LOGGER in other files by importing helper.LOGGER for logging purposes but it's not printing anything on the console nor writing in a log file
My attempt:
I tried adding sys.stdout in StreamHandler() It doesn't worked
Then I tried setting the level of fh but nothing works
I tried adding basicConfig() instead of fileHandler() but then printing to console using print() and the output of logs is not coming in the correct order
Kindly let me know where I go wrong
Any help is appreciated :)
Thanks :)
You are not setting the level on the LOGGER, which by default is warning. This is why your info level log is not appearing. The Python documentation has a flow chart illustrating when a log will be logged: https://docs.python.org/3/howto/logging.html#logging-flow
The first thing it does, is that before a logger sends a log to their handlers it checks if the level is enabled for the logger. You should add logger.setLevel(level) in your get_logger().
Basically, I want to initialise a logger for each purpose, such that each logger will log to a file, but I notice that the content of log has been printed to console as well. I don't want that to happen, but I don't know how to adjust my code.
def getLogger(loggerName:str, fileDir:str,
level = logging.DEBUG,
format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'):
"""return a logger for logging
somehow the logDecorator function does
Args:
loggerName (str): name given to the logger
fileDir (str): directory
level ([type], optional): [description]. Defaults to logging.DEBUG.
format (str, optional): [description]. Defaults to '%(asctime)s - %(name)s - %(levelname)s - %(message)s'.
Returns:
logger: [description]
"""
myLogger = logging.getLogger(loggerName)
myLogger.setLevel(level)
myFormat = logging.Formatter(format)
myHandler = logging.FileHandler(fileDir, mode='a')
myHandler.setFormatter(myFormat)
myLogger.addHandler(myHandler)
# console only prints WARNING & ABOVE
# ideally, what I thought is that if I don't not
# add a SteamHandler, the log won't even print to console at all
# but adding a SteamHandler still prints EVERYTHING to the console
# despite the level set as WARNING
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
console.setFormatter(myFormat)
myLogger.addHandler(console)
return myLogger
I have tried this:
mainLogger = logging.getLogger()
mainLogger.setLevel(logging.WARNING) # only log WARNING and above
but It won't work as well, still printing to console.
So I am lost in my understanding of LOGGING module, I am not sure where have I done wrong.
Example of using the code:
aRandomLogger = getLogger(loggerName = 'ProgressLogger', fileDir = '../logs/ProgressLogger.log')
aRandomLogger.info('This is an information, I want it to be log to file, but not on console')
aRandomLogger.warning('this is a warning, I want it be be log to file, and shown on console')
Turns out, both logs are saved to file and to console.
It has been awhile, but I have managed to figure out the solution to this.
myLogger = logging.getLogger(loggerName)
myLogger.propagate = False
Setting propagate to False will prevent the logger from producing unwanted logs everywhere.
I'm simply trying to have a python logger with a specific format that outputs log messages only to the console. I've tried many different things but I keep getting 2 lines of console output per log call.
Here is my code:
logger = logging.getLogger('my_logger')
logger.setLevel(logging.INFO)
# Create console handler
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(asctime)s - %(name)s - %(message)s')
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
logger.info('TEST LOG info')
With output:
INFO - 2017-08-21 14:30:00,751 - my_logger - TEST LOG info
INFO:my_logger:TEST LOG info
I did exactly this and it didn't work: Disable output of root logger
Any idea what is going on? I don't care whether I use the root logger or not, I just want one line
The above code actually should be working correctly. Although my script was very lean, it was importing a non-system library which, somewhere down the line, had some logging configured which was affecting my output.
Following the question here: How do I log from my Python Spark script, I have been struggling to get:
a) All output into a log file.
b) Writing out to a log file from pyspark
For a) I use the following changes to the config file:
# Set everything to be logged to the console
log4j.rootCategory=ALL, file
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/home/xxx/spark-1.6.1/logging.log
log4j.appender.file.MaxFileSize=5000MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
This produces output and now for b) I would like to add my own input to logging from pyspark, but I cannot find any output written to the logs. Here is the code I am using:
import logging
logger = logging.getLogger('py4j')
#print(logger.handlers)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)
logger.addHandler(sh)
logger.info("TESTING.....")
I can find output in the logfile, but no "TESTING...." I have also tried using the existing logger stream but this does not work either.
import logging
logger = logging.getLogger('py4j')
logger.info("TESTING.....")
Works in my configuration:
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger(__name__)
LOGGER.info("Hello logger...")
All output into a log file & Writing out to a log file from pyspark
import os
import sys
import logging
import logging.handlers
log = logging.getLogger(__name_)
handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error
sys.stdout.write = log.info
(will log every error in "spam.log" in the same directory, nothing will be on console/stdout)
(will log every info in "spam.log" in the same directory,nothing will be on console/stdout)
to print output error/info in both file as well as in console remove above two line.
Happy Coding Cheers!!!
I don't know why this code prints to the screen, but not to the file? File "example1.log" is created, but nothing is written there.
#!/usr/bin/env python3
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(message)s',
handlers=[logging.FileHandler("example1.log"),
logging.StreamHandler()])
logging.debug('This message should go to the log file and to the console')
logging.info('So should this')
logging.warning('And this, too')
I have "bypassed" this problem by creating a logging object, but it keeps bugging me why basicConfig() approach failed?
PS. If I change basicConfig call to:
logging.basicConfig(level=logging.DEBUG,
filename="example2.log",
format='%(asctime)s %(message)s',
handlers=[logging.StreamHandler()])
then all logs are in the file and nothing is displayed in the console.
Try this working fine(tested in python 2.7) for both console and file
# set up logging to file
logging.basicConfig(
filename='log_file_name.log',
level=logging.INFO,
format= '[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s',
datefmt='%H:%M:%S'
)
# set up logging to console
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
logger = logging.getLogger(__name__)
I can't reproduce it on Python 3.3. The messages are written both to the screen and the 'example2.log'. On Python <3.3 it creates the file but it is empty.
The code:
from logging_tree import printout # pip install logging_tree
printout()
shows that FileHandler() is not attached to the root logger on Python <3.3.
The docs for logging.basicConfig() say that handlers argument is added in Python 3.3. The handlers argument isn't mentioned in Python 3.2 documentation.
Another technique using the basicConfig is to setup all your handlers in the statement and retrieve them after the fact, as in...
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(module)s %(funcName)s %(message)s',
handlers=[logging.FileHandler("my_log.log", mode='w'),
logging.StreamHandler()])
stream_handler = [h for h in logging.root.handlers if isinstance(h , logging.StreamHandler)][0]
stream_handler.setLevel(logging.INFO)
More sensibly though is to construct your stream handler instance outside and configure them as standalone objects that you pass to the handlers list as in...
import logging
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(module)s %(funcName)s %(message)s',
handlers=[logging.FileHandler("my_log.log", mode='w'),
stream_handler])
In the example below, you can specify the log destination based on its level. For example, the code below lets all logs over the INFO level go to the log file, and all above ERROR level goes to the console.
import logging
logging.root.handlers = []
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO , filename='ex.log')
# set up logging to console
console = logging.StreamHandler()
console.setLevel(logging.ERROR)
# set a format which is simpler for console use
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(message)s')
console.setFormatter(formatter)
logging.getLogger("").addHandler(console)
logging.debug('debug')
logging.info('info')
logging.warning('warning')
logging.error('error')
logging.exception('exp')
WOOAH!
I just spent about 20 minutes being baffled by this.
Eventually I worked out that the StreamHandler was outputting to stderr, not stdout (in a 'Doze DOS screen these have the same font colour!).
This resulted in me being able to run the code perfectly OK and get sensible results, but in a pytest function things going awry. Until I changed from this:
out, _ = capsys.readouterr()
assert 'test message check on console' in out, f'out was |{out}|'
to this:
_, err = capsys.readouterr()
assert 'test message check on console' in err, f'err was |{err}|'
NB the constructor of StreamHandler is
class logging.StreamHandler(stream=None)
and, as the docs say, "If stream is specified, the instance will use it for logging output; otherwise, sys.stderr will be used."
NB it seems that supplying the level keyword does not run setLevel on the handlers: you'd need to iterate on the resulting handlers and run setLevel on each, if it matters to you.
This is a ValueError if FileHandler and StreamHandler both are present in BasicConfig function
https://docs.python.org/3/library/logging.html#logging.basicConfig
See image below: