Is there any way in python logging module to send info & errors to stdout and debug to file.
Some commands in my script produce a long output which I don't want to send to stdout.
I am using the following logging function which writes logs to file and stdout
def mylog(release_num, logdir='/tmp'):
applog = logging.getLogger()
applog.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s %(levelname)s %(message)s", "%b %d %Y %H:%M:%S")
logfile = "{}/{}.log".format(logdir, release_num)
if not os.path.exists(logdir):
os.makedirs(logdir)
fileHandler = logging.FileHandler(logfile, 'ab')
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(formatter)
applog.addHandler(fileHandler)
cformat = logging.Formatter("[%(levelname)8s] : %(message)s")
consoleHandler = logging.StreamHandler(sys.stdout)
consoleHandler.setFormatter(cformat)
log.addHandler(consoleHandler)
return applog
You need to set the loglevel of consoleHandler to logging.INFO to log messages of level info or higher through the handler:
consoleHandler.setLevel(logging.INFO)
Related
I want log the output generated by loaded modules in my main program.
I have my logger in my main program configured like this:
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
# output handler
handler = logging.FileHandler(os.path.join(
file_path, 'logs/logfile.log'), 'a')
handler.setFormatter(logging.Formatter(
"%(asctime)s : %(levelname)s : [%(filename)s:%(lineno)s - %(funcName)s()] : %(message)s",
"%Y-%m-%d %H:%M:%S"))
log.addHandler(handler)
# streamhandler for stderr and/or stdout from the modules
stream_hander = logging.StreamHandler(sys.stdout)
stream_hander.setLevel(level=logging.INFO)
stream_hander.setFormatter(logging.Formatter(
"%(asctime)s : %(levelname)s : [%(filename)s:%(lineno)s - %(funcName)s()] : %(message)s",
"%Y-%m-%d %H:%M:%S"))
log.addHandler(stream_hander)
stream_hander = logging.StreamHandler(sys.stderr)
stream_hander.setLevel(level=logging.ERROR)
stream_hander.setFormatter(logging.Formatter(
"%(asctime)s : %(levelname)s : [%(filename)s:%(lineno)s - %(funcName)s()] : %(message)s",
"%Y-%m-%d %H:%M:%S"))
log.addHandler(stream_hander)
However in submodules I load a basiclogger is used and prints/logs to stdout. This does print in my terminal, however these lines aren't written to the logfile.
How can I capture this so all everything is written to the logfile.
I prefer to do this in Python and not pipe the output of the script to a file with 2>&1
I have inherited a python script that I'm trying to log in Glue.
Originally it had prints, but they were only sent once job finished, but it was not possible to see the status of the execution in running time.
I've changed the log system to the cloudwatch one, but apparently it doesn't send the logs in streaming way as the Spark one, according to this.
I decided to follow what they recommended and use watchtower for this purposes and I have a code like that:
def initialize_log() -> logging.Logger:
logger = logging.getLogger(__name__)
log_format = "[%(name)s] %(asctime)s %(levelname)-8s %(message)s"
date_format = "%a, %d %b %Y %H:%M:%S %Z"
log_stream = sys.stdout
logging.basicConfig(level=logging.INFO, format=log_format, stream=log_stream, datefmt=date_format)
logger.addHandler(watchtower.CloudWatchLogHandler(log_group='/aws-glue/python-job', stream_name='my_stream_name'))
return logger
def log(logger, message):
logger.info(message)
logger.info(dict(foo="bar", details={}))
However, when I do:
logger = initialize_log()
log(logger, "Message")
I cannot find this into message in Cloudwatch /aws-glue/python-job or any directory.
I would like to ask if you know what I may be doing wrong.
Thank you in advance
Solved with logging library:
import logging
def initialize_log() -> logging.Logger:
logger = logging.getLogger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
def log(message: str):
logger.info(message)
I'm using the logging library and would like to log on to the console and store those logs in a file.
logger = logging.getLogger(module)
handler = logging.StreamHandler()
formatter = logging.Formatter(
'%(asctime)s [%(name)s] %(levelname)-8s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
filehandler=logging.FileHandler('logfile.log')
logger.addHandler(filehandler)
logger.info('hello')
my goal is to see logs in the below format.
[time] [module] [info/error] [log-msg]
I'm getting this format in the console but in log file I'm getting only log-msg
filehandler.setFormatter(formatter)
I have a bootstrap script for a Raspberry Pi that runs in python. I am looking to create a logger that logs to a file as well as to the console.
I was going to do something like this:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s] %(message)s",
handlers=[
logging.FileHandler("{0}/{1}.log".format(logPath, fileName)),
logging.StreamHandler()
])
But what I would really like is to log INFO to the StreamHandler and DEBUG to the FileHandler... I cannot seem to figure that out.
Can anyone help me out?
Using Python 3.7.5
You could build the logger yourself (either through a config file or in pure python)
The tricky thing that I have wasted several hours on is forgetting to set the log level on the logger as well as on each of the handlers. Ensure that the logger is as permissive as the most permissive handler.
example script
# emits the info line to the console and
# both the info & debug lines to the log file
# test_pylog.py
import logging
log_format = logging.Formatter(
'%(asctime)s %(threadName)s %(levelname)s %(message)s'
)
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(log_format)
logger.addHandler(console_handler)
file_handler = logging.FileHandler('logfile.txt')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(log_format)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
if __name__ == '__main__':
logger.debug('Panic! at the disco')
logger.info('Weezer')
I'm trying to figure out how to capture messages generated by Python/NumPy intractive shell when running my script. I would like to log all generated by console messages (errors, warnings) to same file as defined in my code log messages with time stamps:
def LogToFile():
global logger
logger = logging.getLogger('MyApp')
logger.setLevel(logging.DEBUG)
file_log_handler = RotatingFileHandler('logfile.log', maxBytes=1024, backupCount=5)
logger.addHandler(file_log_handler)
stderr_log_handler = logging.StreamHandler()
logger.addHandler(stderr_log_handler)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_log_handler.setFormatter(formatter)
stderr_log_handler.setFormatter(formatter)
return logger
Afaik, you'd have to specify this in basicConfig, not in your logger:
logging.basicConfig(filename=LOG_FILE,
level=logging.DEBUG)
before you do
logger = logging.getLogger('MyApp')
logger.setLevel(logging.DEBUG)