Develop Reference

Python logger - Redirecting STDOUT to logfile as well as any debug messages

I'm trying to use the logging module in Python so that, when I run my program, I end up with a log file, debug.log, containing:
Every log message (logging.DEBUG, logging.WARNING etc.)
Every time my code prints something to STDOUT
When I run the program, I only want the debug messages to appear in the log file, not to be printed on the terminal.
Based on this answer, here's my example code, test.py:
import logging
import sys
root = logging.getLogger()
root.setLevel(logging.DEBUG)
fh = logging.FileHandler('debug.log')
fh.setLevel(logging.DEBUG)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
sh.setFormatter(formatter)
fh.setFormatter(formatter)
root.addHandler(sh)
root.addHandler(fh)
x = 4
y = 5
logging.debug("X: %s", x)
logging.debug("Y: %s", y)
print("x is", x)
print("y is", y)
print("x * y =", x*y)
print("x^y =", x**y)
And here's what I would want to be the contents of debug.log:
2021-02-01 12:10:48,263 - root - DEBUG - X: 4
2021-02-01 12:10:48,264 - root - DEBUG - Y: 5
x is 4
y is 5
x * y = 20
x^y = 1024
Instead, the contents of debug.log is just the first two lines:
2021-02-01 12:10:48,263 - root - DEBUG - X: 4
2021-02-01 12:10:48,264 - root - DEBUG - Y: 5
When I run test.py, I get this output:
2021-02-01 12:17:04,201 - root - DEBUG - X: 4
2021-02-01 12:17:04,201 - root - DEBUG - Y: 5
x is 4
y is 5
x * y = 20
x^y = 1024
So I've actually got the opposite results to what I want: the log file excludes STDOUT prints where I want them included, and the program output includes the debug messages where I want them excluded.
How can I fix this, so that running test.py only outputs the lines in print statements, and the resulting debug.log file contains both the DEBUG logs and the print lines?

Well, I can make it work, but I don't yet know whether there are any repercussions due to it. Perhaps others will be able to point out any potential pitfalls, such as multithreading.
You can set sys.stdout to any file-like object that you like. This would include the file to which your logging.FileHandler() is writing. Try this:
fh = logging.FileHandler('debug.log')
fh.setLevel(logging.DEBUG)
old_stdout = sys.stdout # in case you want to restore later
sys.stdout = fh.stream # the file to which fh writes
And you can remove the code that deals with sh that is hooked up to stdout.

If you really want all output to stdout to end up in the log file, see e.g. mhawke's answer, as well as the linked questions and answers.
However, if you are really just interested in the output from your own print() calls, then I would replace all of those by Logger.log() calls, using a custom logging level. That gives you fine-grained control over what happens.
Below, I define a custom log level with a value higher than logging.CRITICAL, so our console output is always printed, even if the logger's level is CRITICAL. See docs.
Here's a minimal implementation, based on the OP's example:
import sys
import logging
# define a custom log level with a value higher than CRITICAL
CUSTOM_LEVEL = 100
# dedicated formatter that just prints the unformatted message
# (actually this is the default behavior, but we make it explicit here)
# See docs: https://docs.python.org/3/library/logging.html#logging.Formatter
console_formatter = logging.Formatter('%(message)s')
# your basic console stream handler
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(console_formatter)
# only use this handler for our custom level messages (highest level)
console_handler.setLevel(CUSTOM_LEVEL)
# your basic file formatter and file handler
file_formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler = logging.FileHandler('debug.log')
file_handler.setFormatter(file_formatter)
file_handler.setLevel(logging.DEBUG)
# use a module logger instead of the root logger
logger = logging.getLogger(__name__)
# add the handlers
logger.addHandler(console_handler)
logger.addHandler(file_handler)
# include messages with level DEBUG and higher
logger.setLevel(logging.DEBUG)
# NOW, instead of using print(), we use logger.log() with our CUSTOM_LEVEL
x = 4
y = 5
logger.debug(f'X: {x}')
logger.debug(f'Y: {y}')
logger.log(CUSTOM_LEVEL, f'x is {x}')
logger.log(CUSTOM_LEVEL, f'y is {y}')
logger.log(CUSTOM_LEVEL, f'x * y = {x*y}')
logger.log(CUSTOM_LEVEL, f'x^y = {x**y}')

I don't think you want ALL stdout output going to log file.
What you could do is set the logging level of the console handler to logging.INFO and the logging level of the file handler to logging.DEBUG. Then replace your print statements with calls to logging.info. This way only info messages and above will be output to the console.
Something like this:
import logging
import sys
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler(sys.stdout)
file_handler = logging.FileHandler("debug.log")
console_handler.setLevel(logging.INFO)
file_handler.setLevel(logging.DEBUG)
console_formatter = logging.Formatter('%(message)s')
file_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(console_formatter)
file_handler.setFormatter(file_formatter)
logger.addHandler(console_handler)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG) #set root logging level to DEBUG
if __name__ == "__main__":
x = 4
y = 5
logger.debug("X: %s", x)
logger.debug("Y: %s", y)
logger.info("x is {}".format(x))
logger.info("y is {}".format(y))
logger.info("x * y = {}".format(x * y))
logger.info("x^y = {}".format(x ** y))
Demo

Create different log names at each execution time in Python

import os
import logging
os.chdir("C:\Users\SIT\Desktop\Powersupply")
logging.basicConfig(
filename='mylog{0}.txt'.format(i),
format='%(asctime)s %(message)s',
datefmt='%m/%d/%Y %I:%M:%S %p'
)
logging.warning(123)
As above , When I execute it once , I want the file name to generate different log names .
ex:
Once -> mylog(0)
Second->mylog(1)
third ->mylog(3)

Try this:
import datetime
now = datetime.datetime.now()
time_string = now.strftime( '%Y-%m-%d_%H%M%S' )
file_name = 'mylog_%s.txt' % time_string

How to log time spent from when script started instead of actual time when using python logging module?

I am looking for a way to change python logging module to display time spent form when the script started instead of current time.

Use %(relativeCreated)s in your format string, as indicated in the documentation.
Update: You can use normal Python format specifiers to control precision, e.g. %(relativeCreated).0f to show floating point values with zero decimal places.

You can subclass logging.Formatter and reimplement formatTime. Something like that:
start_time = datetime.now()
class MyFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
delta = (datetime.now() - start_time).total_seconds()
return "{}".format(delta)
And then:
handler = logging.StreamHandler()
fmt = MyFormatter('%(filename)s %(levelname)-8s [%(asctime)s] %(message)s')
handler.setFormatter(fmt)
log = logging.getLogger('main')
log.addHandler(handler)
log.debug("=)")

You could try something like this.
Define a custom formatter at the beginning of your script:
import time
import logging
import datetime as dt
class MyFormatter(logging.Formatter):
def __init__(self,fmt=None,datefmt=None):
super(MyFormatter,self).__init__(fmt,datefmt)
self.reftime = dt.datetime.fromtimestamp(time.mktime(time.localtime()))
def formatTime(self, record, datefmt=None):
ctime = dt.datetime.fromtimestamp(time.mktime(self.converter(record.created)))
ct = (ctime - self.reftime).timetuple()
if datefmt:
s = time.strftime(datefmt, ct)
else:
t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
s = "%s,%03d" % (t, record.msecs)
return s
Then setup your logging system:
handler = logging.StreamHandler(MyFormatter())
logger = logging.getLogger()
logger.addHandler(handler)
etc...

Python logging: use milliseconds in time format

By default logging.Formatter('%(asctime)s') prints with the following format:
2011-06-09 10:54:40,638
where 638 is the millisecond. I need to change the comma to a dot:
2011-06-09 10:54:40.638
To format the time I can use:
logging.Formatter(fmt='%(asctime)s',datestr=date_format_str)
however the documentation doesn't specify how to format milliseconds. I've found this SO question which talks about microseconds, but a) I would prefer milliseconds and b) the following doesn't work on Python 2.6 (which I'm working on) due to the %f:
logging.Formatter(fmt='%(asctime)s',datefmt='%Y-%m-%d,%H:%M:%S.%f')

This should work too:
logging.Formatter(
fmt='%(asctime)s.%(msecs)03d',
datefmt='%Y-%m-%d,%H:%M:%S'
)

Please note Craig McDaniel's solution is clearly better.
logging.Formatter's formatTime method looks like this:
def formatTime(self, record, datefmt=None):
ct = self.converter(record.created)
if datefmt:
s = time.strftime(datefmt, ct)
else:
t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
s = "%s,%03d" % (t, record.msecs)
return s
Notice the comma in "%s,%03d". This can not be fixed by specifying a datefmt because ct is a time.struct_time and these objects do not record milliseconds.
If we change the definition of ct to make it a datetime object instead of a struct_time, then (at least with modern versions of Python) we can call ct.strftime and then we can use %f to format microseconds:
import logging
import datetime as dt
class MyFormatter(logging.Formatter):
converter=dt.datetime.fromtimestamp
def formatTime(self, record, datefmt=None):
ct = self.converter(record.created)
if datefmt:
s = ct.strftime(datefmt)
else:
t = ct.strftime("%Y-%m-%d %H:%M:%S")
s = "%s,%03d" % (t, record.msecs)
return s
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console = logging.StreamHandler()
logger.addHandler(console)
formatter = MyFormatter(fmt='%(asctime)s %(message)s',datefmt='%Y-%m-%d,%H:%M:%S.%f')
console.setFormatter(formatter)
logger.debug('Jackdaws love my big sphinx of quartz.')
# 2011-06-09,07:12:36.553554 Jackdaws love my big sphinx of quartz.
Or, to get milliseconds, change the comma to a decimal point, and omit the datefmt argument:
class MyFormatter(logging.Formatter):
converter=dt.datetime.fromtimestamp
def formatTime(self, record, datefmt=None):
ct = self.converter(record.created)
if datefmt:
s = ct.strftime(datefmt)
else:
t = ct.strftime("%Y-%m-%d %H:%M:%S")
s = "%s.%03d" % (t, record.msecs)
return s
...
formatter = MyFormatter(fmt='%(asctime)s %(message)s')
...
logger.debug('Jackdaws love my big sphinx of quartz.')
# 2011-06-09 08:14:38.343 Jackdaws love my big sphinx of quartz.

Adding msecs was the better option, Thanks.
Here is my amendment using this with Python 3.5.3 in Blender
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s.%(msecs)03d %(levelname)s:\t%(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
log = logging.getLogger(__name__)
log.info("Logging Info")
log.debug("Logging Debug")

The simplest way I found was to override default_msec_format:
formatter = logging.Formatter('%(asctime)s')
formatter.default_msec_format = '%s.%03d'

I figured out a two-liner to get the Python logging module to output timestamps in RFC 3339 (ISO 1801 compliant) format, with both properly formatted milliseconds and timezone and without external dependencies:
import datetime
import logging
# Output timestamp, as the default format string does not include it
logging.basicConfig(format="%(asctime)s: level=%(levelname)s module=%(module)s msg=%(message)s")
# Produce RFC 3339 timestamps
logging.Formatter.formatTime = (lambda self, record, datefmt=None: datetime.datetime.fromtimestamp(record.created, datetime.timezone.utc).astimezone().isoformat())
Example:
>>> logging.getLogger().error("Hello, world!")
2021-06-03T13:20:49.417084+02:00: level=ERROR module=<stdin> msg=Hello, world!
Alternatively, that last line could be written out as follows:
def formatTime_RFC3339(self, record, datefmt=None):
return (
datetime.datetime.fromtimestamp(record.created, datetime.timezone.utc)
.astimezone()
.isoformat()
)
logging.Formatter.formatTime = formatTime_RFC3339
That method could also be used on specific formatter instances, rather than overriding at the class level, in which case you will need to remove self from the method signature.

Many outdated, over-complicated and weird answers here. The reason is that the documentation is inadequate and the simple solution is to just use basicConfig() and set it as follows:
logging.basicConfig(datefmt='%Y-%m-%d %H:%M:%S', format='{asctime}.{msecs:0<3.0f} {name} {threadName} {levelname}: {message}', style='{')
The trick here was that you have to also set the datefmt argument, as the default messes it up and is not what is (currently) shown in the how-to python docs. So rather look here.
An alternative and possibly cleaner way, would have been to override the default_msec_format variable with:
formatter = logging.Formatter('%(asctime)s')
formatter.default_msec_format = '%s.%03d'
However, that did not work for unknown reasons.
PS. I am using Python 3.8.

A simple expansion that doesn't require the datetime module and isn't handicapped like some other solutions is to use simple string replacement like so:
import logging
import time
class MyFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
ct = self.converter(record.created)
if datefmt:
if "%F" in datefmt:
msec = "%03d" % record.msecs
datefmt = datefmt.replace("%F", msec)
s = time.strftime(datefmt, ct)
else:
t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
s = "%s,%03d" % (t, record.msecs)
return s
This way a date format can be written however you want, even allowing for region differences, by using %F for milliseconds. For example:
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
sh = logging.StreamHandler()
log.addHandler(sh)
fm = MyFormatter(fmt='%(asctime)s-%(levelname)s-%(message)s',datefmt='%H:%M:%S.%F')
sh.setFormatter(fm)
log.info("Foo, Bar, Baz")
# 03:26:33.757-INFO-Foo, Bar, Baz

After instantiating a Formatter I usually set formatter.converter = gmtime. So in order for #unutbu's answer to work in this case you'll need:
class MyFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
ct = self.converter(record.created)
if datefmt:
s = time.strftime(datefmt, ct)
else:
t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
s = "%s.%03d" % (t, record.msecs)
return s

If you are using arrow or if you don't mind using arrow. You can substitute python's time formatting for arrow's one.
import logging
from arrow.arrow import Arrow
class ArrowTimeFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
arrow_time = Arrow.fromtimestamp(record.created)
if datefmt:
arrow_time = arrow_time.format(datefmt)
return str(arrow_time)
logger = logging.getLogger(__name__)
default_handler = logging.StreamHandler()
default_handler.setFormatter(ArrowTimeFormatter(
fmt='%(asctime)s',
datefmt='YYYY-MM-DD HH:mm:ss.SSS'
))
logger.setLevel(logging.DEBUG)
logger.addHandler(default_handler)
Now you can use all of arrow's time formatting in datefmt attribute.

If you prefer to use style='{', fmt="{asctime}.{msecs:0<3.0f}" will 0-pad your microseconds to three places for consistency.

After burning some of my precious time the below hack worked for me. I just updated my formatter in settings.py and added datefmt as %y/%b/%Y %H:%M:%S and appended the milliseconds to the asctime like this {asctime}.{msecs:0<3.0f}
E.G:
'formatters': {
'verbose': {
'format': '[{asctime}.{msecs:0<3.0f}] {levelname} [{threadName:s}] {module} → {message}',
'datefmt': "%y/%b/%Y %H:%M:%S",
'style': '{',
},
}

Using this smart answer for the timezone and the chosen answer, you can construct the millisecond and timezone with your desired format:
import logging
import time
if __name__ == "__main__":
tz = time.strftime('%z')
logging.basicConfig(
format=(
"%(asctime)s.%(msecs)03d" + tz + " %(levelname)s "
"%(pathname)s:%(lineno)d[%(threadName)s]: %(message)s"
),
level=logging.DEBUG,
datefmt="%Y-%m-%dT%H:%M:%S",
)
logging.info("log example")
Personally, I like to keep all the logs in UTC but also have this explicitly in the log as a datetime without a timezone is meaningless in a multizone application:
logging.Formatter.converter = time.gmtime
logging.basicConfig(
format=(
"%(asctime)s.%(msecs)03d+0000 %(levelname)s "
"%(pathname)s:%(lineno)d[%(threadName)s]: %(message)s"
),
level=logging.DEBUG,
datefmt="%Y-%m-%dT%H:%M:%S",
)

tl;dr for folks looking here for an ISO formatted date:
instead of using something like '%Y-%m-%d %H:%M:%S.%03d%z', create your own class as #unutbu indicated. Here's one for iso date format:
import logging
from time import gmtime, strftime
class ISOFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None):
t = strftime("%Y-%m-%dT%H:%M:%S", gmtime(record.created))
z = strftime("%z",gmtime(record.created))
s = "%s.%03d%s" % (t, record.msecs,z)
return s
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console = logging.StreamHandler()
logger.addHandler(console)
formatter = ISOFormatter(fmt='%(asctime)s - %(module)s - %(levelname)s - %(message)s')
console.setFormatter(formatter)
logger.debug('Jackdaws love my big sphinx of quartz.')
#2020-10-23T17:25:48.310-0800 - <stdin> - DEBUG - Jackdaws love my big sphinx of quartz.

As of now the following works perfectly with python 3 .
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)-8s %(message)s',
datefmt='%Y/%m/%d %H:%M:%S.%03d',
filename=self.log_filepath,
filemode='w')
gives the following output
2020/01/11 18:51:19.011 INFO

How to Customize the time format for Python logging?

I am new to Python's logging package and plan to use it for my project. I would like to customize the time format to my taste. Here is a short code I copied from a tutorial:
import logging
# create logger
logger = logging.getLogger("logging_tryout2")
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter("%(asctime)s;%(levelname)s;%(message)s")
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
# "application" code
logger.debug("debug message")
logger.info("info message")
logger.warn("warn message")
logger.error("error message")
logger.critical("critical message")
And here is the output:
2010-07-10 10:46:28,811;DEBUG;debug message
2010-07-10 10:46:28,812;INFO;info message
2010-07-10 10:46:28,812;WARNING;warn message
2010-07-10 10:46:28,812;ERROR;error message
2010-07-10 10:46:28,813;CRITICAL;critical message
I would like to shorten the time format to just: '2010-07-10 10:46:28', dropping the mili-second suffix. I looked at the Formatter.formatTime, but I am confused.

From the official documentation regarding the Formatter class:
The constructor takes two optional arguments: a message format string and a date format string.
So change
# create formatter
formatter = logging.Formatter("%(asctime)s;%(levelname)s;%(message)s")
to
# create formatter
formatter = logging.Formatter("%(asctime)s;%(levelname)s;%(message)s",
"%Y-%m-%d %H:%M:%S")

Using logging.basicConfig, the following example works for me:
logging.basicConfig(
filename='HISTORYlistener.log',
level=logging.DEBUG,
format='%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
)
This allows you to format & config all in one line. A resulting log record looks as follows:
2014-05-26 12:22:52.376 CRITICAL historylistener - main: History log failed to start

To add to the other answers, here are the variable list from Python Documentation.
Directive Meaning Notes
%a Locale’s abbreviated weekday name.
%A Locale’s full weekday name.
%b Locale’s abbreviated month name.
%B Locale’s full month name.
%c Locale’s appropriate date and time representation.
%d Day of the month as a decimal number [01,31].
%H Hour (24-hour clock) as a decimal number [00,23].
%I Hour (12-hour clock) as a decimal number [01,12].
%j Day of the year as a decimal number [001,366].
%m Month as a decimal number [01,12].
%M Minute as a decimal number [00,59].
%p Locale’s equivalent of either AM or PM. (1)
%S Second as a decimal number [00,61]. (2)
%U Week number of the year (Sunday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Sunday are considered to be in week 0. (3)
%w Weekday as a decimal number [0(Sunday),6].
%W Week number of the year (Monday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Monday are considered to be in week 0. (3)
%x Locale’s appropriate date representation.
%X Locale’s appropriate time representation.
%y Year without century as a decimal number [00,99].
%Y Year with century as a decimal number.
%z Time zone offset indicating a positive or negative time difference from UTC/GMT of the form +HHMM or -HHMM, where H represents decimal hour digits and M represents decimal minute digits [-23:59, +23:59].
%Z Time zone name (no characters if no time zone exists).
%% A literal '%' character.

if using logging.config.fileConfig with a configuration file use something like:
[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=%Y-%m-%d %H:%M:%S

Try These Formats:
Format 1:
'formatters': {
'standard': {
'format' : '%(asctime)s |:| LEVEL: %(levelname)s |:| FILE PATH: %(pathname)s |:| FUNCTION/METHOD: %(funcName)s %(message)s |:| LINE NO.: %(lineno)d |:| PROCESS ID: %(process)d |:| THREAD ID: %(thread)d',
'datefmt' : "%y/%b/%Y %H:%M:%S"
},
}
Output of Format 1:
Format 2:
'formatters': {
'standard': {
'format' : '%(asctime)s |:| LEVEL: %(levelname)s |:| FILE PATH: %(pathname)s |:| FUNCTION/METHOD: %(funcName)s %(message)s |:| LINE NO.: %(lineno)d |:| PROCESS ID: %(process)d |:| THREAD ID: %(thread)d',
'datefmt' : "%Y-%m-%d %H:%M:%S"
},
}
Output of Format 2:

In order to customize time format while logging we can create a logger object and and a fileHandler to it.
import logging
from datetime import datetime
logger = logging.getLogger("OSA")
logger.setLevel(logging.DEBUG)
filename = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + ".log"
fileHandler = logging.FileHandler(filename, mode="a")#'a' for append you can use 'w' for write
formatter = logging.Formatter(
"%(asctime)s : %(levelname)s : [%(filename)s:%(lineno)s - %(funcName)s()] : %(message)s",
"%Y-%m-%d %H:%M:%S")
fileHandler.setFormatter(formatter)
logger.addHandler(fileHandler)