I have a python program that is writing to a log file that is being rotated by Linux's logrotate command. When this happens I need to signal my program to stop writing to the old file and start writing to the new one. I can handle the signal but how do I tell python to write to the new file?
I am opening the file like this:
logging.basicConfig(format='%(asctime)s:%(filename)s:%(levelname)s:%(message)s',filename=log_file, level=logging.INFO)
and writing to it like this:
logging.log(level,"%s" % (msg))
The logging modules look very powerful but also overwhelming. Thanks.
Don't use logging.basicConfig, use WatchedFileHandler. Here's how to use it.
import time
import logging
import logging.handlers
def log_setup():
log_handler = logging.handlers.WatchedFileHandler('my.log')
formatter = logging.Formatter(
'%(asctime)s program_name [%(process)d]: %(message)s',
'%b %d %H:%M:%S')
formatter.converter = time.gmtime # if you want UTC time
log_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(log_handler)
logger.setLevel(logging.DEBUG)
log_setup()
logging.info('Hello, World!')
import os
os.rename('my.log', 'my.log-old')
logging.info('Hello, New World!')
You may want to look at WatchedFileHandler to implement this, or as an alternative, implement log rotation with RotatingFileHandler, both of which are in the logging.handlers module.
from logging import handlers
handler = handlers.TimedRotatingFileHandler(filename, when=LOG_ROTATE)
handler.setFormatter(logging.Formatter(log_format, datefmt="%d-%m-%Y %H:%M:%S"))
#LOG_ROTATE = midnight
#set your log format
This should help you in handling rotating log
Since rotation is already being done by logrotate, in your signal handler you should just call logging.basicConfig(...) again and that should reopen the log file.
Related
I have a main script and multiple modules. Right now I have logging setup where all the logging from all modules go into the same log file. It gets hard to debug when its all in one file. So I would like to separate each module into its own log file. I would also like to see the requests module each module uses into the log of the module that used it. I dont know if this is even possible. I searched everywhere and tried everything I could think of to do it but it always comes back to logging everything into one file or setup logging in each module and from my main module initiate the script instead of import as a module.
main.py
import logging, logging.handlers
import other_script.py
console_debug = True
log = logging.getLogger()
def setup_logging():
filelog = logging.handlers.TimedRotatingFileHandler(path+'logs/api/api.log',
when='midnight', interval=1, backupCount=3)
filelog.setLevel(logging.DEBUG)
fileformatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
filelog.setFormatter(fileformatter)
log.addHandler(filelog)
if console_debug:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(name)-15s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
log.addHandler(console)
if __name__ == '__main__':
setup_logging()
other_script.py
import requests
import logging
log = logging.getLogger(__name__)
One very basic concept of python logging is that every file, stream or other place that logs go is equivalent to one Handler. So if you want every module to log to a different file you will have to give every module it's own handler. This can also be done from a central place. In your main.py you could add this to make the other_script module log to a separate file:
other_logger = logging.getLogger('other_script')
other_logger.addHandler(logging.FileHandler('other_file'))
other_logger.propagate = False
The last line is only required if you add a handler to the root logger. If you keep propagate at the default True you will have all logs be sent to the root loggers handlers too. In your scenario it might be better to not even use the root logger at all, and use a specific named logger like getLogger('__main__') in main.
I am trying to add in multiple log files into a Logs folder, but instead of having to change the code each time you start the program, i want to make the log file's name "Log(the time).log". I'm using logger at the moment, but i can switch. I've also imported time.
Edit: Here is some of the code i am using:
import logging
logger = logging.getLogger('k')
hdlr = logging.FileHandler('Path to the log file/log.log')
formatter = logging.Formatter('At %(asctime)s, KPY returned %(message)s at level %(levelname)s
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
logger.info('hello')
import logging
import time
fname = "Log({the_time}).log".format(the_time=time.time())
logging.basicConfig(level=logging.DEBUG, filename=fname)
logging.info('hello')
You should do it when you set the FileHandler for the logging object. Use datetime instead of time so that you can include the date for each instance of the log in order to differentiate logs on different days at the same time.
fh = logging.FileHandler("Log"+str(datetime.datetime.now())+'.log')
fh.setLevel(logging.DEBUG)
I got help from another website.
You have to change the hdlr to:
({FOLDER LOCATION}/Logs/log{}.log'.format(datetime.datetime.strftime(datetime.datetime.now(), '%Y%m%d%H%M%S_%f')))
I want to make log file in python same as in log4j,
meaning as soon the logger.log file get's to a size of 1K make a copy of this file and call it logger(1).log , In case logger(1),log already exists create logger(2).log and of course delete logger.log so next time it will run it will start a clean log.
This is my code but it is good only for first creation of logger file bakup:
b = os.path.getsize('logger.log')
print b
if b >= 1000:
shutil.copy2('logger.log', 'logger(1).log')
This is my log.py file so it can be used globally:
import os
import logging
from logging.config import fileConfig
from logging import handlers
def setup_custom_logger():
configFolder = os.getcwd() + os.sep + 'Conf'
fileConfig(configFolder + os.sep + 'logging_config.ini')
logger = logging.getLogger()
# create a file handler
handler = logging.handlers.RotatingFileHandler('logger.log', maxBytes=1024, encoding="UTF-8")
handler.doRollover()
# create a logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
You need to setup a RotatingFileHandler:
import logging
from logging import handlers
logger = logging.getLogger(__name__)
handler = handlers.RotatingFileHandler('logger.log', maxBytes=1000, backupCount=10, encoding="UTF-8")
handler.doRollover()
logger.addHandler(handler)
From the documentation:
You can use the maxBytes and backupCount values to allow the file to
rollover at a predetermined size. When the size is about to be
exceeded, the file is closed and a new file is silently opened for
output. Rollover occurs whenever the current log file is nearly
maxBytes in length.
You can use a RotatingFileHandler.
Such a handler can be added by doing something like this:
import logging
logger = logging.getLogger(__name__)
logger.addHandler(RotatingFileHandler(filename, maxBytes=1024, backupCount=10))
Once the log file reaches this size, a rollover will be done and the old log file will be saved with a name filename.log.1, filename.log.2 etc. till filename.log.10.
Try using python logging module with TimedRotatingFileHandler handler.
# Logging
cur_flname = os.path.splitext(os.path.basename(__file__))[0]
LOG_FILENAME = constants.log_dir + os.sep + 'Log_' + cur_flname + '.txt'
util.make_dir_if_missing(constants.log_dir)
logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO, filemode='w',
format='%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s',
datefmt="%m-%d %H:%M") # Logging levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL
# Output to screen
logger = logging.getLogger(cur_flname)
logger.addHandler(logging.StreamHandler())
I use the code above to create a logger that I can use in my module to print messages to screen as well simultaneously to a text file.
On Windows, the messages get output to both file and screen. However, on Mac OS X 10.9.5, they only get output to file. I am using Python 2.7.
Any ideas on how to fix this?
From your question it is clear, you have no problem with creating logger name,
log file name and with logging to a file, so I will keep this part simplified to keep my code succinct.
First thing: To me your solution seems correct as logging.StreamHandler shall
send output to sys.stderr by default. You might have some code around (not
shown in your question), which is modifying sys.stderr, or you are running
your code on OSX in such a way, that output to stderr is not shown (but is
really output).
Solution with logging
Put following code into with_logging.py:
import logging
import sys
logformat = "%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s"
datefmt = "%m-%d %H:%M"
logging.basicConfig(filename="app.log", level=logging.INFO, filemode="w",
format=logformat, datefmt=datefmt)
stream_handler = logging.StreamHandler(sys.stderr)
stream_handler.setFormatter(logging.Formatter(fmt=logformat, datefmt=datefmt))
logger = logging.getLogger("app")
logger.addHandler(stream_handler)
logger.info("information")
logger.warning("warning")
def fun():
logger.info(" fun inf")
logger.warning("fun warn")
if __name__ == "__main__":
fun()
Run it: $ python with_logging.py and you shall see expected log records (properly formatted)
in log file app.log and on stderr too.
Note, if you do not see it on stderr, something is hiding your stderr
output. To see something, change in StreamHandler the stream to sys.stdout.
Solution with logbook
logbook is python package providing alternative logging means. I am showing it here to show main
difference to stdlib logging: with logbook the use and configuration seems simpler to me.
Previous solution rewritten to with_logbook.py
import logbook
import sys
logformat = ("{record.time:%m-%d %H:%M} {record.level_name} {record.module} - "
"{record.func_name}: {record.message}")
kwargs = {"level": logbook.INFO, "format_string": logformat, "bubble": True}
logbook.StreamHandler(sys.stderr, **kwargs).push_application()
logbook.FileHandler("app.log", **kwargs).push_application()
logger = logbook.Logger("app")
logger.info("information")
logger.warning("warning")
def fun():
logger.info(" fun inf")
logger.warning("fun warn")
if __name__ == "__main__":
fun()
The main difference is, that you do not have to attach handlers to loggers, your loggers simply emit
some log records.
These records will be handled by handlers, if they are put in place. One method
is to create a handler and call push_application() on it. This will put the
handler into stack of handlers in use.
As before, we need two handlers, one for the file, another for stderr.
If you run this script $ python with_logbook.py, you shall see exactly the same results as
with logging.
Separate logging setup from module code: short_logbook.py
With stdlib logging I reall do not like the introductory dances to make the logger work. I want
simply emit some log records and want to do that as simply as possible.
Following example is modification of previous one. Instead of setting up
logging on very beginning of the module, I do it just before the code is run
- inside the if __name__ == "__main__" (you may do the same on any other
place).
For practical reasons, I separated the module and calling code to two files:
File funmodule.py
from logbook import Logger
log = Logger(__name__)
log.info("information")
log.warning("warning")
def fun():
log.info(" fun inf")
log.warning("fun warn")
You can notice, that there are really only two lines of code related to making
logging available.
Then, create the calling code, put into short_logbook.py:
import sys
import logbook
if __name__ == "__main__":
logformat = ("{record.time:%m-%d %H:%M} {record.level_name} {record.module}"
"- {record.func_name}: {record.message}")
kwargs = {"level": logbook.INFO, "format_string": logformat, "bubble": True}
logbook.StreamHandler(sys.stderr, **kwargs).push_application()
logbook.FileHandler("app.log", **kwargs).push_application()
from funmodule import fun
fun()
Running the code you will see it working the same way as before, only logger name will be funmodule.
Note, that I did the from funmodule import fun after the logging was set up. If I did it on the
top if the short_logbook.py file, the first two log records from funmodule.py would not be seen
as they happen during module import.
EDIT: added another logging solution to have fair comparison
One more stdlib logging solution
Trying to have fair comparison of logbook and logging I rewrote final logbook example to
logging.
funmodule_logging.py looks like:
import logging
log = logging.getLogger(__name__)
log.info("information")
log.warning("warning")
def fun():
log.info(" fun inf")
log.warning("fun warn")
log.error("no fun at all")
and short_logging.py looks as follows:
import sys
import logging
if __name__ == "__main__":
logformat = "%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s"
datefmt = "%m-%d %H:%M"
logging.basicConfig(filename="app.log", level=logging.INFO, filemode="w",
format=logformat, datefmt=datefmt)
stream_handler = logging.StreamHandler(sys.stderr)
stream_handler.setFormatter(logging.Formatter(fmt=logformat, datefmt=datefmt))
logger = logging.getLogger("funmodule_logging")
logger.addHandler(stream_handler)
from funmodule_logging import fun
fun()
Functionality is very similar.
I still strugle with logging. stdlib logging is not easy to grasp, but it is in stdlib and offers
some nice things like logging.config.dictConfig allowing to configure logging by a dictionary.
logbook was much simpler to start with, but is a bit slower at the moment and lacks the
dictConfig. Anyway, these differences are not relevant to your question.
I've read through the logging module documentation and whilst I may have missed something obvious, the code I've got doesn't appear to be working as intended. I'm using Python 2.6.4.
My program consists of several different python files, from which I want to send logging messages to a text file and, potentially, the screen. I imagine this is a common thing to do so I'm messing this up somewhere.
What my code is doing at the minute is logging to the text file correctly, kinda. But logging to the screen is being duplicated, one with the specified formatting, and one without. Also, when I turn off the screen output, I'm still getting the text printed once, which I don't want - I just want it to be logged to the file.
Anyway, some code:
#logger.py
import logging
from logging.handlers import RotatingFileHandler
import os
def setup_logging(logdir=None, scrnlog=True, txtlog=True, loglevel=logging.DEBUG):
logdir = os.path.abspath(logdir)
if not os.path.exists(logdir):
os.mkdir(logdir)
log = logging.getLogger('stumbler')
log.setLevel(loglevel)
log_formatter = logging.Formatter("%(asctime)s - %(levelname)s :: %(message)s")
if txtlog:
txt_handler = RotatingFileHandler(os.path.join(logdir, "Stumbler.log"), backupCount=5)
txt_handler.doRollover()
txt_handler.setFormatter(log_formatter)
log.addHandler(txt_handler)
log.info("Logger initialised.")
if scrnlog:
console_handler = logging.StreamHandler()
console_handler.setFormatter(log_formatter)
log.addHandler(console_handler)
Nothing unusual there.
#core.py
import logging
corelog = logging.getLogger('stumbler.core') # From what I understand of the docs, this should work :/
class Stumbler:
[...]
corelog.debug("Messages and rainbows...")
The screen output shows how this is being duplicated:
2010-01-08 22:57:07,587 - DEBUG :: SCANZIP: Checking zip contents, file: testscandir/testdir1/music.mp3
DEBUG:stumbler.core:SCANZIP: Checking zip contents, file: testscandir/testdir1/music.mp3
2010-01-08 22:57:07,587 - DEBUG :: SCANZIP: Checking zip contents, file: testscandir/testdir2/subdir/executable.exe
DEBUG:stumbler.core:SCANZIP: Checking zip contents, file: testscandir/testdir2/subdir/executable.exe
Although the textfile is getting the correctly formatted output, turning the screen logging off in logger.py still has the incorrectly formatted output displayed.
From what I understand of the docs, calling corelog.debug(), seeing as corelog is a child of the "stumbler" logger, it should use that formatting and output the logs as such.
Apologies for the essay over such a trivial issue.
TL;DR: How do I do logging from multiple files?
Are you sure no other logging setup is being done in anything you import.
The incorrect output in your console logs look like the default configuration for a logger, so something else may be setting that up.
Running this quick test script:
import logging
from logging.handlers import RotatingFileHandler
import os
def setup_logging(logdir=None, scrnlog=True, txtlog=True, loglevel=logging.DEBUG):
logdir = os.path.abspath(logdir)
if not os.path.exists(logdir):
os.mkdir(logdir)
log = logging.getLogger('stumbler')
log.setLevel(loglevel)
log_formatter = logging.Formatter("%(asctime)s - %(levelname)s :: %(message)s")
if txtlog:
txt_handler = RotatingFileHandler(os.path.join(logdir, "Stumbler.log"), backupCount=5)
txt_handler.doRollover()
txt_handler.setFormatter(log_formatter)
log.addHandler(txt_handler)
log.info("Logger initialised.")
if scrnlog:
console_handler = logging.StreamHandler()
console_handler.setFormatter(log_formatter)
log.addHandler(console_handler)
setup_logging('/tmp/logs')
corelog = logging.getLogger('stumbler.core')
corelog.debug("Messages and rainbows...")
yields this result:
2010-01-08 15:39:25,335 - DEBUG ::
Messages and rainbows...
and in my /tmp/logs/Stumbler.log
2010-01-08 15:39:25,335 - INFO ::
Logger initialised. 2010-01-08
15:39:25,335 - DEBUG :: Messages and
rainbows...
This worked as expected when I ran it in python 2.4, 2.5, and 2.6.4