I am using windows 7 and python 2.7.
I want to limit my log file size to 5MB.
My app, when it starts, writes to log file, and then the app terminates.
When my app starts again, it will write in same log file. So app is not continuously running.
App initiates, processes and terminates.
My code for logging is:
import logging
import logging.handlers
logging.basicConfig(filename=logfile.log, level="info", format='%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')
logging.info("*************************************************")
I tried with RotatingFileHandler but it didn't work
logging.handlers.RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024, backupCount=2, encoding=None, delay=0)
So, how can I enforce a file size limit in python?
Lose basicConfig() and use RotatingFileHandler:
import logging
from logging.handlers import RotatingFileHandler
log_formatter = logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')
logFile = 'C:\\Temp\\log'
my_handler = RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024,
backupCount=2, encoding=None, delay=0)
my_handler.setFormatter(log_formatter)
my_handler.setLevel(logging.INFO)
app_log = logging.getLogger('root')
app_log.setLevel(logging.INFO)
app_log.addHandler(my_handler)
while True:
app_log.info("data")
When you use logging.basicConfig with a file, the log is attached with a file handler to handle writing to the file.
afterwards you created another file handler to the same file with
logging.handlers.RotatingFileHandler
Now, once a rotate is needed, RotatingFileHandler is trying to remove the old file but it can't becuase there is an open file handler
this can be seen if you look directly at the log file handlers -
import logging
from logging.handlers import RotatingFileHandler
log_name = 'c:\\log.log'
logging.basicConfig(filename=log_name)
log = logging.getLogger()
handler = RotatingFileHandler(log_name, maxBytes=1024, backupCount=1)
log.addHandler(handler)
[<logging.FileHandler object at 0x02AB9B50>, <logging.handlers.RotatingFileHandler object at 0x02AC1D90>]
To use BasicConfig and RotatingFileHandler, add RotatingFileHandler as Handler in BasicConfig.
main.py:
import logging
rfh = logging.handlers.RotatingFileHandler(
filename='foo.log',
mode='a',
maxBytes=5*1024*1024,
backupCount=2,
encoding=None,
delay=0
)
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(name)-25s %(levelname)-8s %(message)s",
datefmt="%y-%m-%d %H:%M:%S",
handlers=[
rfh
]
)
logger = logging.getLogger('main')
logger.debug("test")
other.py
import logging
class Other():
def __init(self):
self.logger = logging.getLogger('other')
self.logger.info("test2")
"test" will be written into foo.log with the tag 'main'
"test2" will be written into foo.log with the tag 'other'
Related
What I want to do:
To create a Python package, and in that package, create a class that instantiates a logger. I'd create an instance of this class in the main function of my project and whenever I instantiate any package or class, I'd either import the logger class that I created or I'd pass a reference of the instance I created in the main function.
Why I want to do this:
Referential integrity of the logger filename and not having to write
the same lines of code everywhere to instantiate a logger.
To not have the logger as a global.
To be able to perform file operations before instantiating the
logger file. For example, to create a log folder to store the log
files before starting the logger.
To be able to load program parameters from a config file (my config
file manager will be a separate class in a separate package) and to
be able to load logger parameters from the config file.
The code I implemented as of now (which I want to change):
The main file:
import logging
from logging.handlers import RotatingFileHandler
from operatingSystemFunctions import operatingSystemFunc
from diskOperations import fileAndFolderOperations
logFileName = os.path.join('logs', 'logs.log')
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName, maxBytes=2000, backupCount=5)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log.addHandler(handler)
log.setLevel(logging.INFO)
if __name__ == '__main__':
while True:
log.info("Running")
#do something
Similarly, the custom packages I've imported, each have their own global instantiations of logger:
operatingSystemFunc for example, is:
import subprocess
from sys import platform #to check which OS the program is running on
from abc import ABC, abstractmethod
import logging
from logging.handlers import RotatingFileHandler
logFileName = os.path.join('logs', 'logs.log')
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName , maxBytes = 2000 , backupCount = 5)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') #setting this was necessary to get it to write the format to file. Without this, only the message part of the log would get written to the file
log.addHandler(handler)
log.setLevel(logging.INFO)
class AudioNotifier(ABC):#Abstract parent class
#abstractmethod
def __init__(self):
#self.id = "someUniqueNotifierName" #has to be implemented in the child class
pass
Right now the code throws an error FileNotFoundError: [Errno 2] No such file or directory: '/home/nav/prog/logs/logs.log' because I haven't yet created the logs folder.
One concern I have is that if I implement the logger in a separate package and use it by importing that package into all other packages or by passing a reference to an instance of the class to all other classes, then would it be a bad practice?
Update:
Learning from Pablo's answer, I changed the code to ensure that the RotatingFileHandler was simultaneously able to write to a file. So this is the resulting code:
import logging
from logging.handlers import RotatingFileHandler
from operatingSystemFunctions import operatingSystemFunc
from diskOperations import fileAndFolderOperations
logFileName = 'logs.log'
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
#log = logging.getLogger(__name__)
handler = RotatingFileHandler(logFileName, maxBytes=2000, backupCount=2)#TODO: Shift to config file
handler.formatter = logging.Formatter(fmt='%(levelname)s %(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') #setting this was necessary to get it to write the format to file. Without this, only the message part of the log would get written to the file
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)
if __name__ == '__main__':
logging.info("\n\n---------------------------------")
logging.info("program started")
operatingSystemCheck = operatingSystemFunc.OperatingSystemFunc()
logging.info("Monitoring ...")
while True:
pass
and it's used in the packages like this:
import logging
class OperatingSystemFunc:
def __init__(self):
logging.info("Some message")
Few considerations, the python logger is already a singleton that runs in his own thread, so only one instance of the logger is ever made. When you call it, or you try to create it in different places, what you are doing is creating it just the first time and calling the same singleton again and again. So the handlers defined at the first place are accessible everywhere without passing them nor importing them.
If you make a instance of the logger once in your main function using a name, later you will only have to use logging.getLogger("your chosen name"), so in most cases there is no necessity to create a class inside a package etc etc. You can use the handlers, that you have defined elsewhere doing that and without directly importing them.
So I would say that what you are proposing is not a good practice, because it is not necessary, maybe with a few exceptions. If you really want save writing "log = logging.getLogger('...')" which is the only thing you will save, then you can definitely write that line in one package and import the "log" instead the logging module and that will be OK. But then I advice to you not putting the logger inside a class, or if in a class, make the instance in the same file and import the instance, not the class.
its my logger file what i used in my projects:
import logging
class CustomFormatter(logging.Formatter):
grey = "\x1b[38;21m"
green = "\x1b[0;30;42m"
yellow = "\x1b[1;30;43m"
red = "\x1b[0:30;41m"
bold_red = "\x1b[1;30;1m"
reset = "\x1b[0m"
FORMATS = {
logging.DEBUG: f"%(asctime)s - {grey}%(levelname)s{reset} - %(message)s",
logging.INFO: f"%(asctime)s - {green}%(levelname)s{reset} - %(message)s",
logging.WARNING: f"%(asctime)s - {yellow}%(levelname)s{reset} - %(message)s",
logging.ERROR: f"%(asctime)s - {red}%(levelname)s{reset} - %(message)s",
logging.CRITICAL: f"%(asctime)s - {bold_red}%(levelname)s{reset} - %(message)s"
}
def format(self, record):
log_fmt = self.FORMATS.get(record.levelno)
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
logger = logging.getLogger("core")
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(CustomFormatter())
logger.addHandler(ch)
and put it in
_ init _.py
from .logger import logger as LOGGER
and made an ini file like
log.ini
[loggers]
keys=root
[handlers]
keys=logfile,logconsole
[formatters]
keys=logformatter
[logger_root]
level=INFO
handlers=logfile, logconsole
[formatter_logformatter]
format=[%(asctime)s.%(msecs)03d] %(levelname)s [%(thread)d] - %(message)s
[handler_logfile]
class=handlers.RotatingFileHandler
level=INFO
args=('logs.log','a')
formatter=logformatter
[handler_logconsole]
class=handlers.logging.StreamHandler
level=INFO
args=()
formatter=logformatter
these helped me too much, i hope it will help you
I am working on rotating file handler. I need another file to be created once the file size crosses the given maxBytes value. but the code does not seems to working fine
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(message)s",
handlers=[
RotatingFileHandler("./app.log", maxBytes=100, backupCount=5)
],
)
logger = logging.getLogger("DemoLogger")
#router.get('/demo_api/v1')
def default_rd(db: Session = Depends(get_db)):
try:
Category_data = db.query(models.table_new.id, models.table_new.Label).filter(models.table_new.classCode=='Category').all()
except Exception:
logger.exception("Database Error")
else:
logger.warning("Fetched Category data")
A minimal example based on yours happily creates app.log, app.log.1, app.log.2, app.log.3, app.log.4, app.log.5 in the directory the program is run from. Note that /app.log was changed to ./app.log.
If applying changes like these doesn't help, there must be something wrong with other code in your app you aren't showing.
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(message)s",
handlers=[
RotatingFileHandler("./app.log", maxBytes=100, backupCount=5)
],
)
logger = logging.getLogger("DemoLogger")
for x in range(1000):
logger.info("Blah, %s", x)
I have a program in which I wrote logs both as info and debug.
Since the debug contains also calls to slow functions, my program is running slow even if I set debugging to INFO.
Is it possible to completely skip those line from computation?
in the next example 10 seconds have to pass before the info log is executed.
import logging.handlers
import sys
import time
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging_stream_handler = logging.StreamHandler(sys.stdout)
logging_stream_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s --- %(message)s'))
logger.addHandler(logging_stream_handler)
logger.debug("aaa", time.sleep(10))
logger.debug("bbb")
logger.info("ccc")
You could check if the logger is enabled for such a level with the isEnabledFor method:
if logger.isEnabledFor(logging.DEBUG):
logger.debug("aaa", time.sleep(10))
logger.debug("bbb")
You shouldn't have logging inside debug commands. If you must, then in order to skip it you must branch your code.
import logging.handlers
import sys
import time
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging_stream_handler = logging.StreamHandler(sys.stdout)
logging_stream_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s --- %(message)s'))
logger.addHandler(logging_stream_handler)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("aaa", time.sleep(10))
logger.debug("bbb")
logger.info("ccc")
I am trying to use an FTP server stub during tests. I don't want the console output, but I would like to capture the logging to a file.
I want the FTP server to run in a different process, so I use multiprocessing.
My code as follows sets all logging to level WARNING:
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
import pyftpdlib.log as pyftpdliblog
import os
import logging
import multiprocessing as mp
authorizer = DummyAuthorizer()
authorizer.add_user('user', '12345', '.', perm='elradfmwM')
handler = FTPHandler
handler.authorizer = authorizer
pyftpdliblog.LEVEL = logging.WARNING
logging.basicConfig(filename='pyftpd.log', level=logging.INFO)
server = FTPServer(('', 2121), handler)
def main():
p = mp.Process(target=server.serve_forever)
p.start()
if __name__ == '__main__':
main()
How do I set only the console logging to level WARNING, or even better, completely shutdown without giving up the file logging?
So, after digging inside the code, I found out the following hint:
# This is a method of FTPServer and it is called before
# server.serve_forever
def _log_start(self):
if not logging.getLogger('pyftpdlib').handlers:
# If we get to this point it means the user hasn't
# configured logger. We want to log by default so
# we configure logging ourselves so that it will
# print to stderr.
from pyftpdlib.ioloop import _config_logging
_config_logging()
So, all I had to do is to define my own appropriate handlers:
logger = logging.getLogger('pyftpdlib')
logger.setLevel(logging.INFO)
hdlr = logging.FileHandler('pyftpd.log' )
logger.addHandler(hdlr)
Now, there is file logging, but console logging will not start.
Something like this:
import logging
date_format = "%Y/%m/%d %H:%M:%S"
log_file_path = "my_file.txt"
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
# own_module_logger = logging.getLogger(__name__)
pyftpdlib_logger = logging.getLogger("pyftpdlib")
# Setup logging to file (Only pyftpdlib)
filehandler = logging.FileHandler(filename = log_file_path)
filehandler.setLevel(logging.DEBUG)
fileformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
filehandler.setFormatter(fileformatter)
pyftpdlib_logger.addHandler(filehandler)
pyftpdlib_logger.propagate = False
# Setup logging to console (All other)
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
console.setFormatter(consoleformatter)
root_logger.addHandler(console)
# Do your loggings
a = logging.getLogger()
a.info('root I')
a.debug('root D')
b = logging.getLogger("pyftpdlib")
b.info('P I')
b.debug('P D')
logging.shutdown()
So loggings of pyftpdlib go to file. Everything from your module to console. One of the key thing here is the propagate!
In my programs, I generally want to log to both a file and to the screen. If I import a module, I want the calls to logger within that module's functions/classes to also result in a log to a file and the screen. Test_module_A does this, and test_module_B logs to the file but not the screen. Is this because the loggers in the modules propagate up to the root logger which through basicConfig is only setup to log to the file? I just want to make sure I'm using logging properly, as I am going to rewrite all my code to use this type of logging rather than "print." Going forward, I am only going to use logging and not print, as I figure logging is more flexible. Is this a good idea?
main.py
import logging
logger = logging.getLogger(__name__)
task_file='temp.txt'
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
logging.basicConfig(filename=task_file, format=format_str, datefmt=datefmt_str,
level=logging.INFO)
console = logging.StreamHandler()
#console.setLevel(logging.INFO)
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("from main.py")
import test_module_A
import test_module_B
test_module_A.py
import logging
logger = logging.getLogger(__name__)
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
console = logging.StreamHandler()
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("upon import of test_module_A")
test_module_B.py
import logging
logger = logging.getLogger(__name__)
logger.info("upon import of test_module_B")
Upon running main.py:
#screen output
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
# test.txt
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
05/06/2014 12:36:33AM test_module_B INFO: upon import of test_module_B
It's not good practice to configure logging (add handlers, etc.) in each module: the way to do it is to just have
logger = logging.getLogger(__name__)
and actual logging calls in each module as required, and only have the configuration done in one place (called from the main script early on)
if __name__ == '__main__':
logging.basicConfig(level=..., format=..., ...) #or other configuration code
handler = logging.FileHandler(...)
# set formatters, filters etc.
logging.getLogger().addHandler(handler) # add to root logger
main()
The basicConfig() call adds to the root logger a handler which writes to sys.stderr, and together with the FileHandler added explicitly, the logs from all modules should be written to both screen and file.