I'm trying to understand the python logging module and am currently running into an issue. I'm adding lines to my own code to log to a file, but I'm using the discord bot module that also outputs data to the console.
I'd like to log both my own stuff and the module logs to a file, creating a new file every day.
I can output my own logs to a file using the TimedRotatingFileHandler, then add that same filename to basicConfig, but then my own logs get into the file 2x for each log action I do.
I can also get everything to log to the file in basicConfig, but I have no idea how to create a TimedRotationFileHandler to that, so I don't know how to make a log file for each day using that method.
Anyone who can help me out with this? Thanks a bunch!
from logging.handlers import TimedRotatingFileHandler
def createLog(name):
logging.basicConfig(level=logging.DEBUG, filename=name, filemode="a", format="%(asctime)s - %(levelname)s - %(message)s")
logfile = logging.getLogger(name)
if (logfile.hasHandlers()):
logfile.handlers.clear()
else:
print("No handlers")
handler = TimedRotatingFileHandler(name, when="midnight", interval=1)
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
handler.suffix = "%Y%m%d"
logfile.addHandler(handler)
logfile.setLevel(logging.DEBUG)
return logfile```
LOG_DIR_NAME = {log file name}
MAX_DAYS = { maximum number of days }
formatter = logging.Formatter('%(asctime)s | %(lineno)3d | %(levelname)5s | %(message)s')
logHandler = handlers.TimedRotatingFileHandler(LOG_DIR_NAME + logfile_name, when="midnight", interval=1, backupCount=MAX_DAYS)
Hope it will be helpful to you
or you can also check this answerPython: How to create log file everyday using logging module?
Related
I have a file named helper.py
import logging
import os
from json import load
def get_config(value):
with open('config.json','r') as f:
result=load(f)[value]
return result
def get_logger(name,level):
logpath=get_config("log_path")
if not os.path.exists(logpath):
os.mkdir(logpath)
logger = logging.getLogger(name)
if not bool(logger.handlers):
formatter = logging.Formatter('%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',datefmt='%Y-%m-%d %H:%M:%S')
fh = logging.FileHandler(os.path.join(logpath,f'{get_config("log_file_name")}.log'),mode="w",encoding='utf-8')
fh.setFormatter(formatter)
logger.addHandler(fh)
ch = logging.StreamHandler()
ch.setFormatter(formatter)
ch.setLevel(level)
logger.addHandler(ch)
return logger
LOGGER=get_logger("MyLogger",logging.INFO)
This is config.json:
{
"save_path" : "results/",
"log_path" : "logs/",
"log_file_name" : "MyLog"
}
let's say I am using LOGGER from helper it in x.py
from helper import LOGGER
logger=LOGGER
def div(x,y):
try:
logger.info("inside div")
return x/y
except Exception as e:
logger.error(f"div failed due to {e.message if 'message' in dir(e) else e}")
I am using this LOGGER in other files by importing helper.LOGGER for logging purposes but it's not printing anything on the console nor writing in a log file
My attempt:
I tried adding sys.stdout in StreamHandler() It doesn't worked
Then I tried setting the level of fh but nothing works
I tried adding basicConfig() instead of fileHandler() but then printing to console using print() and the output of logs is not coming in the correct order
Kindly let me know where I go wrong
Any help is appreciated :)
Thanks :)
You are not setting the level on the LOGGER, which by default is warning. This is why your info level log is not appearing. The Python documentation has a flow chart illustrating when a log will be logged: https://docs.python.org/3/howto/logging.html#logging-flow
The first thing it does, is that before a logger sends a log to their handlers it checks if the level is enabled for the logger. You should add logger.setLevel(level) in your get_logger().
I am trying to add in multiple log files into a Logs folder, but instead of having to change the code each time you start the program, i want to make the log file's name "Log(the time).log". I'm using logger at the moment, but i can switch. I've also imported time.
Edit: Here is some of the code i am using:
import logging
logger = logging.getLogger('k')
hdlr = logging.FileHandler('Path to the log file/log.log')
formatter = logging.Formatter('At %(asctime)s, KPY returned %(message)s at level %(levelname)s
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
logger.info('hello')
import logging
import time
fname = "Log({the_time}).log".format(the_time=time.time())
logging.basicConfig(level=logging.DEBUG, filename=fname)
logging.info('hello')
You should do it when you set the FileHandler for the logging object. Use datetime instead of time so that you can include the date for each instance of the log in order to differentiate logs on different days at the same time.
fh = logging.FileHandler("Log"+str(datetime.datetime.now())+'.log')
fh.setLevel(logging.DEBUG)
I got help from another website.
You have to change the hdlr to:
({FOLDER LOCATION}/Logs/log{}.log'.format(datetime.datetime.strftime(datetime.datetime.now(), '%Y%m%d%H%M%S_%f')))
I am sorry to ask, as it is not the first time, but I have not been able to fix it my self by looking at older posts.
Im trying to create and write to a logfile using python. I am creating the file just fine, but there is no input to the file that I am making.
here is my code:
import logging
log_file = '/home/user/Test.log'
form = "%(asctime)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=log_file, filemode='w', level=logging.DEBUG , format=form)
logging.debug = ('DEBUG MESSAGE!')
Well you need to change:
logging.debug = ('DEBUG MESSAGE!')
to
logging.debug('DEBUG MESSAGE!')
This way it should work!
Following the question here: How do I log from my Python Spark script, I have been struggling to get:
a) All output into a log file.
b) Writing out to a log file from pyspark
For a) I use the following changes to the config file:
# Set everything to be logged to the console
log4j.rootCategory=ALL, file
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/home/xxx/spark-1.6.1/logging.log
log4j.appender.file.MaxFileSize=5000MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
This produces output and now for b) I would like to add my own input to logging from pyspark, but I cannot find any output written to the logs. Here is the code I am using:
import logging
logger = logging.getLogger('py4j')
#print(logger.handlers)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)
logger.addHandler(sh)
logger.info("TESTING.....")
I can find output in the logfile, but no "TESTING...." I have also tried using the existing logger stream but this does not work either.
import logging
logger = logging.getLogger('py4j')
logger.info("TESTING.....")
Works in my configuration:
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger(__name__)
LOGGER.info("Hello logger...")
All output into a log file & Writing out to a log file from pyspark
import os
import sys
import logging
import logging.handlers
log = logging.getLogger(__name_)
handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error
sys.stdout.write = log.info
(will log every error in "spam.log" in the same directory, nothing will be on console/stdout)
(will log every info in "spam.log" in the same directory,nothing will be on console/stdout)
to print output error/info in both file as well as in console remove above two line.
Happy Coding Cheers!!!
I'd like to use the "logging" module in Python to write errors to a log file. However, I want the file to only be created when there are errors. I use the following code:
import logging
f = 'test.conf'
logger = logging.getLogger("test_logger")
logger.setLevel(logging.INFO)
ch_file = logging.FileHandler("test_logger.conf")
ch_file.setLevel(logging.ERROR)
logger.addHandler(ch_file)
ch_file.close()
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.info("info")
logger.warn("warning")
#logger.error("error")
When logger.error("error") is uncommented, I expect the file "test_logger.conf" to be made with the error in it. However, when the line is commented out, I find that the test_logger.conf file is still made and is empty. How can I make it so this file is NOT made unless there are errors to report?
Thanks.
You're in luck. The FileHandler has a delay parameter designed for this purpose:
ch_file = logging.FileHandler("test_logger.conf",delay=True)