I am trying to add in multiple log files into a Logs folder, but instead of having to change the code each time you start the program, i want to make the log file's name "Log(the time).log". I'm using logger at the moment, but i can switch. I've also imported time.
Edit: Here is some of the code i am using:
import logging
logger = logging.getLogger('k')
hdlr = logging.FileHandler('Path to the log file/log.log')
formatter = logging.Formatter('At %(asctime)s, KPY returned %(message)s at level %(levelname)s
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
logger.info('hello')
import logging
import time
fname = "Log({the_time}).log".format(the_time=time.time())
logging.basicConfig(level=logging.DEBUG, filename=fname)
logging.info('hello')
You should do it when you set the FileHandler for the logging object. Use datetime instead of time so that you can include the date for each instance of the log in order to differentiate logs on different days at the same time.
fh = logging.FileHandler("Log"+str(datetime.datetime.now())+'.log')
fh.setLevel(logging.DEBUG)
I got help from another website.
You have to change the hdlr to:
({FOLDER LOCATION}/Logs/log{}.log'.format(datetime.datetime.strftime(datetime.datetime.now(), '%Y%m%d%H%M%S_%f')))
Related
I'm trying to understand the python logging module and am currently running into an issue. I'm adding lines to my own code to log to a file, but I'm using the discord bot module that also outputs data to the console.
I'd like to log both my own stuff and the module logs to a file, creating a new file every day.
I can output my own logs to a file using the TimedRotatingFileHandler, then add that same filename to basicConfig, but then my own logs get into the file 2x for each log action I do.
I can also get everything to log to the file in basicConfig, but I have no idea how to create a TimedRotationFileHandler to that, so I don't know how to make a log file for each day using that method.
Anyone who can help me out with this? Thanks a bunch!
from logging.handlers import TimedRotatingFileHandler
def createLog(name):
logging.basicConfig(level=logging.DEBUG, filename=name, filemode="a", format="%(asctime)s - %(levelname)s - %(message)s")
logfile = logging.getLogger(name)
if (logfile.hasHandlers()):
logfile.handlers.clear()
else:
print("No handlers")
handler = TimedRotatingFileHandler(name, when="midnight", interval=1)
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
handler.suffix = "%Y%m%d"
logfile.addHandler(handler)
logfile.setLevel(logging.DEBUG)
return logfile```
LOG_DIR_NAME = {log file name}
MAX_DAYS = { maximum number of days }
formatter = logging.Formatter('%(asctime)s | %(lineno)3d | %(levelname)5s | %(message)s')
logHandler = handlers.TimedRotatingFileHandler(LOG_DIR_NAME + logfile_name, when="midnight", interval=1, backupCount=MAX_DAYS)
Hope it will be helpful to you
or you can also check this answerPython: How to create log file everyday using logging module?
I am trying to set loggers for my python code, I want to set the level of the log from the configuration file. But unable to do by me. Here the code is given below, If you noticed that in the given below code can see logger.setLevel(logging.INFO). I don't want to directly mention as a hardcoded value logging.INFO. Need to get this from the config file, is it possible?
import logging
from logging.config import fileConfig
from datetime import date
class Log:
#staticmethod
def trace():
today = date.today()
# dd/mm/YY
d1 = today.strftime("%d_%m_%Y")
# Gets or creates a logger
logger = logging.getLogger(__name__)
# set log level
logger.setLevel(logging.INFO)
# define file handler and set formatter
file_handler = logging.FileHandler('log/'+d1+'_logfile.log')
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(name)s : %(message)s')
file_handler.setFormatter(formatter)
# add file handler to logger
logger.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
return logger
You can always use Python built-in Configuration file parser
Have the log levels in a config file and read that value. Since that value will be in string, you can define the dictionary mapping in your code. See below for an example.
import configparser
config= configparser.ConfigParser()
config.read('configfile')
log_level_info = {'logging.DEBUG': logging.DEBUG,
'logging.INFO': logging.INFO,
'logging.WARNING': logging.WARNING,
'logging.ERROR': logging.ERROR,
}
print(config['DEFAULT']['LOG_LEVEL'])
my_log_level_from_config = config['DEFAULT']['LOG_LEVEL']
my_log_level = log_level_info.get(my_log_level_from_config, logging.ERROR)
logger.setLevel(my_log_level)
Your config file would be like below:
user#Inspiron:~/code/advanced_python$ cat configfile
[DEFAULT]
LOG_LEVEL = logging.INFO
user#Inspiron:~/code/advanced_python$
If I understood correctly, you need a way to set your logging level at runtime instead of a hard-coded value. I would say you have two options.
The first solution would be to parse your configuration file, and set the level of logging accordingly. If you don't want to parse it everytime the Log class is invoked, in your main you can set a variable that you pass to the Log class.
The second one, that I also suggest, would be to set handlers with python logging class https://docs.python.org/3/library/logging.config.html
logging level (logging.INFO) is an integer value. can you pass numbers from your config file to set log level
print(logging.INFO)
print(logging.WARN)
print(logging.DEBUG)
print(logging.ERROR)
20
30
10
40
Following the question here: How do I log from my Python Spark script, I have been struggling to get:
a) All output into a log file.
b) Writing out to a log file from pyspark
For a) I use the following changes to the config file:
# Set everything to be logged to the console
log4j.rootCategory=ALL, file
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/home/xxx/spark-1.6.1/logging.log
log4j.appender.file.MaxFileSize=5000MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
This produces output and now for b) I would like to add my own input to logging from pyspark, but I cannot find any output written to the logs. Here is the code I am using:
import logging
logger = logging.getLogger('py4j')
#print(logger.handlers)
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.DEBUG)
logger.addHandler(sh)
logger.info("TESTING.....")
I can find output in the logfile, but no "TESTING...." I have also tried using the existing logger stream but this does not work either.
import logging
logger = logging.getLogger('py4j')
logger.info("TESTING.....")
Works in my configuration:
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger(__name__)
LOGGER.info("Hello logger...")
All output into a log file & Writing out to a log file from pyspark
import os
import sys
import logging
import logging.handlers
log = logging.getLogger(__name_)
handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error
sys.stdout.write = log.info
(will log every error in "spam.log" in the same directory, nothing will be on console/stdout)
(will log every info in "spam.log" in the same directory,nothing will be on console/stdout)
to print output error/info in both file as well as in console remove above two line.
Happy Coding Cheers!!!
I have two log handlers in my code: a StreamHandler to write INFO level logs from the same module to stdout, and a FileHandler to write more verbose, DEBUG logs to a file. This is my code:
import sys
import logging
log = logging.getLogger('mymodule')
log.setLevel(logging.DEBUG)
logf = logging.FileHandler('file.log')
logf.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
log.addHandler(logf)
logs = logging.StreamHandler(sys.stdout)
logs.setLevel(logging.INFO)
logs.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
log.addHandler(logs)
However, I want the FileHandler to also write DEBUG info from other modules. I can achieve this if I remove the name from logging.getLogger(), but this will also affect my StreamHandler, which I only want to print output from my own module.
So is there a way to have either of the handlers use a different name?
There's a name attribute available on the base Handler class. I took a look at the Python source and doesn't look like it's used internally so you could manually set each handler to a different name.
I have a python program that is writing to a log file that is being rotated by Linux's logrotate command. When this happens I need to signal my program to stop writing to the old file and start writing to the new one. I can handle the signal but how do I tell python to write to the new file?
I am opening the file like this:
logging.basicConfig(format='%(asctime)s:%(filename)s:%(levelname)s:%(message)s',filename=log_file, level=logging.INFO)
and writing to it like this:
logging.log(level,"%s" % (msg))
The logging modules look very powerful but also overwhelming. Thanks.
Don't use logging.basicConfig, use WatchedFileHandler. Here's how to use it.
import time
import logging
import logging.handlers
def log_setup():
log_handler = logging.handlers.WatchedFileHandler('my.log')
formatter = logging.Formatter(
'%(asctime)s program_name [%(process)d]: %(message)s',
'%b %d %H:%M:%S')
formatter.converter = time.gmtime # if you want UTC time
log_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(log_handler)
logger.setLevel(logging.DEBUG)
log_setup()
logging.info('Hello, World!')
import os
os.rename('my.log', 'my.log-old')
logging.info('Hello, New World!')
You may want to look at WatchedFileHandler to implement this, or as an alternative, implement log rotation with RotatingFileHandler, both of which are in the logging.handlers module.
from logging import handlers
handler = handlers.TimedRotatingFileHandler(filename, when=LOG_ROTATE)
handler.setFormatter(logging.Formatter(log_format, datefmt="%d-%m-%Y %H:%M:%S"))
#LOG_ROTATE = midnight
#set your log format
This should help you in handling rotating log
Since rotation is already being done by logrotate, in your signal handler you should just call logging.basicConfig(...) again and that should reopen the log file.