Trying to level up my devOps skill I put log where I want/need to my code.
Catching an env variable I can setup if I want DEBUG/INFO log (dev) on the standard output or WARNING and above (prod) on a file.
But in python I didn't find how to set a logger conf once (in the main file ?) and use it to the whole project without having to re-write everything or transfer the logging object everywhere. I'm pretty sure I'm missing something.
EDIT : I made a log.py file that looks like this
import os
import logging
from dotenv import load_dotenv
from utils import get_timestamp
def get_logger():
load_dotenv(".env")
env_dev = os.getenv('ENV_DEV', "development")
logger = logging.getLogger(__name__)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
if env_dev == "prod":
handler = logging.FileHandler(f'log/{get_timestamp("%Y%m%d")}_app.log')
handler.setLevel(logging.WARNING)
handler.setFormatter(log_format)
logger.addHandler(handler)
else: # DEV
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
handler.setFormatter(log_format)
logger.addHandler(handler)
return logger
And I use it like :
from log.logging import get_logger
# On Dev env
logger = get_logger()
logger.info("Do stuff")
...
But I have no error nor log on my term.
You don't need to transfer the logging object. When you configure or name a logger once it is globally available. So in your main file you would set up the logger and in all other places just use it.
main file
import logging
logger = logging.getLogger('mylog')
if debug:
mylog.setLevel(logging.DEBUG)
mylog.addHandler(...)
# do all your setup
logger.log("log that") # use logger
other file
import logging
logger = logging.getLogger('mylog')
logger.log("log this") # use logger, it is already configured
Related
I am trying to set loggers for my python code, I want to set the level of the log from the configuration file. But unable to do by me. Here the code is given below, If you noticed that in the given below code can see logger.setLevel(logging.INFO). I don't want to directly mention as a hardcoded value logging.INFO. Need to get this from the config file, is it possible?
import logging
from logging.config import fileConfig
from datetime import date
class Log:
#staticmethod
def trace():
today = date.today()
# dd/mm/YY
d1 = today.strftime("%d_%m_%Y")
# Gets or creates a logger
logger = logging.getLogger(__name__)
# set log level
logger.setLevel(logging.INFO)
# define file handler and set formatter
file_handler = logging.FileHandler('log/'+d1+'_logfile.log')
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(name)s : %(message)s')
file_handler.setFormatter(formatter)
# add file handler to logger
logger.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
return logger
You can always use Python built-in Configuration file parser
Have the log levels in a config file and read that value. Since that value will be in string, you can define the dictionary mapping in your code. See below for an example.
import configparser
config= configparser.ConfigParser()
config.read('configfile')
log_level_info = {'logging.DEBUG': logging.DEBUG,
'logging.INFO': logging.INFO,
'logging.WARNING': logging.WARNING,
'logging.ERROR': logging.ERROR,
}
print(config['DEFAULT']['LOG_LEVEL'])
my_log_level_from_config = config['DEFAULT']['LOG_LEVEL']
my_log_level = log_level_info.get(my_log_level_from_config, logging.ERROR)
logger.setLevel(my_log_level)
Your config file would be like below:
user#Inspiron:~/code/advanced_python$ cat configfile
[DEFAULT]
LOG_LEVEL = logging.INFO
user#Inspiron:~/code/advanced_python$
If I understood correctly, you need a way to set your logging level at runtime instead of a hard-coded value. I would say you have two options.
The first solution would be to parse your configuration file, and set the level of logging accordingly. If you don't want to parse it everytime the Log class is invoked, in your main you can set a variable that you pass to the Log class.
The second one, that I also suggest, would be to set handlers with python logging class https://docs.python.org/3/library/logging.config.html
logging level (logging.INFO) is an integer value. can you pass numbers from your config file to set log level
print(logging.INFO)
print(logging.WARN)
print(logging.DEBUG)
print(logging.ERROR)
20
30
10
40
I have a main script and multiple modules. Right now I have logging setup where all the logging from all modules go into the same log file. It gets hard to debug when its all in one file. So I would like to separate each module into its own log file. I would also like to see the requests module each module uses into the log of the module that used it. I dont know if this is even possible. I searched everywhere and tried everything I could think of to do it but it always comes back to logging everything into one file or setup logging in each module and from my main module initiate the script instead of import as a module.
main.py
import logging, logging.handlers
import other_script.py
console_debug = True
log = logging.getLogger()
def setup_logging():
filelog = logging.handlers.TimedRotatingFileHandler(path+'logs/api/api.log',
when='midnight', interval=1, backupCount=3)
filelog.setLevel(logging.DEBUG)
fileformatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
filelog.setFormatter(fileformatter)
log.addHandler(filelog)
if console_debug:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(name)-15s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
log.addHandler(console)
if __name__ == '__main__':
setup_logging()
other_script.py
import requests
import logging
log = logging.getLogger(__name__)
One very basic concept of python logging is that every file, stream or other place that logs go is equivalent to one Handler. So if you want every module to log to a different file you will have to give every module it's own handler. This can also be done from a central place. In your main.py you could add this to make the other_script module log to a separate file:
other_logger = logging.getLogger('other_script')
other_logger.addHandler(logging.FileHandler('other_file'))
other_logger.propagate = False
The last line is only required if you add a handler to the root logger. If you keep propagate at the default True you will have all logs be sent to the root loggers handlers too. In your scenario it might be better to not even use the root logger at all, and use a specific named logger like getLogger('__main__') in main.
I am trying to configure two loggers, one logger is for INFO level, and the other logger is for DEBUG level. I would like DEBUG content to only go to my log file, and I would like the INFO content to go both a log file and to the console. Please see the below code. Nothing is being written into my files and nothing is being displayed in the console.
logFileDir = os.path.join(os.getcwd(), '.logs')
if not os.path.exists(logFileDir):
os.mkdir(logFileDir)
infoLogFileDir = os.path.join(logFileDir, 'INFO')
if not os.path.exists(infoLogFileDir):
os.mkdir(infoLogFileDir)
debugLogFileDir = os.path.join(logFileDir, 'DEBUG')
if not os.path.exists(debugLogFileDir):
os.mkdir(debugLogFileDir)
LOG_FORMAT = ("%(asctime)s [%(levelname)s]: %(message)s in %(pathname)s:%(lineno)d")
#DEBUG LOGGER
debugLogFileName = os.path.join(debugLogFileDir, 'EFDebugLog.log')
debugLogger = logging.getLogger("debugLogger")
debugLogger.setLevel(logging.DEBUG)
debugHandler = logging.handlers.RotatingFileHandler(filename=debugLogFileName,maxBytes=5000000, backupCount=100)
debugHandler.setLevel(logging.DEBUG)
debugHandler.setFormatter(Formatter(LOG_FORMAT))
debugLogger.addHandler(debugHandler)
#INFO LOGGER
infoLogFileName = os.path.join(infoLogFileDir, 'EFInfoLog.log')
infoLogger = logging.getLogger("infoLogger")
infoLogger.setLevel(logging.INFO)
infoHandler = logging.handlers.RotatingFileHandler(filename=infoLogFileName,maxBytes=5000000, backupCount=100)
infoHandler.setLevel(logging.INFO)
infoHandler.setFormatter(Formatter(LOG_FORMAT))
infoLogger.addHandler(infoHandler)
infoLogger.addHandler(logging.StreamHandler())
The logging.* functions that you are calling log to the root logger. That's why you don't see any output; you haven't configured any handlers for the root logger. You have only configured handlers for your own loggers, which you are not using.
If you want to use the logging.* functions, you need to first configure the root logger, which you can get by calling getLogger without any arguments. So the code might look like:
import logging
import logging.handlers
root_logger = logging.getLogger()
info_handler = logging.handlers.RotatingFileHandler(filename='infolog.txt')
info_handler.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
debug_handler = logging.handlers.RotatingFileHandler(filename='debuglog.txt')
debug_handler.setLevel(logging.DEBUG)
root_logger.addHandler(stream_handler)
root_logger.addHandler(debug_handler)
root_logger.addHandler(info_handler)
# this is needed, since the severity is WARNING by default,
# i.e. it would not log any debug messages
root_logger.setLevel(logging.DEBUG)
root_logger.debug('this is a debug message')
root_logger.info('this is an info message')
I'm trying to create my own logger in python 3.6.8 to send output both to stdout and a log file (chosen by date, if the log file doesn't exist yet for today's date it gets created, if there already is a file with the same date just append).
from datetime import date
import logging
import logging.handlers
class Log:
def __init__(self):
pass
def getCleanerLogger(self,moduleName, logFolder, format):
filename = logFolder+ str(date.today()) + '-log.log'
handler = logging.FileHandler(filename)
shandler = logging.StreamHandler()
shandler.setLevel(logging.INFO)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(format)
handler.setFormatter(formatter)
shandler.setFormatter(formatter)
logger = logging.getLogger(moduleName)
logger.addHandler(handler)
logger.addHandler(shandler)
print("I've been called")
return logger
import Conf
conf = Conf.configuration()
print(conf['logFolder'] + " " + conf['logFormat'])
logger = Log()
logger = logger.getCleanerLogger("Log", conf['logFolder'], conf['logFormat'])
logger.info('initializing')
logger.debug('initializing debug')
in the json conf file these are the keys I load
"logFolder": "log/",
"logFormat": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
the log file gets created with the correct logic but there is no logging in either the console or the log file, only the prints go to stdout, no error or exception is raised, I really don't understand why this isn't working. I can only log with logging.root.level('msg') after loading a basiconfig.
Every handler has own logging level but logger has also global logging level which has bigger prioritet so you have to change this level to level which doesn't block handlers - ie.
logger.setLevel(logging.DEBUG)
Mininal working code with few smaller changes.
It doesn't use file with settings so everyone can easily copy it and run it.
from datetime import date
import os
import logging
import logging.handlers
class Log:
def get_cleaner_logger(self, module_name, log_folder, format):
if not os.path.exists(log_folder):
os.makedirs(log_folder)
filename = os.path.join(log_folder, date.today().strftime('%Y-%m-%d-log.log'))
print(filename)
logger = logging.getLogger(module_name)
print('before:', logger.level)
logger.setLevel(logging.DEBUG)
print('after:', logger.level)
formatter = logging.Formatter(format)
handler = logging.FileHandler(filename)
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)
logger.addHandler(handler)
shandler = logging.StreamHandler()
shandler.setLevel(logging.INFO)
shandler.setFormatter(formatter)
logger.addHandler(shandler)
print("I've been called")
return logger
conf = {
"logFolder": "log/",
"logFormat": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
}
logger = Log()
logger = logger.get_cleaner_logger("Log", conf['logFolder'], conf['logFormat'])
logger.info('initializing')
logger.debug('initializing debug')
BTW: I changed some names based on PEP 8 -- Style Guide for Python Code
referring to this question here: LINK
How can I set up a config, that will only log my root script and my own sub-scripts? The question of the link asked for disabling all imported modules, but that is not my intention.
My root setup:
import logging
from exchangehandler import send_mail
log_wp = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s - %(levelname)s [%(filename)s]: %(name)s %(funcName)20s - Message: %(message)s',
datefmt='%d.%m.%Y %H:%M:%S',
filename='C:/log/myapp.log',
filemode='a')
handler = logging.StreamHandler()
log_wp.addHandler(handler)
log_wp.debug('This is from root')
send_mail('address#eg.com', 'Request', 'Hi there')
My sub-module exchangehandler.py:
import logging
log_wp = logging.getLogger(__name__)
def send_mail(mail_to,mail_subject,mail_body, mail_attachment=None):
log_wp.debug('Hey this is from exchangehandler.py!')
m.send_and_save()
myapp.log:
16.07.2018 10:27:40 - DEBUG [test_script.py]: __main__ <module> - Message: This is from root
16.07.2018 10:28:02 - DEBUG [exchangehandler.py]: exchangehandler send_mail - Message: Hey this is from exchangehandler.py!
16.07.2018 10:28:02 - DEBUG [folders.py]: exchangelib.folders get_default_folder - Message: Testing default <class 'exchangelib.folders.SentItems'> folder with GetFolder
16.07.2018 10:28:02 - DEBUG [services.py]: exchangelib.services get_payload - Message: Getting folder ArchiveDeletedItems (archivedeleteditems)
16.07.2018 10:28:02 - DEBUG [services.py]: exchangelib.services get_payload - Message: Getting folder ArchiveInbox (archiveinbox)
My problem is, that the log-file contains also a lot of information of the exchangelib-module, that is imported in exchangehandler.py. Either the imported exchangelib-module is configured incorrectly or I have made a mistake. So how can I reduce the log-output only to my logging messages?
EDIT:
An extract of the folder.py of the exchangelib-module. This is not anything that I have written:
import logging
log = logging.getLogger(__name__)
def get_default_folder(self, folder_cls):
try:
# Get the default folder
log.debug('Testing default %s folder with GetFolder', folder_cls)
# Use cached instance if available
for f in self._folders_map.values():
if isinstance(f, folder_cls) and f.has_distinguished_name:
return f
return folder_cls.get_distinguished(account=self.account)
The imported exchangelib module is not configured at all when it comes to logging. You are configuring it implicitly by calling logging.basicConfig() in your main module.
exchangelib does create loggers and logs to them, but by default these loggers do not have handlers and formatters attached, so they don't do anything visible. What they do, is propagating up to the root logger, which by default also has no handlers and formatters attached.
By calling logging.basicConfig in your main module, you actually attach handlers to the root logger. Your own, desired loggers propagate to the root logger, hence the messages are written to the handlers, but the same is true for the exchangelib loggers from that point onwards.
You have at least two options here. You can explicitly configure "your" named logger(s):
main module
import logging
log_wp = logging.getLogger(__name__) # or pass an explicit name here, e.g. "mylogger"
hdlr = logging.StreamHandler()
fhdlr = logging.FileHandler("myapp.log")
log_wp.addHandler(hdlr)
log_wp.addHandler(fhdlr)
log_wp.setLevel(logging.DEBUG)
The above is very simplified. To explicitly configure multiple named loggers, refer to the logging.config HowTo
If you rather want to stick to just using the root logger (configured via basicConfig()), you can also explicitly disable the undesired loggers after exchangelib has been imported and these loggers have been created:
logging.getLogger("exchangelib.folders").disabled = True
logging.getLogger("exchangelib.services").disabled = True
If you don't know the names of the loggers to disable, logging has a dictionary holding all the known loggers. So you could temporarily do this to see all the loggers your program creates:
# e.g. after the line 'log_wp.addHandler(handler)'
print([k for k in logging.Logger.manager.loggerDict])
Using the dict would also allow you to do sth. like this:
for v in logging.Logger.manager.loggerDict.values():
if v.name.startswith('exchangelib'):
v.disabled = True