I have two python scripts in the same directory. I try to catch logging messages from both of them:
#script.py
import requests
import logging
logger = logging.getLogger(__name__)
class Downloader:
def __init__(self, url):
self.url = url
def download(self):
logger.debug(f'Downloading {self.url}')
req = requests.get(self.url, timeout=1)
return req
#main.py
import logging
from script import Downloader
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
logger.debug('create object')
d = Downloader('https://www.google.com')
res = d.download()
Basically I want to get rid of the debug-messages from the requests-module, so using logging.basicConfig() is not an option. But the way I do it, I do not get the debug-message from the imported script. Apparently because in script.py __name__ is not main.script.
How can I achieve this without hard coding anything to a string?
In a different module (e.g. logger.py):
import logging
def setup_logger(name, logfile, formatter, stream_handler=False, level=logging.DEBUG):
"""Function to create loggers."""
file_handler = logging.FileHandler(log_file)
stdout_handler = logging.StreamHandler()
file_handler.setFormatter(formatter)
stdout_handler.setFormatter(formatter)
logger = logging.getLogger(name)
if not logger.handlers:
logger.setLevel(level)
logger.addHandler(file_handler)
if stream_handler:
logger.addHandler(stdout_handler)
return logger
# Example formatter
formatter = logging.Formatter('%(asctime)s - %(levelname)s -> %(message)s\n')
# Generate the log object
log = setup_logger('logger_name', 'path_to_logfile', formatter)
Import this log object from your other modules to use it: from logger import log
Related
I am trying to add logging to my python application that has several modules and submodules. Several sites say to create child loggers in modules. The advantage I see is that the child logger inheriting the parent logging config, it will provide consistency for the logging output (handlers, formatters, ...).
So far I am defining the __main__ class logger name in each class and concatenate it with the current class’ name (parentName.childName) to get the module’s class loggers. It does not feel right, nor scalable. How could I improve this so I don’t have to hard code the __main__ class logger name in each class? Here is what my code looks like:
Py file that I run:
###app.py
import logging
from SubModule1 import *
class myApplication():
loggerName='myAPI'
def __init__(self, config_item_1=None,config_item_2=None,...config_item_N=None):
self.logger=self.logConfig()
self.logger.info("Starting my Application")
self.object1=mySubClass1(arg1, arg2, ... argM)
def logConfig(self):
fileLogFormat='%(asctime)s - %(levelname)s - %(message)s'
consoleLogFormat='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
# create logger
#logger = logging.getLogger(__name__)
logger = logging.getLogger(self.loggerName)
logger.setLevel(logging.DEBUG)
####CONSOLE HANDLER####
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
# create formatter
consoleFormatter = logging.Formatter(consoleLogFormat)
# add formatter to ch
ch.setFormatter(consoleFormatter)
# add ch to logger
logger.addHandler(ch)
####FILE HANDLER####
# create file handler and set level to debug
fh = logging.FileHandler('API_Log.txt')
fh.setLevel(logging.DEBUG)
# create formatter
fileFormatter = logging.Formatter(fileLogFormat)
# add formatter to ch
fh.setFormatter(fileFormatter)
# add ch to logger
logger.addHandler(fh)
logger.info('Logging is started!!')
return logger
def connect(self):
self.logger.info("Connecting to API")
self.object1.connect()
if __name__=="__main__":
config={"config_item_1": x,
"config_item_2": y,
...
"config_item_N": z
}
myApp=myApplication(config['config_item_1'],config['config_item_2'],...config['config_item_N'])
myApp.connect()
Module:
###SubModule1.py
import logging
import app
class mySubClass1():
appRootLoggerName='myAPI'
def __init__(self,item1):
self.logger=logging.getLogger(self.appRootLoggerName + '.'+mySubClass1.__name__)
self.logger.debug("mySubClass1 object created - mySubClass1")
self.classObject1=item1
The line below is the one that is bothering me. What alternative to self.appRootLoggerName + '.'+mySubClass1.__name__) would keep the same logging configuration shared across my application’s modules/classes?
logging.getLogger(self.appRootLoggerName + '.'+mySubClass1.__name__)
I am using the Logger model in Python.
I want the log to also print the file name from which the logger is called.
How could this be done?
This is my current implemation:
class Logger(object):
def __init__(self, name):
self.logger = logging.getLogger(name)
self.logger.setLevel(logging.INFO)
# create the logging file handler
fh = logging.FileHandler("{}.log".format(name))
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
# Add handle to logger object
self.logger.addHandler(fh)
def get_logger(self):
return self.logger
import logging
LOG = logging.getLogger(__name__)
if __name__ == "__main__":
logging.basicConfig()
LOG.error("Foo")
LOG.error(__file__)
Results in:
ERROR:__main__:Foo
ERROR:__main__:mylog.py
For something more complex, you can use the logger configuration to set a a log message formatter that uses the filename attribute of the LogRecords to include the filename in every log line without making it part of the log message body.
E.g.
logging.basicConfig(format="%(filename)s: %(message)s")
will result in log lines:
mylog.py: Foo
mylog.py: mylog.py
I am trying to use an FTP server stub during tests. I don't want the console output, but I would like to capture the logging to a file.
I want the FTP server to run in a different process, so I use multiprocessing.
My code as follows sets all logging to level WARNING:
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
import pyftpdlib.log as pyftpdliblog
import os
import logging
import multiprocessing as mp
authorizer = DummyAuthorizer()
authorizer.add_user('user', '12345', '.', perm='elradfmwM')
handler = FTPHandler
handler.authorizer = authorizer
pyftpdliblog.LEVEL = logging.WARNING
logging.basicConfig(filename='pyftpd.log', level=logging.INFO)
server = FTPServer(('', 2121), handler)
def main():
p = mp.Process(target=server.serve_forever)
p.start()
if __name__ == '__main__':
main()
How do I set only the console logging to level WARNING, or even better, completely shutdown without giving up the file logging?
So, after digging inside the code, I found out the following hint:
# This is a method of FTPServer and it is called before
# server.serve_forever
def _log_start(self):
if not logging.getLogger('pyftpdlib').handlers:
# If we get to this point it means the user hasn't
# configured logger. We want to log by default so
# we configure logging ourselves so that it will
# print to stderr.
from pyftpdlib.ioloop import _config_logging
_config_logging()
So, all I had to do is to define my own appropriate handlers:
logger = logging.getLogger('pyftpdlib')
logger.setLevel(logging.INFO)
hdlr = logging.FileHandler('pyftpd.log' )
logger.addHandler(hdlr)
Now, there is file logging, but console logging will not start.
Something like this:
import logging
date_format = "%Y/%m/%d %H:%M:%S"
log_file_path = "my_file.txt"
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
# own_module_logger = logging.getLogger(__name__)
pyftpdlib_logger = logging.getLogger("pyftpdlib")
# Setup logging to file (Only pyftpdlib)
filehandler = logging.FileHandler(filename = log_file_path)
filehandler.setLevel(logging.DEBUG)
fileformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
filehandler.setFormatter(fileformatter)
pyftpdlib_logger.addHandler(filehandler)
pyftpdlib_logger.propagate = False
# Setup logging to console (All other)
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
console.setFormatter(consoleformatter)
root_logger.addHandler(console)
# Do your loggings
a = logging.getLogger()
a.info('root I')
a.debug('root D')
b = logging.getLogger("pyftpdlib")
b.info('P I')
b.debug('P D')
logging.shutdown()
So loggings of pyftpdlib go to file. Everything from your module to console. One of the key thing here is the propagate!
I am trying out python's logger module to log info both in console and in a log and it works fine.I wanted that script to get invoked during machine star-tup and hence invoked my script from rc.local.
And my script gets invoked during boot-up, but the problem is, only console logging is working at that time. Logging to a file is not happening. Any clue on what could be the issue.
Is it anything related to syslog daemon?
import time
import logging
import datetime
global logger
global LOG_FILE
def initialize():
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
time = datetime.datetime.now()
LOG_FILE = "sample.log." + time.strftime("%Y%m%d")
log_file_handler = logging.FileHandler(LOG_FILE)
log_file_handler.setLevel(logging.DEBUG)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
log_file_handler = logging.StreamHandler()
log_file_handler.setLevel(logging.INFO)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
if __name__ == '__main__':
initialize()
logger.info("This should go both in console and log")
logger.debug("This should go only to Log")
import time
import logging
import datetime
def initialize():
# because logging is global module, so you can
# always get logger by logging.getLogger(__name__)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
time = datetime.datetime.now()
# if you only use LOG_FILE in function, you don't
# have to declare it outside.
LOG_FILE = "sample.log." + time.strftime("%Y%m%d")
log_file_handler = logging.FileHandler(LOG_FILE)
log_file_handler.setLevel(logging.DEBUG)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
log_file_handler = logging.StreamHandler()
log_file_handler.setLevel(logging.INFO)
log_file_formatter = logging.Formatter('%(message)s')
log_file_handler.setFormatter(log_file_formatter)
logger.addHandler(log_file_handler)
if __name__ == '__main__':
initialize()
logger = logging.getLogger(__name__)
logger.info("This should go both in console and log")
logger.debug("This should go only to Log")
I have a simple Python script that uses the in-built logging.
I'm configuring logging inside a function. Basic structure would be something like this:
#!/usr/bin/env python
import logging
import ...
def configure_logging():
logger = logging.getLogger("my logger")
logger.setLevel(logging.DEBUG)
# Format for our loglines
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Setup console logging
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
# Setup file logging as well
fh = logging.FileHandler(LOG_FILENAME)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
def count_parrots():
...
logger.debug??
if __name__ == '__main__':
logger = configure_logging()
logger.debug("I'm a log file")
parrots = count_parrots()
I can call logger fine from inside __main__. However, how do I call logger from inside the count_parrots() function? What's the most pythonic way of handling configuring a logger like this?
You can either use the root (default) logger, and thus the module level functions logging.debug, ... or get your logger in the function using it.
Indeed, the getLogger function is a factory-like function with a registry (singleton like), i.e. it always returns the same instance for the given logger name.
You can thus get your logger in count_parrots by simply using
logger = logging.getLogger("my logger")
at the beginning. However, the convention is to use a dotted hierarchical name for your logger. See http://docs.python.org/library/logging.html#logging.getLogger
EDIT:
You can use a decorator to add the logging behaviour to your individual functions, for example:
def debug(loggername):
logger = logging.getLogger(loggername)
def log_(enter_message, exit_message=None):
def wrapper(f):
def wrapped(*args, **kargs):
logger.debug(enter_message)
r = f(*args, **kargs)
if exit_message:
logger.debug(exit_message)
return r
return wrapped
return wrapper
return log_
my_debug = debug('my.logger')
#my_debug('enter foo', 'exit foo')
def foo(a, b):
return a+b
you can "hardcode" the logger name and remove the top-level closure and my_debug.
You can just do :
logger = logging.getLogger("my logger")
in your count_parrots() method. When you pass the name that was used earlier (i.e. "my logger") the logging module would return the same instance that was created corresponding to that name.
Update: From the logging tutorial
(emphais mine)
getLogger() returns a reference to a
logger instance with the specified
name if it is provided, or root if
not. The names are period-separated
hierarchical structures. Multiple
calls to getLogger() with the same
name will return a reference to the
same logger object.
The typical way to handle logging is to have a per-module logger stored in a global variable. Any functions and methods within that module then just reference that same logger instance.
This is discussed briefly in the intro to the advance logging tutorial in the documentation:
http://docs.python.org/howto/logging.html#advanced-logging-tutorial
You can pass logger instances around as parameters, but doing so is typically rare.
I got confused by how global variables work in Python. Within a function you only need to declare global logger if you were doing something like logger = logging.getLogger("my logger") and hoping to modify the global logger.
So to modify your example, you can create a global logger object at the start of the file. If your module can be imported by another one, you should add the NullHandler so that if the importer of the library doesn't want logging enabled, they don't have any issues with your lib (ref).
#!/usr/bin/env python
import logging
import ...
logger = logging.getLogger("my logger").addHandler(logging.NullHandler())
def configure_logging():
logger.setLevel(logging.DEBUG)
# Format for our loglines
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Setup console logging
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
# Setup file logging as well
fh = logging.FileHandler(LOG_FILENAME)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
def count_parrots():
...
logger.debug('counting parrots')
...
return parrots
if __name__ == '__main__':
configure_logging()
logger.debug("I'm a log file")
parrots = count_parrots()
If you don't need the log messages on your console, you can use in a minimalist way.
Alternatively you can use tail -f myapp.log to see the messages on the console.
import logging
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', \
filename='myapp.log', \
level=logging.INFO)
def do_something():
logging.info('Doing something')
def main():
logging.info('Started')
do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
You can give logger as argument to count_parrots() Or, what I would do, create class parrots and use logger as one of its method.