Develop Reference

How to get all previous logs written during current run (python logging)

I'm using the python logging module. How can I get all of the previously outputted logs that have been written-out by logger since the application started?
Let's say I have some large application. When the application first starts, it sets-up logging with something like this
import loggging
logging.basicConfig(
filename = '/path/to/log/file.log,
filemode = 'a',
format = '%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
datefmt = '%H:%M:%S',
level = logging.DEBUG
)
logging.info("===============================================================================")
logging.info( "INFO: Starting Application Logging" )
A few hours or days later I want to be able to get the history of all of the log messages ever written by logging and store that to a variable.
How can I get the history of messages written by the python logging module since the application started?

If you're writing to a log file, then you can simply get the path to the log file and then read its contents
import logging
logger = logging.getLogger( __name__ )
# attempt to get debug log contents
if logger.root.hasHandlers():
logfile_path = logger.root.handlers[0].baseFilename
with open(logfile_path) as log_file:
log_history = log_file.read()

Python logging multiple modules each module log to separate files

I have a main script and multiple modules. Right now I have logging setup where all the logging from all modules go into the same log file. It gets hard to debug when its all in one file. So I would like to separate each module into its own log file. I would also like to see the requests module each module uses into the log of the module that used it. I dont know if this is even possible. I searched everywhere and tried everything I could think of to do it but it always comes back to logging everything into one file or setup logging in each module and from my main module initiate the script instead of import as a module.
main.py
import logging, logging.handlers
import other_script.py
console_debug = True
log = logging.getLogger()
def setup_logging():
filelog = logging.handlers.TimedRotatingFileHandler(path+'logs/api/api.log',
when='midnight', interval=1, backupCount=3)
filelog.setLevel(logging.DEBUG)
fileformatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
filelog.setFormatter(fileformatter)
log.addHandler(filelog)
if console_debug:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(name)-15s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
log.addHandler(console)
if __name__ == '__main__':
setup_logging()
other_script.py
import requests
import logging
log = logging.getLogger(__name__)

One very basic concept of python logging is that every file, stream or other place that logs go is equivalent to one Handler. So if you want every module to log to a different file you will have to give every module it's own handler. This can also be done from a central place. In your main.py you could add this to make the other_script module log to a separate file:
other_logger = logging.getLogger('other_script')
other_logger.addHandler(logging.FileHandler('other_file'))
other_logger.propagate = False
The last line is only required if you add a handler to the root logger. If you keep propagate at the default True you will have all logs be sent to the root loggers handlers too. In your scenario it might be better to not even use the root logger at all, and use a specific named logger like getLogger('__main__') in main.

Python logging working on Windows but not Mac OS

# Logging
cur_flname = os.path.splitext(os.path.basename(__file__))[0]
LOG_FILENAME = constants.log_dir + os.sep + 'Log_' + cur_flname + '.txt'
util.make_dir_if_missing(constants.log_dir)
logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO, filemode='w',
format='%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s',
datefmt="%m-%d %H:%M") # Logging levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL
# Output to screen
logger = logging.getLogger(cur_flname)
logger.addHandler(logging.StreamHandler())
I use the code above to create a logger that I can use in my module to print messages to screen as well simultaneously to a text file.
On Windows, the messages get output to both file and screen. However, on Mac OS X 10.9.5, they only get output to file. I am using Python 2.7.
Any ideas on how to fix this?

From your question it is clear, you have no problem with creating logger name,
log file name and with logging to a file, so I will keep this part simplified to keep my code succinct.
First thing: To me your solution seems correct as logging.StreamHandler shall
send output to sys.stderr by default. You might have some code around (not
shown in your question), which is modifying sys.stderr, or you are running
your code on OSX in such a way, that output to stderr is not shown (but is
really output).
Solution with logging
Put following code into with_logging.py:
import logging
import sys
logformat = "%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s"
datefmt = "%m-%d %H:%M"
logging.basicConfig(filename="app.log", level=logging.INFO, filemode="w",
format=logformat, datefmt=datefmt)
stream_handler = logging.StreamHandler(sys.stderr)
stream_handler.setFormatter(logging.Formatter(fmt=logformat, datefmt=datefmt))
logger = logging.getLogger("app")
logger.addHandler(stream_handler)
logger.info("information")
logger.warning("warning")
def fun():
logger.info(" fun inf")
logger.warning("fun warn")
if __name__ == "__main__":
fun()
Run it: $ python with_logging.py and you shall see expected log records (properly formatted)
in log file app.log and on stderr too.
Note, if you do not see it on stderr, something is hiding your stderr
output. To see something, change in StreamHandler the stream to sys.stdout.
Solution with logbook
logbook is python package providing alternative logging means. I am showing it here to show main
difference to stdlib logging: with logbook the use and configuration seems simpler to me.
Previous solution rewritten to with_logbook.py
import logbook
import sys
logformat = ("{record.time:%m-%d %H:%M} {record.level_name} {record.module} - "
"{record.func_name}: {record.message}")
kwargs = {"level": logbook.INFO, "format_string": logformat, "bubble": True}
logbook.StreamHandler(sys.stderr, **kwargs).push_application()
logbook.FileHandler("app.log", **kwargs).push_application()
logger = logbook.Logger("app")
logger.info("information")
logger.warning("warning")
def fun():
logger.info(" fun inf")
logger.warning("fun warn")
if __name__ == "__main__":
fun()
The main difference is, that you do not have to attach handlers to loggers, your loggers simply emit
some log records.
These records will be handled by handlers, if they are put in place. One method
is to create a handler and call push_application() on it. This will put the
handler into stack of handlers in use.
As before, we need two handlers, one for the file, another for stderr.
If you run this script $ python with_logbook.py, you shall see exactly the same results as
with logging.
Separate logging setup from module code: short_logbook.py
With stdlib logging I reall do not like the introductory dances to make the logger work. I want
simply emit some log records and want to do that as simply as possible.
Following example is modification of previous one. Instead of setting up
logging on very beginning of the module, I do it just before the code is run
- inside the if __name__ == "__main__" (you may do the same on any other
place).
For practical reasons, I separated the module and calling code to two files:
File funmodule.py
from logbook import Logger
log = Logger(__name__)
log.info("information")
log.warning("warning")
def fun():
log.info(" fun inf")
log.warning("fun warn")
You can notice, that there are really only two lines of code related to making
logging available.
Then, create the calling code, put into short_logbook.py:
import sys
import logbook
if __name__ == "__main__":
logformat = ("{record.time:%m-%d %H:%M} {record.level_name} {record.module}"
"- {record.func_name}: {record.message}")
kwargs = {"level": logbook.INFO, "format_string": logformat, "bubble": True}
logbook.StreamHandler(sys.stderr, **kwargs).push_application()
logbook.FileHandler("app.log", **kwargs).push_application()
from funmodule import fun
fun()
Running the code you will see it working the same way as before, only logger name will be funmodule.
Note, that I did the from funmodule import fun after the logging was set up. If I did it on the
top if the short_logbook.py file, the first two log records from funmodule.py would not be seen
as they happen during module import.
EDIT: added another logging solution to have fair comparison
One more stdlib logging solution
Trying to have fair comparison of logbook and logging I rewrote final logbook example to
logging.
funmodule_logging.py looks like:
import logging
log = logging.getLogger(__name__)
log.info("information")
log.warning("warning")
def fun():
log.info(" fun inf")
log.warning("fun warn")
log.error("no fun at all")
and short_logging.py looks as follows:
import sys
import logging
if __name__ == "__main__":
logformat = "%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s"
datefmt = "%m-%d %H:%M"
logging.basicConfig(filename="app.log", level=logging.INFO, filemode="w",
format=logformat, datefmt=datefmt)
stream_handler = logging.StreamHandler(sys.stderr)
stream_handler.setFormatter(logging.Formatter(fmt=logformat, datefmt=datefmt))
logger = logging.getLogger("funmodule_logging")
logger.addHandler(stream_handler)
from funmodule_logging import fun
fun()
Functionality is very similar.
I still strugle with logging. stdlib logging is not easy to grasp, but it is in stdlib and offers
some nice things like logging.config.dictConfig allowing to configure logging by a dictionary.
logbook was much simpler to start with, but is a bit slower at the moment and lacks the
dictConfig. Anyway, these differences are not relevant to your question.

Logging basicConfig not creating log file when I run in PyCharm?

When I run below code in terminal its create a log file
import logging
logging.basicConfig(filename='ramexample.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
but when I run the same code (with different filename='ram.log') in PyCharm it's not creating any log file. Why?
import logging
logging.basicConfig(filename='ram.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
What I have to do to create a log file with PyCharm?

I encountered same issue and found none of the answers previously provided here would work. Maybe this issue had been solved long ago to Ramnath Reddy, but I could not find the correct answer anywhere online.
Luckily, I found a solution from a colleague's code by adding the following lines before logging.basicConfig().
# Remove all handlers associated with the root logger object.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
Try and see if it helps for whomever had the same issue.
Python 3.8: A new option, force, has been made available to automatically remove the root handlers while calling basicConfig().
For example:
logging.basicConfig(filename='ramexample.log', level=logging.DEBUG, force=True)`
See logging.basicConfig parameters:
force: If this keyword argument is specified as true, any existing handlers attached to the root logger are removed and closed, before carrying out the configuration as specified by the other arguments.

I can't remember where I got this otherwise I would have provided a link. But had the same problem some time ago using in jupyter notebooks and this fixed it:
import logging
logger = logging.getLogger()
fhandler = logging.FileHandler(filename='mylog.log', mode='a')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.setLevel(logging.DEBUG)

The answer why this error happens is this:
The call to basicConfig() should come before any calls to debug(), info() etc.
If you do so the basicConfig can not create and write a new file.
Here I called logging.info() right before logging.basicConfig().
Don't:
import logging
logging.info("root") # call to info too early
logging.basicConfig(filename="rec/test.log", level=logging.DEBUG) # no file created

Maximas is right. File path is relative to execution environment. However instead of writing down the absolute path you could try the dynamic path resolution approach:
filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ram.log')
logging.basicConfig(filename=filename, level=logging.DEBUG)
This assumes that ram.log resides in the same directory with the one that contains the above code (that's why __file__ is used for).

By using force it get solve.
like logging.basicConfig(filename="test.log", force=True)

This does create a log within the pycharm terminal using the Py terminal within it. You need to check the location of where the terminal is (try dir on Windows or pwd on linux/mac). Instead of just putting in ram.log, use the full file path of where you would like the file to appear.
E.G.
logging.basicConfig(filename='/Users/Donkey/Test/ram.log', level=logging.DEBUG)

I used to get this error, but I solved by adding force=True in basicConfig:
logging.basicConfig(level=logging.INFO,filename='C:\\Users\\sukal\\PycharmProjects\\Test\\Logs\\Automation.log',format=Log_Format,force=True)

import logging
class LogGen:
#staticmethod
def loggen():
logger = logging.getLogger()
fhandler = logging.FileHandler(filename='.\\logs\\automation.log', mode='a')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.setLevel(logging.INFO)
return logger

Logging over multiple modules

How can I log everything using Python 'logging' to 1 text file, over multiple modules?
Main.py:
import logging
logging.basicConfig(format='localhost - - [%(asctime)s] %(message)s', level=logging.DEBUG)
log_handler = logging.handlers.RotatingFileHandler('debug.out', maxBytes=2048576)
log = logging.getLogger('logger')
log.addHandler(log_handler)
import test
Test.py:
import logging
log = logging.getLogger('logger')
log.error('test')
debug.out stays empty. I'm not sure what to try next, even after reading the logging documentation.
Edit: Fixed with the code above.

Set the correct logging level (at least ERROR if you want to get all messages with level ERROR or higher) and add a handler to write all messages into a file. For more details have a look at https://docs.python.org/2/howto/logging-cookbook.html.