I made a little game as my school project & i want to save critical errors to my .log file that I had created using logging module, but i just can't figure out how to log them.
Logging configuration:
logging.basicConfig(level=logging.DEBUG, filename='log.log', filemode='w', format='%(asctime)s - %(name)s:%(levelname)s: %(message)s')
What i want to do:
Every time a program will encounter exception, that can't be handled (like TypeError or SyntaxError) it will be saved in log.log file. Then program will exit. But it usually just close program and pritn error in terminal, which is not intended.
I have tried to use:
import sys
class ShutdownHandler(logging.Handler):
def emit(self, record):
logging.critical(record.msg)
logging.shutdown()
sys.exit(1)
logging.basicConfig(level=logging.DEBUG, filename='log.log', filemode='w', format='%(asctime)s - %(name)s:%(levelname)s: %(message)s')
But it didn't saved output to log.log file. Instead i have got error message in terminal.
As discussed above you can use the following to record any errors which will be returned to you in the terminal :
import subprocess
a = subprocess.run(["python3","hello.py"], stdout=subprocess.PIPE)
print(a.stdout.decode('utf-8'))
Once you have the output, you can search for errors ad log your experiments accordingly .
Related
I'm using the python logging module. How can I get all of the previously outputted logs that have been written-out by logger since the application started?
Let's say I have some large application. When the application first starts, it sets-up logging with something like this
import loggging
logging.basicConfig(
filename = '/path/to/log/file.log,
filemode = 'a',
format = '%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
datefmt = '%H:%M:%S',
level = logging.DEBUG
)
logging.info("===============================================================================")
logging.info( "INFO: Starting Application Logging" )
A few hours or days later I want to be able to get the history of all of the log messages ever written by logging and store that to a variable.
How can I get the history of messages written by the python logging module since the application started?
If you're writing to a log file, then you can simply get the path to the log file and then read its contents
import logging
logger = logging.getLogger( __name__ )
# attempt to get debug log contents
if logger.root.hasHandlers():
logfile_path = logger.root.handlers[0].baseFilename
with open(logfile_path) as log_file:
log_history = log_file.read()
I'd like to log Python warnings and errors to a file. More specifically, I'd like to have loggers printing logs to the console as it is done by default. On top of this, I'd like to log warnings and errors to a file so that I can do post-mortem analysis if the Python app crashes.
I wrote the following code that, inside the function enable_local_log, creates a logger to log the errors and warnings to a file.
Do you think this will effectively work? Will it log all the Python errors to the log file?
I am asking this because my Python process is being killed after 1-2 days of execution. I found the following log using dmesg -T command. It was logged at about the same time my Python app stopped working:
[Tue Jul 5 00:17:36 2022] audit: type=1701 audit(1656980258.453:147): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=2203731 comm="python3" exe="/usr/bin/python3.6" sig=6 res=1
However, no error is being logged to my Python log file (the one create the Python code below). Unfortunately, I am deploying the software using a tool that does not show me the old log message. That's why I am trying to log the errors to a file.
Here's my code:
import glob
import logging
import traceback
from logging.handlers import RotatingFileHandler
import os
import sys
LOGS_FOLDER = "/logs/"
LOG_FILEPATH = os.path.join(LOGS_FOLDER, "log.txt")
LOG_FORMAT = f"%(threadName)s %(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(
level=logging.INFO,
format=LOG_FORMAT
)
def enable_local_log():
if LOG_FILEPATH is not None:
os.makedirs(
os.path.dirname(LOG_FILEPATH),
exist_ok=True
)
# Create formatter to log on local file
log_formatter = logging.Formatter(LOG_FORMAT)
my_handler = RotatingFileHandler(
LOG_FILEPATH,
mode="a",
maxBytes=5 * (1024 ** 2), # 5 Megabytes * (1024 ** 2)
backupCount=2,
encoding=None,
delay=0
)
my_handler.setFormatter(log_formatter)
my_handler.setLevel(logging.WARNING)
root_logger = logging.getLogger()
root_logger.addHandler(my_handler)
The first two lines of code of my Python project import the enable_local_log() function and calls it:
from .myfile import enable_local_log
enable_local_log()
# Logging
cur_flname = os.path.splitext(os.path.basename(__file__))[0]
LOG_FILENAME = constants.log_dir + os.sep + 'Log_' + cur_flname + '.txt'
util.make_dir_if_missing(constants.log_dir)
logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO, filemode='w',
format='%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s',
datefmt="%m-%d %H:%M") # Logging levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL
# Output to screen
logger = logging.getLogger(cur_flname)
logger.addHandler(logging.StreamHandler())
I use the code above to create a logger that I can use in my module to print messages to screen as well simultaneously to a text file.
On Windows, the messages get output to both file and screen. However, on Mac OS X 10.9.5, they only get output to file. I am using Python 2.7.
Any ideas on how to fix this?
From your question it is clear, you have no problem with creating logger name,
log file name and with logging to a file, so I will keep this part simplified to keep my code succinct.
First thing: To me your solution seems correct as logging.StreamHandler shall
send output to sys.stderr by default. You might have some code around (not
shown in your question), which is modifying sys.stderr, or you are running
your code on OSX in such a way, that output to stderr is not shown (but is
really output).
Solution with logging
Put following code into with_logging.py:
import logging
import sys
logformat = "%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s"
datefmt = "%m-%d %H:%M"
logging.basicConfig(filename="app.log", level=logging.INFO, filemode="w",
format=logformat, datefmt=datefmt)
stream_handler = logging.StreamHandler(sys.stderr)
stream_handler.setFormatter(logging.Formatter(fmt=logformat, datefmt=datefmt))
logger = logging.getLogger("app")
logger.addHandler(stream_handler)
logger.info("information")
logger.warning("warning")
def fun():
logger.info(" fun inf")
logger.warning("fun warn")
if __name__ == "__main__":
fun()
Run it: $ python with_logging.py and you shall see expected log records (properly formatted)
in log file app.log and on stderr too.
Note, if you do not see it on stderr, something is hiding your stderr
output. To see something, change in StreamHandler the stream to sys.stdout.
Solution with logbook
logbook is python package providing alternative logging means. I am showing it here to show main
difference to stdlib logging: with logbook the use and configuration seems simpler to me.
Previous solution rewritten to with_logbook.py
import logbook
import sys
logformat = ("{record.time:%m-%d %H:%M} {record.level_name} {record.module} - "
"{record.func_name}: {record.message}")
kwargs = {"level": logbook.INFO, "format_string": logformat, "bubble": True}
logbook.StreamHandler(sys.stderr, **kwargs).push_application()
logbook.FileHandler("app.log", **kwargs).push_application()
logger = logbook.Logger("app")
logger.info("information")
logger.warning("warning")
def fun():
logger.info(" fun inf")
logger.warning("fun warn")
if __name__ == "__main__":
fun()
The main difference is, that you do not have to attach handlers to loggers, your loggers simply emit
some log records.
These records will be handled by handlers, if they are put in place. One method
is to create a handler and call push_application() on it. This will put the
handler into stack of handlers in use.
As before, we need two handlers, one for the file, another for stderr.
If you run this script $ python with_logbook.py, you shall see exactly the same results as
with logging.
Separate logging setup from module code: short_logbook.py
With stdlib logging I reall do not like the introductory dances to make the logger work. I want
simply emit some log records and want to do that as simply as possible.
Following example is modification of previous one. Instead of setting up
logging on very beginning of the module, I do it just before the code is run
- inside the if __name__ == "__main__" (you may do the same on any other
place).
For practical reasons, I separated the module and calling code to two files:
File funmodule.py
from logbook import Logger
log = Logger(__name__)
log.info("information")
log.warning("warning")
def fun():
log.info(" fun inf")
log.warning("fun warn")
You can notice, that there are really only two lines of code related to making
logging available.
Then, create the calling code, put into short_logbook.py:
import sys
import logbook
if __name__ == "__main__":
logformat = ("{record.time:%m-%d %H:%M} {record.level_name} {record.module}"
"- {record.func_name}: {record.message}")
kwargs = {"level": logbook.INFO, "format_string": logformat, "bubble": True}
logbook.StreamHandler(sys.stderr, **kwargs).push_application()
logbook.FileHandler("app.log", **kwargs).push_application()
from funmodule import fun
fun()
Running the code you will see it working the same way as before, only logger name will be funmodule.
Note, that I did the from funmodule import fun after the logging was set up. If I did it on the
top if the short_logbook.py file, the first two log records from funmodule.py would not be seen
as they happen during module import.
EDIT: added another logging solution to have fair comparison
One more stdlib logging solution
Trying to have fair comparison of logbook and logging I rewrote final logbook example to
logging.
funmodule_logging.py looks like:
import logging
log = logging.getLogger(__name__)
log.info("information")
log.warning("warning")
def fun():
log.info(" fun inf")
log.warning("fun warn")
log.error("no fun at all")
and short_logging.py looks as follows:
import sys
import logging
if __name__ == "__main__":
logformat = "%(asctime)s %(levelname)s %(module)s - %(funcName)s: %(message)s"
datefmt = "%m-%d %H:%M"
logging.basicConfig(filename="app.log", level=logging.INFO, filemode="w",
format=logformat, datefmt=datefmt)
stream_handler = logging.StreamHandler(sys.stderr)
stream_handler.setFormatter(logging.Formatter(fmt=logformat, datefmt=datefmt))
logger = logging.getLogger("funmodule_logging")
logger.addHandler(stream_handler)
from funmodule_logging import fun
fun()
Functionality is very similar.
I still strugle with logging. stdlib logging is not easy to grasp, but it is in stdlib and offers
some nice things like logging.config.dictConfig allowing to configure logging by a dictionary.
logbook was much simpler to start with, but is a bit slower at the moment and lacks the
dictConfig. Anyway, these differences are not relevant to your question.
How can I log everything using Python 'logging' to 1 text file, over multiple modules?
Main.py:
import logging
logging.basicConfig(format='localhost - - [%(asctime)s] %(message)s', level=logging.DEBUG)
log_handler = logging.handlers.RotatingFileHandler('debug.out', maxBytes=2048576)
log = logging.getLogger('logger')
log.addHandler(log_handler)
import test
Test.py:
import logging
log = logging.getLogger('logger')
log.error('test')
debug.out stays empty. I'm not sure what to try next, even after reading the logging documentation.
Edit: Fixed with the code above.
Set the correct logging level (at least ERROR if you want to get all messages with level ERROR or higher) and add a handler to write all messages into a file. For more details have a look at https://docs.python.org/2/howto/logging-cookbook.html.
I've read through the logging module documentation and whilst I may have missed something obvious, the code I've got doesn't appear to be working as intended. I'm using Python 2.6.4.
My program consists of several different python files, from which I want to send logging messages to a text file and, potentially, the screen. I imagine this is a common thing to do so I'm messing this up somewhere.
What my code is doing at the minute is logging to the text file correctly, kinda. But logging to the screen is being duplicated, one with the specified formatting, and one without. Also, when I turn off the screen output, I'm still getting the text printed once, which I don't want - I just want it to be logged to the file.
Anyway, some code:
#logger.py
import logging
from logging.handlers import RotatingFileHandler
import os
def setup_logging(logdir=None, scrnlog=True, txtlog=True, loglevel=logging.DEBUG):
logdir = os.path.abspath(logdir)
if not os.path.exists(logdir):
os.mkdir(logdir)
log = logging.getLogger('stumbler')
log.setLevel(loglevel)
log_formatter = logging.Formatter("%(asctime)s - %(levelname)s :: %(message)s")
if txtlog:
txt_handler = RotatingFileHandler(os.path.join(logdir, "Stumbler.log"), backupCount=5)
txt_handler.doRollover()
txt_handler.setFormatter(log_formatter)
log.addHandler(txt_handler)
log.info("Logger initialised.")
if scrnlog:
console_handler = logging.StreamHandler()
console_handler.setFormatter(log_formatter)
log.addHandler(console_handler)
Nothing unusual there.
#core.py
import logging
corelog = logging.getLogger('stumbler.core') # From what I understand of the docs, this should work :/
class Stumbler:
[...]
corelog.debug("Messages and rainbows...")
The screen output shows how this is being duplicated:
2010-01-08 22:57:07,587 - DEBUG :: SCANZIP: Checking zip contents, file: testscandir/testdir1/music.mp3
DEBUG:stumbler.core:SCANZIP: Checking zip contents, file: testscandir/testdir1/music.mp3
2010-01-08 22:57:07,587 - DEBUG :: SCANZIP: Checking zip contents, file: testscandir/testdir2/subdir/executable.exe
DEBUG:stumbler.core:SCANZIP: Checking zip contents, file: testscandir/testdir2/subdir/executable.exe
Although the textfile is getting the correctly formatted output, turning the screen logging off in logger.py still has the incorrectly formatted output displayed.
From what I understand of the docs, calling corelog.debug(), seeing as corelog is a child of the "stumbler" logger, it should use that formatting and output the logs as such.
Apologies for the essay over such a trivial issue.
TL;DR: How do I do logging from multiple files?
Are you sure no other logging setup is being done in anything you import.
The incorrect output in your console logs look like the default configuration for a logger, so something else may be setting that up.
Running this quick test script:
import logging
from logging.handlers import RotatingFileHandler
import os
def setup_logging(logdir=None, scrnlog=True, txtlog=True, loglevel=logging.DEBUG):
logdir = os.path.abspath(logdir)
if not os.path.exists(logdir):
os.mkdir(logdir)
log = logging.getLogger('stumbler')
log.setLevel(loglevel)
log_formatter = logging.Formatter("%(asctime)s - %(levelname)s :: %(message)s")
if txtlog:
txt_handler = RotatingFileHandler(os.path.join(logdir, "Stumbler.log"), backupCount=5)
txt_handler.doRollover()
txt_handler.setFormatter(log_formatter)
log.addHandler(txt_handler)
log.info("Logger initialised.")
if scrnlog:
console_handler = logging.StreamHandler()
console_handler.setFormatter(log_formatter)
log.addHandler(console_handler)
setup_logging('/tmp/logs')
corelog = logging.getLogger('stumbler.core')
corelog.debug("Messages and rainbows...")
yields this result:
2010-01-08 15:39:25,335 - DEBUG ::
Messages and rainbows...
and in my /tmp/logs/Stumbler.log
2010-01-08 15:39:25,335 - INFO ::
Logger initialised. 2010-01-08
15:39:25,335 - DEBUG :: Messages and
rainbows...
This worked as expected when I ran it in python 2.4, 2.5, and 2.6.4