I have a python script which every 10 minutes sends me an email with everything written in the console. I am running this with the crontab in my ubuntu 18.04 vps.
Sometimes it doesn't send the mail so I assume that when an error happens execution stops but how can I get the errors to be written in a txt file so I can analyze the error ?
Logging Module
To demonstrate the approach with the logging module, this would be the general approach
import logging
# Create a logging instance
logger = logging.getLogger('my_application')
logger.setLevel(logging.INFO) # you can set this to be DEBUG, INFO, ERROR
# Assign a file-handler to that instance
fh = logging.FileHandler("file_dir.txt")
fh.setLevel(logging.INFO) # again, you can set this differently
# Format your logs (optional)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter) # This will set the format to the file handler
# Add the handler to your logging instance
logger.addHandler(fh)
try:
raise ValueError("Some error occurred")
except ValueError as e:
logger.exception(e) # Will send the errors to the file
And if I cat file_dir.txt
2019-03-14 14:52:50,676 - my_application - ERROR - Some error occurred
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: Some error occurred
Print to File
As I pointed out in the comments, you could accomplish this with print as well (I'm not sure you will be applauded for it)
# Set your stdout pointer to a file handler
with open('my_errors.txt', 'a') as fh:
try:
raise ValueError("Some error occurred")
except ValueError as e:
print(e, file=fh)
cat my_errors.txt
Some error occurred
Note that logging.exception includes the traceback in this case, which is one of the many huge benefits of that module
Edit
In the interest of completeness, the traceback module leverages a similar approach as print, where you can supply a file handle:
import traceback
import sys
with open('error.txt', 'a') as fh:
try:
raise ValueError("Some error occurred")
except ValueError as e:
e_type, e_val, e_tb = sys.exc_info()
traceback.print_exception(e_type, e_val, e_tb, file=fh)
This will include all of the information you want from logging
You can use the logging module as suggested in the comments (possibly superior but outside the scope of my knowledge), or catch the errors with try and except like:
try:
pass
#run the code you currently have
except Exception as e: # catch ALLLLLL errors!!!
print(e) # or more likely you'd want something like "email_to_me(e)"
Although this is generally frowned upon to catch all exceptions, because then should your program fail for whatever reason it will get gobbled up in the except clause so a better approach is to figure out what specific error you are encountering like IndexError and then just catch this specific error like:
try:
pass
#run the code you currently have
except IndexError as e: # catch only indexing errors!!!
print(e) # or more likely you'd want something like "email_to_me(e)"
To be able to debug and not only know the kind of error that happened, you can also get the error stack using traceback module (usually in the starting package of modules):
import traceback
try:
my_function()
except Exception as e:
print(e)
traceback.print_exc()
And then run your code 'my_code.py' in console usig >>
python my_code.py >> my_prints.txt
All the prints of your code will then be written in this .txt file, including the printed error and its stack. This is very interesting in your case or while running code on a docker if you want to detach yourself from it with ctrl+p+q and still know what is printed.
Related
I'm setting up a logger in my script like shown at the bottom. This works fine for my purposes and logs my __main__ log messages and those of any modules I use to stdout and a log file.
During program execution a module call that I'm using xarray.open_dataset(file, engine="cfgrib") raises an Error in some conditions and produces the following log output:
2023-02-18 10:02:06,731 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
...
How can I access this output during program execution?
The raised error in the cfgrib module is handled there gracefully and program execution can continue, but the logic of my program requires that I access the error message, in particular the part saying shortName='i10fg' in order to handle the error exhaustively.
Here is how my logger is set up:
def init_log():
"""initialize logging
returns logger using log settings from the config file (settings.toml)
"""
# all settings from a settings file with reasonable defaults
lg.basicConfig(
level=settings.logging.log_level,
format=settings.logging.format,
filemode=settings.logging.logfile_mode,
filename=settings.logging.filename,
)
mylogger = lg.getLogger(__name__)
stream = lg.StreamHandler()
mylogger.addHandler(stream)
clg.install(
level=settings.logging.log_level,
logger=mylogger,
fmt="%(asctime)s %(levelname)s:\t%(message)s",
)
return mylogger
# main
log = init_log()
log.info('...reading files...')
I went through the python logging documentation and cookbook. While this contains ample examples on how to modify logging for various purposes, I could not find an example for accessing and reacting to a log message during program execution.
The Exception in my logs look this:
2023-02-20 12:22:37,209 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 660, in build_dataset_components
dict_merge(variables, coord_vars)
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 591, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='time' value=Variable(dimensions=('time',), data=array([1640995200, 1640998800, 1641002400, ..., 1672520400, 1672524000,
1672527600])) new_value=Variable(dimensions=('time',), data=array([1640973600, 1641016800, 1641060000, 1641103200, 1641146400,
1641189600, 1641232800, 1641276000, 1641319200, 1641362400,
I cannot catch the Exception directly for some reason:
...
import sys
from cfgrib.dataset import DatasetBuildError
...
try:
df = xr.open_dataset(file, engine="cfgrib").to_dataframe()
# triggering error manually like with the two lines below works as expected
# raise Exception()
# raise DatasetBuildError()
except Exception as e:
print('got an Exception')
print(e)
print(e.args)
except BaseException as e:
print('got a BaseException')
print(e.args)
except DatasetBuildError as e:
print(e)
except:
print('got any and all exception')
type, value, traceback = sys.exc_info()
print(type)
print(value)
print(traceback)
Unless I uncomment the two lines where I raise the exception manually, the except clauses are never triggered, event though I can see the DatabaseBuildError in my logs.
Not sure if this has any bearing, but while I can see the Exception as quoted above in my file log, it is not printed to stdout.
When I want to log some specific Exception, but otherwise ignore it, I can do that like so:
import logging
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError:
logger.error("It failed with:", exc_info=True)
(This is in fact an MRE, as something_that_may_fail hasn't been defined, so the try block will raise NameError with message name 'something_that_may_fail' is not defined. š)
This however will also log the stack trace:
It failed with:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
NameError: name 'something_that_may_fail' is not defined
Sometimes that isn't what I want: In some cases, I already know that exception type and exception message (together with my custom log message) will suffice, and don't want to expand the log with stack traces that don't tell me anything new. So I'd want a log entry that simply is
It failed with:
NameError: name 'something_that_may_fail' is not defined
I can achieve that by passing a 3-tuple as the exc_info, with the stack trace component replaced by None:
import logging
import sys
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError:
exc_type, exc_value, _trace = sys.exc_info()
logger.error("It failed with:", exc_info=(exc_type, exc_value, None))
But I'm not sure how reliable that is. (The documentation doesn't mention how the tuple may or may not deviate from one returned by sys.exc_info().)
Examining the exception myself with
...
except NameError as e:
...
comes with its own problems:
f"{type(e)}" gives string <class 'NameError'> instead of just string NameError
The proper solution to get the fully qualified type name, including packages/modules but without builtin. is rather unwieldy and not something I'd want in exception handling code. See the currently accepted answer to Get fully qualified class name of an object in Python.
Can I rely on the message always being e.args[0]? (I might have uses for other exceptions (with more sub-types) than just NameError, which I've used here only as an example.)
So what is the proper way to log exception type and message without the stack trace? Is there a cleaner way than my make-the-trace-None hack above?
traceback.format_exception_only can be used for that:
import logging
import sys
import traceback
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError:
exc_type, exc_value, _trace = sys.exc_info()
exc_desc_lines = traceback.format_exception_only(exc_type, exc_value)
exc_desc = ''.join(exc_desc_lines).rstrip()
logger.error(f"It failed with:\n{exc_desc}")
or without sys:
import logging
import traceback
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError as e:
exc_desc_lines = traceback.format_exception_only(type(e), e)
exc_desc = ''.join(exc_desc_lines).rstrip()
logger.error(f"It failed with:\n{exc_desc}")
(Found this by looking how the logging module actually extracts and formats information from exc_info. There traceback.print_exception is being used, so I looked what else is available in the traceback module.)
I accidentally called logging.info() in a wrong way using Python3, and the library itself threw an error to the console, but the FileHandler failed to catch such error. So is there a way to catch all the errors no matter where they are thrown.
The error message looks like:
--- Logging error ---
...
TypeError: not all arguments converted during string formatting
Call stack:
File "<ipython-input-12-5ba547bc4aeb>", line 1, in <module>
logging.info(1,1)
Message: 1
Arguments: (1,)
Copying the following code can reproduce my question. The log file can catch logging.info() and the error of ZeroDivisionError, but it failed to catch error messages thrown by the logging library.
import logging
logger_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s: %(message)s')
logger_handler = logging.FileHandler('/Users/your_name/Desktop/logging.log')
logger_handler.setLevel(logging.DEBUG)
logger_handler.setFormatter(logger_formatter)
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(logger_handler)
try:
logging.info('test')
logging.info(1,1)
1/0
except:
logging.exception("")
Log file output:
2021-03-10 18:07:32,315 - root - INFO: test
2021-03-10 18:07:32,324 - root - ERROR:
Traceback (most recent call last):
File "<ipython-input-1-6a4f609a80ca>", line 17, in <module>
1/0
ZeroDivisionError: division by zero
Logging all errors that can happen during logging is impossible, because the error may be such that it breaks logging and if that would trigger a logging call it would lead to an infinite loop. You can however implement custom error handling by overriding handleError for your Handlers, and if you are feeling particularly brave, attempt to write a log within that error handler. Based on your code it would look something like this:
import logging
class MyFileHandler(logging.FileHandler):
def handleError(self, record):
logging.error('There was an error when writing a log. Msg was: '+str(record.msg))
logger_handler = MyFileHandler('custom.log')
logger_handler.setLevel(logging.DEBUG)
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(logger_handler)
try:
logging.info('test')
logging.info(1,1)
1/0
except:
logging.exception("")
Of course if you would rather have an exception that bubbles up instead you could just raise from handleError.
I'm a newbie in Flask and I am trying to display the Built-In Exceptions in python but I can't seem to have them display on my end.
NOTE:
set FLASK_DEBUG = 0
CODE:
def do_something:
try:
doing_something()
except Exception as err:
return f"{err}"
Expectation:
It will display one of the built-in exceptions:
KeyError
IndexError
NameError
Etc.
Reality:
It will return the line of code that didn't worked which is more ambiguous to the end user.
Also:
I have no problem seeing the errors when the debug mode is ON but that's not something that I want to do if I open them in public
Flask supplies you with a function that enables you to register an error handler throughout your entire app; you can do something as shown below:
def handle_exceptions(e):
# Log exception in your logs
# get traceback and sys exception info and log as required
# app.logger.error(getattr(e, 'description', str(e)))
# Print traceback
# return your response using getattr(e, 'code', 500) etc.
# Exception is used to catch all exceptions
app.register_error_handler(Exception, handle_exceptions)
In my honest opinion, this is the way to go. - Following the structure found in werkzeug.exceptions.HTTPException as an example is a solid foundation.
Having a unified exception handler that will standardise your Exception handling, visualisation and logging will make your life a tad better. :)
Try with this:
def do_something:
try:
doing_something()
except Exception as err:
return f"{err.__class__.__name__}: {err}"
I'm using Fabric to automate, including the task of creating a directory. Here is my fabfile.py:
#!/usr/bin/env python
from fabric.api import *
def init():
try:
local('mkdir ./www')
except ##what exception?##:
#print exception name to put in above
Run fab fabfile.py and f I already have ./www created an error is raised, but I don't know what kind, so I don't know how to handle the error yet. Fabric only prints out the following:
mkdir: cannot create directory ā./wwwā: File exists
Fatal error: local() encountered an error (return code 1) while executing 'mkdir ./www'
Aborting.
What I want to do is be able to find out the error type so that I can except my errors properly without blanket statements. It would be really helpful if an answer does not just tell me how to handle a mkdir exception, but print (or otherwise find the name to) any exception I may run into down the line (mkdir is just an example).
Thank you!
The issue is that fabric uses subprocess for doing these sorts of things. If you look at the source code for local you can see it doesn't actually raise an exception. It calls suprocess.Popen and uses communicate() to read stdout and stderr. If there is a non-zero return code then it returns a call to either warn or abort. The default is abort. So, to do what you want, try this:
def init():
with settings(warn_only=True):
local('mkdir ./www')
If you look at the source for abort, it looks like this:
10 def abort(msg):
21 from fabric.state import output
22 if output.aborts:
23 sys.stderr.write("\nFatal error: %s\n" % str(msg))
24 sys.stderr.write("\nAborting.\n")
25 sys.exit(1)
So, the exception would be a SystemExit exception. While you could catch this, the proper way to do it is outlined above using settings.
It is nothing to handle with exception, it is from the fabric api
try to set the entire script's warn_only setting to be true with
env.warn_only = True
Normally, when you get an uncaught exception, Python will print the exception type along with the error message:
>>> raise IOError("Error message.")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: Error message.
If that's not happening, you're probably not getting an exception at all.
If you really want to catch an arbitrary exception and print it, you want to catch Exception or BaseException. BaseException will include even things like KeyboardInterrupt, though, so be careful with that.
def init():
try:
local('mkdir ./www')
except BaseException as e:
print "local() threw a", type(e).__name__
raise # Reraise the exception
In general:
try:
some_code()
except Exception, e:
print 'Hit An Exception', e
raise
Will tell you what the exception was but if you are not planning on actually handling some of the exceptions then simply getting rid of the try: except: lines will have exactly the same effect.
Also if you run your code under a debugger then you can look at the exception(s) that you hit in more detail.
def init():
try:
local('mkdir ./www')
except Exception as e:
print e.__class__.__name__
That's all there is to it!
edit: Just re-read your question and realized that my code would only print "Fatal" in your case. It looks like fabric is throwing an error and returning their own error code so you would have to look at the documentation. I don't have any experience with fabric so I'd suggest to look here if you haven't already. Sorry if this isn't helpful!