I'm working with the pdb module in python; I just recently found out about it, so I'm a beginner. What I want to do is have a variable that, if True, will call set_trace() on ALL failures that occur in the script without putting it all in a try/except statement. For example, I want the following functionality without the try/except:
from pdb import set_trace
debug = True
try:
#entire script here
except Exception, e:
if debug:
set_trace()
else:
print e
Is there a way to do this without that huge try except statement (also without having to do an if-statement for every single command that could have a failure)?
Thanks.
You could make a custom excepthook.
When an exception is raised and uncaught, the interpreter calls sys.excepthook
with three arguments, the exception class, exception instance, and a traceback
object
import sys
import pdb
debug = True
def excepthook(type_, value, traceback):
if debug:
pdb.set_trace()
else:
print(value)
# Per mgilson's suggestion, to see the full traceback error message
# sys.__excepthook__(type_, value, traceback)
sys.excepthook = excepthook
1 / 0
If you want the usual traceback error message when debug is False, then the above could be simplified to
if debug:
sys.excepthook = lambda type_, value, traceback: pdb.set_trace()
Related
I'm setting up a logger in my script like shown at the bottom. This works fine for my purposes and logs my __main__ log messages and those of any modules I use to stdout and a log file.
During program execution a module call that I'm using xarray.open_dataset(file, engine="cfgrib") raises an Error in some conditions and produces the following log output:
2023-02-18 10:02:06,731 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
...
How can I access this output during program execution?
The raised error in the cfgrib module is handled there gracefully and program execution can continue, but the logic of my program requires that I access the error message, in particular the part saying shortName='i10fg' in order to handle the error exhaustively.
Here is how my logger is set up:
def init_log():
"""initialize logging
returns logger using log settings from the config file (settings.toml)
"""
# all settings from a settings file with reasonable defaults
lg.basicConfig(
level=settings.logging.log_level,
format=settings.logging.format,
filemode=settings.logging.logfile_mode,
filename=settings.logging.filename,
)
mylogger = lg.getLogger(__name__)
stream = lg.StreamHandler()
mylogger.addHandler(stream)
clg.install(
level=settings.logging.log_level,
logger=mylogger,
fmt="%(asctime)s %(levelname)s:\t%(message)s",
)
return mylogger
# main
log = init_log()
log.info('...reading files...')
I went through the python logging documentation and cookbook. While this contains ample examples on how to modify logging for various purposes, I could not find an example for accessing and reacting to a log message during program execution.
The Exception in my logs look this:
2023-02-20 12:22:37,209 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 660, in build_dataset_components
dict_merge(variables, coord_vars)
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 591, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='time' value=Variable(dimensions=('time',), data=array([1640995200, 1640998800, 1641002400, ..., 1672520400, 1672524000,
1672527600])) new_value=Variable(dimensions=('time',), data=array([1640973600, 1641016800, 1641060000, 1641103200, 1641146400,
1641189600, 1641232800, 1641276000, 1641319200, 1641362400,
I cannot catch the Exception directly for some reason:
...
import sys
from cfgrib.dataset import DatasetBuildError
...
try:
df = xr.open_dataset(file, engine="cfgrib").to_dataframe()
# triggering error manually like with the two lines below works as expected
# raise Exception()
# raise DatasetBuildError()
except Exception as e:
print('got an Exception')
print(e)
print(e.args)
except BaseException as e:
print('got a BaseException')
print(e.args)
except DatasetBuildError as e:
print(e)
except:
print('got any and all exception')
type, value, traceback = sys.exc_info()
print(type)
print(value)
print(traceback)
Unless I uncomment the two lines where I raise the exception manually, the except clauses are never triggered, event though I can see the DatabaseBuildError in my logs.
Not sure if this has any bearing, but while I can see the Exception as quoted above in my file log, it is not printed to stdout.
If I write something in Python and things go awry, I automatically get a traceback.
For example:
#!/usr/bin/env python
print("this will raise a division by zero exception")
print(2/0)
It automatically throws a traceback with the exception
>>> %Run test.py
this will raise a division by zero exception
Traceback (most recent call last):
File "/home/pi/Desktop/test.py", line 4, in <module>
print(2/0)
ZeroDivisionError: division by zero
>>>
Now, let's assume that I don't want the program to crash-and-burn, but I want to handle the exception gracefully - say I need to close files or whatever.
Viz.:
#!/usr/bin/env python
import sys
try:
print("Assume I'm doing something with a resource, like a file")
print(". . . and something goes wrong\n")
print("(This will raise a division by zero exception when I try to \"print(2/0)\")\n")
print(2/0)
except BaseException as e:
print(". . . and I want to handle the exception gracefully:\n")
print("Oops! Something happened! (",e, ")\n")
sys.exit(0)
Which gets me the expected:
>>> %Run test.py
Assume I'm doing something with a resource, like a file
. . . and something goes wrong
(This will raise a division by zero exception when I try to "print(2/0)")
. . . and I want to handle the exception gracefully:
Oops! Something happened! ( division by zero )
─────────────────────────────────────────────────────────────────────────────────────────────────
Python 3.7.3 (/usr/bin/python3)
>>>
However, if I try to handle the exception manually, I am told that in order to get the system provided traceback, (the call stack and location of the error), I have to import traceback.
When I do that, I can get all the information I want.
#!/usr/bin/env python
import sys
import traceback
try:
print("Assume I'm doing something with a resource, like a file")
print(". . . and something goes wrong\n")
print("(This will raise a division by zero exception when I try to \"print(2/0)\")\n")
print(2/0)
except BaseException as e:
print(". . . and I want to handle the exception gracefully:\n")
print("Oops! Something happened! (",e, ")\n")
traceback.print_exc()
sys.exit(0)
And the expected result:
────────────────────────────────────────────────────────────────────────────────────────────────
Python 3.7.3 (/usr/bin/python3)
>>> %Run test.py
Assume I'm doing something with a resource, like a file
. . . and something goes wrong
(This will raise a division by zero exception when I try to "print(2/0)")
. . . and I want to handle the exception gracefully:
Oops! Something happened! ( division by zero )
Traceback (most recent call last):
File "/home/pi/Desktop/test.py", line 9, in <module>
print(2/0)
ZeroDivisionError: division by zero
─────────────────────────────────────────────────────────────────────────────────────────────────
Python 3.7.3 (/usr/bin/python3)
>>>
"Traceback" appears to already exist as the traceback is automatic on unhandled exceptions
Why do I have to import traceback if I use a try/except block, but don't if I don't?
The traceback reflects the state of the Python Interpreter at a given point during the execution of some code. The Python Interpreter obviously has access to its own state, and shares it with the user, by printing that error message, when an exception is thrown and not handled. It also gives the code access to it – via the traceback module, which is why you need to import it.
The default behavior of Python when an exception is raised that isn't caught in a try...except is to print a traceback and exit. If you do catch it, then the default doesn't happen. You have to decide what to do with it. If you decide to print the traceback you now have to explicitly do it by importing the traceback module and using the methods there. However, that's not strictly necessary as you could just inspect the current traceback object (from sys.exc_info()), and write your own formatter.
"Traceback" appears to already exist as the traceback is automatic on unhandled exceptions
What you're seeing is the default behavior of sys.excepthook, which is to print a traceback out to stderr anyway, just before the program exits from an unhandled exception.
See for yourself with:
# example.py
import sys
sys.excepthook = lambda *args: None
print("before")
1/0
print("after")
The script above will exit non-zero, but no traceback will be printed since the hook was replaced with a no-op. And "after" will not get printed, because the process will have already exited. So, it is not quite that a traceback is "automatic" on unhandled exceptions, but rather that is the default configuration.
By calling sys.exit(0) you're intentionally suppressing the ZeroDivisionError exception. The interpreter will exit with a zero return code, and as far as the except hook is concerned you have "handled" the ZeroDivisionError and are opting out of the traceback printing (src).
The traceback exists, and the interpreter can use it (e.g. print it) , but you can only use it if you import it. If you do not use a try/except block you do not have a chance to import it.
I'm a newbie in Flask and I am trying to display the Built-In Exceptions in python but I can't seem to have them display on my end.
NOTE:
set FLASK_DEBUG = 0
CODE:
def do_something:
try:
doing_something()
except Exception as err:
return f"{err}"
Expectation:
It will display one of the built-in exceptions:
KeyError
IndexError
NameError
Etc.
Reality:
It will return the line of code that didn't worked which is more ambiguous to the end user.
Also:
I have no problem seeing the errors when the debug mode is ON but that's not something that I want to do if I open them in public
Flask supplies you with a function that enables you to register an error handler throughout your entire app; you can do something as shown below:
def handle_exceptions(e):
# Log exception in your logs
# get traceback and sys exception info and log as required
# app.logger.error(getattr(e, 'description', str(e)))
# Print traceback
# return your response using getattr(e, 'code', 500) etc.
# Exception is used to catch all exceptions
app.register_error_handler(Exception, handle_exceptions)
In my honest opinion, this is the way to go. - Following the structure found in werkzeug.exceptions.HTTPException as an example is a solid foundation.
Having a unified exception handler that will standardise your Exception handling, visualisation and logging will make your life a tad better. :)
Try with this:
def do_something:
try:
doing_something()
except Exception as err:
return f"{err.__class__.__name__}: {err}"
I have a python script which every 10 minutes sends me an email with everything written in the console. I am running this with the crontab in my ubuntu 18.04 vps.
Sometimes it doesn't send the mail so I assume that when an error happens execution stops but how can I get the errors to be written in a txt file so I can analyze the error ?
Logging Module
To demonstrate the approach with the logging module, this would be the general approach
import logging
# Create a logging instance
logger = logging.getLogger('my_application')
logger.setLevel(logging.INFO) # you can set this to be DEBUG, INFO, ERROR
# Assign a file-handler to that instance
fh = logging.FileHandler("file_dir.txt")
fh.setLevel(logging.INFO) # again, you can set this differently
# Format your logs (optional)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter) # This will set the format to the file handler
# Add the handler to your logging instance
logger.addHandler(fh)
try:
raise ValueError("Some error occurred")
except ValueError as e:
logger.exception(e) # Will send the errors to the file
And if I cat file_dir.txt
2019-03-14 14:52:50,676 - my_application - ERROR - Some error occurred
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: Some error occurred
Print to File
As I pointed out in the comments, you could accomplish this with print as well (I'm not sure you will be applauded for it)
# Set your stdout pointer to a file handler
with open('my_errors.txt', 'a') as fh:
try:
raise ValueError("Some error occurred")
except ValueError as e:
print(e, file=fh)
cat my_errors.txt
Some error occurred
Note that logging.exception includes the traceback in this case, which is one of the many huge benefits of that module
Edit
In the interest of completeness, the traceback module leverages a similar approach as print, where you can supply a file handle:
import traceback
import sys
with open('error.txt', 'a') as fh:
try:
raise ValueError("Some error occurred")
except ValueError as e:
e_type, e_val, e_tb = sys.exc_info()
traceback.print_exception(e_type, e_val, e_tb, file=fh)
This will include all of the information you want from logging
You can use the logging module as suggested in the comments (possibly superior but outside the scope of my knowledge), or catch the errors with try and except like:
try:
pass
#run the code you currently have
except Exception as e: # catch ALLLLLL errors!!!
print(e) # or more likely you'd want something like "email_to_me(e)"
Although this is generally frowned upon to catch all exceptions, because then should your program fail for whatever reason it will get gobbled up in the except clause so a better approach is to figure out what specific error you are encountering like IndexError and then just catch this specific error like:
try:
pass
#run the code you currently have
except IndexError as e: # catch only indexing errors!!!
print(e) # or more likely you'd want something like "email_to_me(e)"
To be able to debug and not only know the kind of error that happened, you can also get the error stack using traceback module (usually in the starting package of modules):
import traceback
try:
my_function()
except Exception as e:
print(e)
traceback.print_exc()
And then run your code 'my_code.py' in console usig >>
python my_code.py >> my_prints.txt
All the prints of your code will then be written in this .txt file, including the printed error and its stack. This is very interesting in your case or while running code on a docker if you want to detach yourself from it with ctrl+p+q and still know what is printed.
I'm using Fabric to automate, including the task of creating a directory. Here is my fabfile.py:
#!/usr/bin/env python
from fabric.api import *
def init():
try:
local('mkdir ./www')
except ##what exception?##:
#print exception name to put in above
Run fab fabfile.py and f I already have ./www created an error is raised, but I don't know what kind, so I don't know how to handle the error yet. Fabric only prints out the following:
mkdir: cannot create directory ‘./www’: File exists
Fatal error: local() encountered an error (return code 1) while executing 'mkdir ./www'
Aborting.
What I want to do is be able to find out the error type so that I can except my errors properly without blanket statements. It would be really helpful if an answer does not just tell me how to handle a mkdir exception, but print (or otherwise find the name to) any exception I may run into down the line (mkdir is just an example).
Thank you!
The issue is that fabric uses subprocess for doing these sorts of things. If you look at the source code for local you can see it doesn't actually raise an exception. It calls suprocess.Popen and uses communicate() to read stdout and stderr. If there is a non-zero return code then it returns a call to either warn or abort. The default is abort. So, to do what you want, try this:
def init():
with settings(warn_only=True):
local('mkdir ./www')
If you look at the source for abort, it looks like this:
10 def abort(msg):
21 from fabric.state import output
22 if output.aborts:
23 sys.stderr.write("\nFatal error: %s\n" % str(msg))
24 sys.stderr.write("\nAborting.\n")
25 sys.exit(1)
So, the exception would be a SystemExit exception. While you could catch this, the proper way to do it is outlined above using settings.
It is nothing to handle with exception, it is from the fabric api
try to set the entire script's warn_only setting to be true with
env.warn_only = True
Normally, when you get an uncaught exception, Python will print the exception type along with the error message:
>>> raise IOError("Error message.")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: Error message.
If that's not happening, you're probably not getting an exception at all.
If you really want to catch an arbitrary exception and print it, you want to catch Exception or BaseException. BaseException will include even things like KeyboardInterrupt, though, so be careful with that.
def init():
try:
local('mkdir ./www')
except BaseException as e:
print "local() threw a", type(e).__name__
raise # Reraise the exception
In general:
try:
some_code()
except Exception, e:
print 'Hit An Exception', e
raise
Will tell you what the exception was but if you are not planning on actually handling some of the exceptions then simply getting rid of the try: except: lines will have exactly the same effect.
Also if you run your code under a debugger then you can look at the exception(s) that you hit in more detail.
def init():
try:
local('mkdir ./www')
except Exception as e:
print e.__class__.__name__
That's all there is to it!
edit: Just re-read your question and realized that my code would only print "Fatal" in your case. It looks like fabric is throwing an error and returning their own error code so you would have to look at the documentation. I don't have any experience with fabric so I'd suggest to look here if you haven't already. Sorry if this isn't helpful!