I am trying to execute a list of queries in Spark, but if the query does not run correctly, Spark throws me the following error:
AnalysisException: "ALTER TABLE CHANGE COLUMN is not supported for changing ...
This is part of my code (i'm using python and Spark SQL on Databricks):
for index, row in df_tables.iterrows():
query = row["query"]
print ("Executing query: ")
try:
spark.sql(query)
print ("Query executed")
except (ValueError, RuntimeError, TypeError, NameError):
print("Unable to process your query dude!!")
else:
#do another thing
Is there any way to catch that exception? ValueError, RuntimeError, TypeError, NameError seems not working.
There's no so much information about that in the Spark webpage.
I found AnalysisException defined in pyspark.sql.utils.
https://spark.apache.org/docs/3.0.1/api/python/_modules/pyspark/sql/utils.html
import pyspark.sql.utils
try:
spark.sql(query)
print ("Query executed")
except pyspark.sql.utils.AnalysisException:
print("Unable to process your query dude!!")
You can modify the try except statement as below :
try:
spark.sql(query)
print ("Query executed")
except Exception as x:
print("Unable to process your query dude!!" + \
"\n" + "ERROR : " + str(x))
I think it depends on your requirements. If you're running full workflow on this query and if you want them to pass through then your code will work fine. But let's say you want your workflow or datapipeline to fail, then you should exit from that except block.
The exact exception you may not get, but you can definitely get overview using
except Exception as x:
print(str(x))
You can use logging module for putting more information in logs for further investigation.
I want to propose a way to pick specific Exceptions.
I had the problem to find if some table already existed. The simplest way that i have found to do this is like this. Of course that this can break if Spark maintainers change the message of the exception, but i think that they do not have reason to do this, in this case.
import pyspark.sql.utils
try:
spark.read.parquet(SOMEPATH)
except pyspark.sql.utils.AnalysisException as e:
if "Path does not exist:" in str(e):
# Finding specific message of Exception.
pass # run some code to address this specific case.
else:
# if this is not the AnalysisException that i was waiting,
# i throw again the exception
raise (e)
except Exception as e:
# if is another exception i can catch like this
print(e)
raise (e)
Related
I have a script which uses pygetwindow module to do some operations on a specific window. While the script runs, I get the following exception:
File "C:\Program Files (x86)\Python38-32\lib\site-packages\pygetwindow\_pygetwindow_win.py", line 237, in activate
_raiseWithLastError()
File "C:\Program Files (x86)\Python38-32\lib\site-packages\pygetwindow\_pygetwindow_win.py", line 97, in _raiseWithLastError
raise PyGetWindowException('Error code from Windows: %s - %s' % (errorCode, _formatMessage(errorCode)))
pygetwindow.PyGetWindowException: Error code from Windows: 0 - The operation completed successfully.
I'm okay with the exception occurring but I want to catch this exception explicitly. I have done the following to try and catch this exception:
try:
#implementation
except pygetwindow.PyGetWindowException:
#handle exception
and
try:
#implementation
except PyGetWindowException:
#handle exception
Neither of the above catches the exception. If I use either of the above, I get another exception:
NameError: name 'PyGetWindowException' is not defined
or
NameError: name 'pygetwindow' is not defined
I don't want to catch the general Exception and then handle it since in case of other exceptions, I want to handle it differently. Is there something wrong in how I'm trying to catch this exception or is there a way to avoid this exception altogether?
EDIT: To be very clear, I have already imported pygetwindow.
You should have import pygetwindow at the begging of your script. It complains about not knowing what pygetwindow is.
Update
From the source file, it was clear that in order to use PyGetWindowException, you need to import the exception specifically (and not just import pygetwindow). Therefore, in order to catch the exception, one will have to do:
from pygetwindow import PyGetWindowException
After this import, you can use the exception in the normal way:
try:
#implementation
except PyGetWindowException:
#handle exception
Update 2
Another general way to do this would be to get the exception name from general exception and compare.
try:
try:
#implementation
except Exception as e:
if e.__class__.__name__ == 'PyGetWindowException':
#handle exception
else:
raise e
except Exception as e:
#handle other exceptions except pygetwindow exception
Original answer (not recommended)
Found a way to solve this question in this answer.
From the source of pygetwindow, it was clear that, whenever PyGetWindowException is raised, it is accompanied by the text:
"Error code from Windows:"
which indicates the error code given by Windows.
Based on this information, I did the following:
try:
try:
#Implementation
except Exception as e:
if "Error code from Windows" in str(e)
# Handle pygetwindow exception
else:
raise e
except Exception as e:
#handle other exceptions
This is another way (although the first one and second one are the correct and straightforward solutions) to solve the problem.
I have a try/except where I repeat the except portion frequently in my code. This led me to believe that it would be better to separate the except portion into a function.
Below is my use case:
try:
...
except api.error.ReadError as e:
...
except api.error.APIConnectionError as e:
...
except Exception as e:
...
How would I go about separating this logic into a function so I can do something as simple as:
try:
...
except:
my_error_handling_function(e)
Just define the function:
def my_error_handling(e):
#Do stuff
...and pass in the exception object as the parameter:
try:
#some code
except Exception as e:
my_error_handling(e)
Using just a generic Exception type will allow you to have a single except clause and handle and test for different error types within your handling function.
In order to check for the name of the caught exception, you can get it by doing:
type(e).__name__
Which will print the name, such as ValueError, IOError, etc.
I would suggest refactoring your code so the try/except block is only present in a single location.
For instance, an API class with a send() method, seems like a reasonable candidate for containing the error handling logic you have described in your question.
Define your function:
def my_error_handling(e):
#Handle exception
And do what you're proposing:
try:
...
except Exception as e:
my_error_handling_function(e)
You can handle logic by getting the type of the exception 'e' within your function. See: python: How do I know what type of exception occurred?
If you don't like try-catch statement, you can use exception-decouple package and decorator.
from exception_decouple import redirect_exceptions
def my_error_handling(arg, e):
#Do stuff
#redirect_exceptions(my_error_handling, api.error.ReadError, api.error.APIConnectionError)
def foo(arg):
...
I have:
MY_PATH_DIR = 'path/to/my/json/file.json'
try:
with open(MY_PATH_DIR, 'r') as f:
MY_PATH_DIR = json.load(f)
except IOError, RuntimeError, ValueError:
pass
except PermissionDenied:
pass
And I want to catch all possible errors. With
IOError - I am catching errors when the file doesn't exist or has a
syntax error (non valid JSON).
RuntimeError - couldn't test it but I think that makes sense from the
documentation in case of an unexpected error
ValueError - I got from here in case nothing got returned
PermissionDenied - is a specific Django error
Are there any other Exceptions that would make sense? I'm not sure if OSError makes sense here. I think that would be raised earlier, right?
The purpose of capturing exceptions is to control the program's behavior when something bad happened, but in an expected way. If you are not even sure what would cause that exception happen, capturing it would only swallow the underlying programming errors you might have.
I wouldn't add as many kinds of exception as possible to that single block of code, you should only add what you care about. To take it to extreme, each line of code would yield certain exceptions but for obvious reason you couldn't do try except for all of them.
Edit:
For the sake of correctness, since you mentioned I don't want my code to break in any case, you could simply do:
try:
# json.load
except Exception as e:
print "Let's just ignore all exceptions, like this one: %s" % str(e)
This is would give you what exception happens as output.
import random
import sys
def main():
"""Demonstrate the handling of various kinds of exceptions."""
# This is like what you are doing in your code.
exceptions = IOError, RuntimeError, ValueError
try:
raise random.choice(exceptions)()
except exceptions as error:
print('Currently handling:', repr(error))
# The following is not much different from Shang Wang's answer.
try:
raise random.choice(exceptions)()
except Exception as error:
print('Currently handling:', repr(error))
# However, the following code will sometimes not handle the exception.
exceptions += SystemExit, KeyboardInterrupt, GeneratorExit
try:
raise random.choice(exceptions)()
except Exception as error:
print('Currently handling:', repr(error))
# The code can be slightly altered to take the new errors into account.
try:
raise random.choice(exceptions)()
except BaseException as error:
print('Currently handling:', repr(error))
# This does not take into account classes not in the exception hierarchy.
class Death:
pass
try:
raise Death()
except BaseException as error:
print('Currently handling:', repr(error))
# If your version of Python does not consider raising an exception from an
# instance of a class not derived from the BaseException class, the way to
# get around this problem would be with the following code instead.
try:
raise Death()
except:
error = sys.exc_info()[1]
print('Currently handling:', repr(error))
if __name__ == '__main__':
main()
I have:
try:
...
except Exception, e:
print "Problem. %s" % str(e)
However, somewhere in try, i will need it to behave as if it encountered an Exception. Is it un-pythonic to do:
try:
...
raise Exception, 'Type 1 error'
...
except Exception, e:
print "Problem. Type 2 error %s" % str(e)
I think this is a bad design. If you need to take some action if (and only if) an exception wasn't raised, that is what the else clause is there for. If you need to take some action unconditionally, that's what finally is for. here's a demonstration:
def myraise(arg):
try:
if arg:
raise ValueError('arg is True')
except ValueError as e:
print(e)
else:
print('arg is False')
finally:
print("see this no matter what")
myraise(1)
myraise(0)
You need to factor the unconditional code into finally and put the other stuff in except/else as appropriate.
I think what you are doing is "unPythonic". Trys should really only cover the small part (ideally one line) of the code which you expect might sometimes fail in a certain way. You should be able to use try/except/else/finally to get the required behaviour:
try:
#line which might fail
except ExceptionType: # the exception type which you are worried about
#what to do if it raises the exception
else:
#this gets done if try is successful
finally:
#this gets done last in both cases (try successful or not)
I have code:
try:
print test.qwerq]
try:
print test.sdqwe]
except:
pass
except:
pass
How to print debug info for all errors in nested try ?
Re-raise exceptions.
try:
print test[qwerq]
try:
print test[qwe]
except:
# Do something with the exception.
raise
except:
# Do something here too, just for fun.
raise
It should be noted that in general you don't want to do this. You're better off not catching the exception if you're not going to do anything about it.
If you want to just print the call stack and not crash, look into the traceback module.