I have a decent sized code in python which I changed a bit and now I see that when executing the code, the script does not bail on language errors like missing function definitions.
I did not think it was possible to continue running a script with missing function definition. I used a function in one place with the intention of actually defining the function before running but forgot about it and ran the code. To my surprise I just got the following line printed to the console --
Worker instance has no attribute update_parse_status
Where Worker is the class I was supposed to add the update_parse_status() call to.
I realized that I had a try and a generic catch all exception handler around the code in question like this --
try:
Worker.update_parse_status()
except Exception as e:
print e
So python just throws a AttributeError and I was unknowingly catching it. It taught me a valuable lesson to not have catch all exception handlers. Also coming from a compiled language is there a better way to handle this? I mean, can I some how make sure python just exits on language errors? It will be very helpful to atleast weed out the obvious bugs (though I understand that bad code got me into this situation).
In python all names are looked up at runtime. Therefor, what you refer to as "language" errors are no different than runtime errors.
This makes python different than many other languages. One of the advantages of that fact is that you can easily customize the way names are looked up (e.g. by overriding class's __getattr__) for example.
You can make use of analyzers (e.g. pyflakes is pretty cool, but there are many others) to detect some of the errors before running your program, but no tool would be able to detect all.
You already answered your own question.
Don't catch exceptions if you don't need too ...
except Exception is dangerous
if you must catch the exception but use some logging mechanism to track it down.
Finally, make sure python just exits on errors: don't catch exceptions where it is not needed to guarantee the flow of the program.
Python, being an interpreter, does this. The best way to counter this is unit testing. The unittest module can help here, and setuptools can run these tests for you. You can package these with your source distribution, as well.
Of course, as you discovered, never use the catch-all exception. That also masks errors that unit tests could catch. You can also run your code under a debugger when developing it so this can be tracked more easily as well. Using the catch-all also makes debugging harder (or even impossible).
You can also do static analysys, using tools such as pylint or pyflakes.
Related
I was trying to document the possible errors that could occur as a result of a remote procedure call (RPC).
It occurred to me that it might be nice if you could do this with a little magic. Is there anything in python to infer possible exceptions (kinda like checked exceptions in java).
I'm a little afraid of how complicated astroid and friends can be though...
I'm thinking of using some hideous hacks in unit tests for this..
Possible interesting links
https://github.com/python/mypy/issues/1773
When running some code I had written I was surprised to see that a function threw an exception that was not caught and so my code crashed. The thing is, Spyder never informed me that the function could even throw an exception.
Is there a setting somewhere that I have to turn on to be informed of this?
Python isn't Java. Your IDE will not warn you about uncaught exceptions, because Python's dynamic nature means that exceptions could be raised (or caught) almost anywhere and there's no amount of static analysis that'll work for every -- or even most -- cases. Often when you develop Flask or Django applications, you actually want exceptions to float all the way up to the "root" exception handlers.
Bottom line: No Python IDE is going to do this, and it would not be expected or considered generally desirable in Python programming.
I'm trying to write a highly-reliable piece of code in Python. The most common issue I run into is that after running for a while, some edge case will occur and raise an exception that I haven't handled. This most often happens when using external libraries - without reading the source code, I don't know of an easy way to get a list of all exceptions that might be raised when using a specific library function, so it's hard to know what to handle. I understand that it's bad practice to use catch-all handlers, so I haven't been doing that.
Is there a good solution for this? It seems like it should be possible for a static analysis tool to check that all exceptions are handled, but I haven't found one. Does this exist? If not, why? (is it impossible? a bad idea? etc) I especially would like it to analyze imported code for the reason explained above.
"it's bad practice to use catch-all handlers" to ignore exceptions:
Our web service has an except which wraps the main loop.
except:
log_exception()
attempt_recovery()
This is good, as it notifies us (necessary) of the unexpected error and then tries to recover (not necessary). We can then go look at those logs and figure out what went wrong so we can prevent it from hitting our general exception again.
This is what you want to avoid:
except:
pass
Because it ignores the error... then you don't know an error happened and your data may be corrupted/invalid/gone/stolen by bears. Your server may be up/down/on fire. We have no idea because we ignored the exception.
Python doesn't require registering of what exceptions might be thrown, so there are no checks for all exceptions a module might throw, but most will give you some idea of what you should be ready to handle in the docs. Depending on your service, when it gets an unhandled exception, you might want to:
Log it and crash
Log it and attempt to continue
Log it and restart
Notice a trend? The action changes, but you never want to ignore it.
Great question.
You can try approaching the problem statically (by, for instance, introducing a custom flake8 rule?), but, I think, it's a problem of testing and test coverage scope. I would approach the problem by adding more "negative path"/"error handling" checks/tests to the places where third-party packages are used, introducing mock side effects whenever needed and monitoring coverage reports at the same time.
I would also look into Mutation Testing idea (check out Cosmic Ray Python package). I am not sure if mutants can be configured to also throw exceptions, but see if it could be helpful.
Instead of trying to handle all exceptions, which is very hard as you described, why don't you catch all, but exclude some, for example the KeyboardInterrupt you mentioned in the comments?
This may help
I just wonder, when using try.. except, what's the difference between using the Base class "Except" and using a specific exception like "ImportError" or "IOError" or any other specific exception. Are there pros and cons between one and the other?
Never catch the base exception. Always capture only the specific exceptions that you know how to handle. Everything else should be left alone; otherwise you are potentially hiding important errors.
Of course it has advantages using the correct exception for the corresponding issue. However Python already defined all possible errors for coding problems. But you can make your own exception classes by inheriting from the Exception class. This way you can make more meaningful errors for spesific parts of your code. You can even make the expections print errors like this.
SomeError: 10 should have been 5.
Provides easier debugging of the code.
For more info.
In Python 2.7 (which we need to support), the initialization function for a C/C++ extension should be declared with the PyMODINIT_FUNC macro, which effectively makes the function void. However, I'm not sure how we should handle errors that occurs during this function. We could throw a C++ exception within the function, but I'm not thrilled about that idea. Is there a better way?
Here's the background: In order to work around an architectural problem that we cannot address in a single release, we need to have the user call Python via a script that we provide rather than directly from the Python executable. By checking the process name, we can detect the situation where the user calls via the executable rather than the script. In this case, we would like to issue an error message and then terminate gracefully.
You can use one of the PyErr_Set* methods.
Exceptions are always checked after calling init_module_name().
It's not explicitly stated in the documentation, but if you look at the examples, or if you read the source, you'll see that it is true.