Solution at the bottom of this question!
In my situation, my client sends data to the server to do CPU-intensive workloads. I would like for my XML-RPC server to be able to raise an exception when there is a problem with the input data.
I have tried a simple raise Exception('description of the error') on the server side, but on my client, I get: xmlrpc.client.Fault: <Fault 1: "<class 'TypeError'>:exceptions must derive from BaseException">
On the client side, I would like to be able to do the following:
try:
proxy.processData(data)
except:
[... code to handle the exception ...]
and on the server side:
def processData(data):
if 'somethingRequired' not in data:
raise Exception('Something Required not in data!')
[... continue processing ...]
I think it may also be ideal to have the exception be a custom type on the client side, so that exceptions caused by the XML-RPC module (for example, not being able to connect to the XML-RPC server) don't get interpreted as data input errors.
Thanks!
Update: Solution
Above my processData method, I added the following (a custom exception):
class ProcessingError(Exception):
pass
then, in my client code, I used the following for error handling::
try:
proxy.processData(data)
except xmlrpc.client.Fault as e:
print(e.faultString)
Note that if you would like to differentiate between custom exceptions and exceptions from the XML-RPC module (like connection errors), you can parse str(e) in the except: block to check if your exception was in the error or not. I don't have that in my code above.
The xmlrpc.client.Fault class has a few attributes that can be found here.
May need to raise custom exception in your case, Exception is trying to refer to a parent class, which you maybe missing. Something like this:
class SomeSortOfException(Exception):
pass
This should be at the top or before the scope of the processData() function.
I've encountered a pretty standard issue and was wondering if my solution is the right way to go.
Every time an exception occures I want it to be caught and logged by the caller, and then re-raised.
Since I dont want to repeat logging messages every time, I created a custom exception which saves the message data and also logs.
class LoggingException(Exception):
def __init__(self, message, package_id):
# Get caller informat
caller = getframeinfo(stack()[2][0])
self.filename = caller.filename
self.function = caller.function
# Set message info and log
self.message = message
if (LogManager.log_handler is None):
print(message)
else:
LogManager.l(package_id, LogLevelEnum.ERROR, message)
Use case:
def main_func():
try:
secondary_func()
except Exception as ex:
raise LoggingException("Some log") from ex
def secondary_func():
raise LoggingException("Exception info")
The problem is im not completley sure having an exception do any operations is a good idea, and this to generic for it to not have a standard python solution.
Note: I am not using the python logging module due to product constraints.
Trying to get caller information like that is going to be unreliable. Also, there will be cases where it's not the immediate caller that you are interested in.
The idea of the exception logging itself seems sensible. To compromise, I would move the logging functionality into a separate method that you could trigger explicitly. Exceptions are, after all, mostly regular objects:
class LoggingException(Exception):
def __init__(self, message, package_id):
# Set message info
super.__init__(message)
self.package_id = package_id
def log(self, manager=None):
if manager.log_handler is None:
print(super().__str__())
else:
manager.l(self.package_id, LogLevelEnum.ERROR, super()..__str__())
Now you can trigger the logging operation whenever you want, without having to reconstruct the message:
try:
...
except LoggingException as e:
e.log(some_manager)
raise
This gives you the option of really re-raising the error, as shown here, or chaining it as in your example. I highly recommend against chaining unless you have a really good reason to do it.
We've been experimenting with sqlalchemy's disconnect handling, and how it integrates with ORM. We've studied the docs, and the advice seems to be to catch the disconnect exception, issue a rollback() and retry the code.
eg:
import sqlalchemy as SA
retry = 2
while retry:
retry -= 1
try:
for name in session.query(Names):
print name
break
except SA.exc.DBAPIError as exc:
if retry and exc.connection_invalidated:
session.rollback()
else:
raise
I follow the rationale -- you have to rollback any active transactions and replay them to ensure a consistent ordering of your actions.
BUT -- this means a lot of extra code added to every function that wants to work with data. Furthermore, in the case of SELECT, we're not modifying data and the concept of rollback/re-request is not only unsightly, but a violation of the principle of DRY (don't repeat yourself).
I was wondering if others would mind sharing how they handle disconnects with sqlalchemy.
FYI: we're using sqlalchemy 0.9.8 and Postgres 9.2.9
The way I like to approach this is place all my database code in a lambda or closure, and pass that into a helper function that will handle catching the disconnect exception, and retrying.
So with your example:
import sqlalchemy as SA
def main():
def query():
for name in session.query(Names):
print name
run_query(query)
def run_query(f, attempts=2):
while attempts > 0:
attempts -= 1
try:
return f() # "break" if query was successful and return any results
except SA.exc.DBAPIError as exc:
if attempts > 0 and exc.connection_invalidated:
session.rollback()
else:
raise
You can make this more fancy by passing a boolean into run_query to handle the case where you are only doing a read, and therefore want to retry without rolling back.
This helps you satisfy the DRY principle since all the ugly boiler-plate code for managing retries + rollbacks is placed in one location.
Using exponential backoff (https://github.com/litl/backoff):
#backoff.on_exception(
backoff.expo,
sqlalchemy.exc.DBAPIError,
factor=7,
max_tries=3,
on_backoff=lambda details: LocalSession.get_main_sql_session().rollback(),
on_giveup=lambda details: LocalSession.get_main_sql_session().flush(), # flush the session
logger=logging
)
def pessimistic_insertion(document_metadata):
LocalSession.get_main_sql_session().add(document_metadata)
LocalSession.get_main_sql_session().commit()
Assuming that LocalSession.get_main_sql_session() returns a singleton.
I want to know what is the best way of checking an condition in Python definition and prevent it from further execution if condition is not satisfied. Right now i am following the below mentioned scheme but it actually prints the whole trace stack. I want it to print only an error message and do not execute the rest of code. Is there any other cleaner solution for doing it.
def Mydef(n1,n2):
if (n1>n2):
raise ValueError("Arg1 should be less than Arg2)
# Some Code
Mydef(2,1)
That is what exceptions are created for. Your scheme of raising exception is good in general; you just need to add some code to catch it and process it
try:
Mydef(2,1)
except ValueError, e:
# Do some stuff when exception is raised, e.message will contain your message
In this case, execution of Mydef stops when it encounters raise ValueError line of code, and goes to the code block under except.
You can read more about exceptions processing in the documentation.
If you don't want to deal with exceptions processing, you can gracefully stop function to execute further code with return statement.
def Mydef(n1,n2):
if (n1>n2):
return
def Mydef(n1,n2):
if (n1>n2):
print "Arg1 should be less than Arg2"
return None
# Some Code
Mydef(2,1)
Functions stop executing when they reach to return statement or they run the until the end of definition. You should read about flow control in general (not specifically to python)
I can't understand what sort of exceptions I should handle 'here and now', and what sort of exceptions I should re-raise or just don't handle here, and what to do with them later (on higher tier). For example: I wrote client/server application using python3 with ssl communication. Client is supposed to verify files on any differences on them, and if diff exists then it should send this 'updated' file to server.
class BasicConnection:
#blablabla
def sendMessage(self, sock, url, port, fileToSend, buffSize):
try:
sock.connect((url, port))
while True:
data = fileToSend.read(buffSize)
if not data: break
sock.send(data)
return True
except socket.timeout as toErr:
raise ConnectionError("TimeOutError trying to send File to remote socket: %s:%d"
% (url,port)) from toErr
except socket.error as sErr:
raise ConnectionError("Error trying to send File to remote socket: %s:%d"
% (url,port)) from sErr
except ssl.SSLError as sslErr:
raise ConnectionError("SSLError trying to send File to remote socket: %s:%d"
% (url,port)) from sslErr
finally:
sock.close()
Is it right way to use exceptions in python? The problem is: what if file.read() throws IOError? Should I handle it here, or just do nothing and catch it later? And many other possible exceptions?
Client use this class (BasicConnection) to send updated files to server:
class PClient():
def __init__(self, DATA):
'''DATA = { 'sendTo' : {'host':'','port':''},
'use_ssl' : {'use_ssl':'', 'fileKey':'', 'fileCert':'', 'fileCaCert':''},
'dirToCheck' : '',
'localStorage': '',
'timeToCheck' : '',
'buffSize' : '',
'logFile' : ''} '''
self._DATA = DATA
self._running = False
self.configureLogging()
def configureLogging(self):
#blablabla
def isRun(self):
return self._running
def initPClient(self):
try:
#blablabla
return True
except ConnectionError as conErr:
self._mainLogger.exception(conErr)
return False
except FileCheckingError as fcErr:
self._mainLogger.exception(fcErr)
return False
except IOError as ioErr:
self._mainLogger.exception(ioErr)
return False
except OSError as osErr:
self._mainLogger.exception(osErr)
return False
def startPClient(self):
try:
self._running = True
while self.isRun():
try :
self._mainLogger.debug("Checking differences")
diffFiles = FileChecker().checkDictionary(self._dict)
if len(diffFiles) != 0:
for fileName in diffFiles:
try:
self._mainLogger.info("Sending updated file: %s to remote socket: %s:%d"
% (fileName,self._DATA['sendTo']['host'],self._DATA['sendTo']['port']))
fileToSend = io.open(fileName, "rb")
result = False
result = BasicConnection().sendMessage(self._sock, self._DATA['sendTo']['host'],
self._DATA['sendTo']['port'], fileToSend, self._DATA['buffSize'])
if result:
self._mainLogger.info("Updated file: %s was successfully delivered to remote socket: %s:%d"
% (fileName,self._DATA['sendTo']['host'],self._DATA['sendTo']['port']))
except ConnectionError as conErr:
self._mainLogger.exception(conErr)
except IOError as ioErr:
self._mainLogger.exception(ioErr)
except OSError as osErr:
self._mainLogger.exception(osErr)
self._mainLogger.debug("Updating localStorage %s from %s " %(self._DATA['localStorage'], self._DATA['dirToCheck']))
FileChecker().updateLocalStorage(self._DATA['dirToCheck'],
self._DATA['localStorage'])
self._mainLogger.info("Directory %s were checked" %(self._DATA['dirToCheck']))
time.sleep(self._DATA['timeToCheck'])
except FileCheckingError as fcErr:
self._mainLogger.exception(fcErr)
except IOError as ioErr:
self._mainLogger.exception(ioErr)
except OSError as osErr:
self._mainLogger.exception(osErr)
except KeyboardInterrupt:
self._mainLogger.info("Shutting down...")
self.stopPClient()
except Exception as exc:
self._mainLogger.exception(exc)
self.stopPClient()
raise RuntimeError("Something goes wrong...") from exc
def stopPClient(self):
self._running = False
Is it correct? May be someone spend his own time and just help me to understand pythonic style of handling exceptions? I can't understand what to do with such exceptions as NameError, TypeError, KeyError, ValueError...and so on.......They could be thrown at any statement, at any time... and what to do with them, if I want to logged everything.
And what information should people usually log? If error occurs, what info about it I should log? All traceback, or just relevant message about it or something else?
I hope somebody helps me.
Thanks a lot.
In general, you should "catch" the exceptions that you expect to happen (because they may be caused by user error, or other environmental problems outside of your program's control), especially if you know what your code might be able to do about them. Just giving more details in an error report is a marginal issue, though some programs' specs may require doing that (e.g. a long-running server that's not supposed to crash due to such problems, but rather log a lot of state information, give the user a summary explanation, and just keep working for future queries).
NameError, TypeError, KeyError, ValueError, SyntaxError, AttributeError, and so on, can be thought of as due to errors in the program -- bugs, not problems outside of the programmer's control. If you're releasing a library or framework, so that your code is going to be called by other code outside of your control, then such bugs may quite likely be in that other code; you should normally let the exception propagate to help the other programmer debug their own bugs. If you're releasing an application, you own the bugs, and you must pick the strategy that helps you find them.
If your bugs show up while an end-user is running the program, you should log a lot of state information, and give the user a summary explanation and apologies (perhaps with a request to send you the log info, if you can't automate that -- or, at least, ask permission before you send anything from the user's machine to yours). You may be able to save some of the user's work so far, but often (in a program that's known to be buggy) that may not work anyway.
Most bugs should show up during your own testing of course; in that case, propagating the exception is useful as you can hook it up to a debugger and explore the bug's details.
Sometimes some exceptions like these show up just because "it's easier to ask forgiveness than permission" (EAFP) -- a perfectly acceptable programming technique in Python. In that case of course you should handle them at once. For example:
try:
return mylist[theindex]
except IndexError:
return None
here you might expect that theindex is generally a valid index into mylist, but occasionally outside of mylist's bounds -- and the latter case, by the semantics of the hypothetic app in which this snippet belongs, is not an error, just a little anomaly to be fixed by considering the list to be conceptually extended on both sides with infinite numbers of Nones. It's easier to just try/except than to properly check for positive and negative values of the index (and faster, if being out of bounds is a truly rare occurrence).
Similarly appropriate cases for KeyError and AttributeError happen less frequently, thanks to the getattr builtin and get method of dicts (which let you provide a default value), collections.defaultdict, etc; but lists have no direct equivalent of those, so the try/except is seen more frequently for IndexError.
Trying to catch syntax errors, type errors, value errors, name errors, etc, is a bit rarer and more controversial -- though it would surely be appropriate if the error was diagnosed in a "plug-in", third-party code outside your control which your framework/application is trying to load and execute dynamically (indeed that's the case where you're supplying a library or the like and need to coexist peacefully with code out of your control which might well be buggy). Type and value errors may sometimes occur within an EAFP pattern -- e.g. when you try to overload a function to accept either a string or a number and behave slightly differently in each case, catching such errors may be better than trying to check types -- but the very concept of functions thus overloaded is more often than not quite dubious.
Back to "user and environmental errors", users will inevitably make mistakes when they give you input, indicate a filename that's not actually around (or that you don't have permission to read, or to write if that's what you're supposed to be doing), and so on: all such errors should of course be caught and result in a clear explanation to the user about what's gone wrong, and another chance to get the input right. Networks sometime go down, databases or other external servers may not respond as expected, and so forth -- sometimes it's worth catching such problems and retrying (maybe after a little wait -- maybe with an indication to the user about what's wrong, e.g. they may have accidentally unplugged a cable and you want to give them a chance to fix things and tell you when to try again), sometimes (especially in unattended long-running programs) there's nothing much you can do except an ordered shutdown (and detailed logging of every possibly-relevant aspect of the environment).
So, in brief, the answer to your Q's title is, "it depends";-). I hope I have been of use in listing many of the situations and aspects on which it can depend, and recommending what's generally the most useful attitude to take towards such issues.
To start with, you don't need any _mainLogger.
If you want to catch any exceptions, maybe to log or send them by email or whatever, do that at the highest possible level -- certainly not inside this class.
Also, you definitely don't want to convert every Exception to a RuntimeError. Let it emerge. The stopClient() method has no purpose right now. When it has, we'll look at it..
You could basically wrap the ConnectionError, IOError and OSError together (like, re-raise as something else), but not much more than that...