I have a Python application for which I have a logger.
At different steps of the execution, the application must read various input files. Input files which can have different information but are all read through the same function.
One of the particular information I am looking at is one called computation_id and MUST be present in one of the file but can be absent from all others.
And I am interested to know what is a correct way of handling this situation. Currently I am handling it like this :
def input_reading(filename):
results = {}
[...]
try:
results['computation_id'] = read_computation_id()
except KeyError:
pass
[...]
return results
so if the computation_id is absent from the file being read, the code shall keep running. However, at some point, I will need this computation id and therefore I need to check if it was correctly read from the file where I expect to find it.
I actually need this value far down the code. But running the code up to this point (which takes some time) to then fail is wasted computation time. So my idea is to check for this value as soon as I can and handle the error the following way :
def specifc_file_read(filename):
[...]
results = input_reading('my_file')
try:
results['computation_id']
except KeyError:
logger.exception('no computation id provided, aborting')
raise SystemExit('no computation id provided, aborting')
[...]
Is this good practice ?
I have a feeling it's not since I need to write special code lines to check for the error "as soon as I can" in the code to avoid wasting computation time.
Since I don't have much experience with error handling, I want to know if this is good practice or not in order not to keep bad habits.
It comes down to what you prefer, imo.
I think that would be more readable
if 'computation_id' not in results: raise ...
although if you want to check if it's available in any file before you begin some heavy data processing, you could to
for f in get_files():
if 'computation_id' in f:
break
else:
raise SystemExit
So it will raise SystemExit if it didn't break, so if it's not available in at least one file.
Related
so i have this mission to build a python cashier program using only python. without db use.
i use the dict and keys method to build it.
so i have 2 questions:
can i get an idea how to keep the program running after the moneybalance part, while i'm still saving the current moneybalance status? like, to make it an continuous program and not just one time.
i would like to get overall review on how is the idea for this mission, how it's written, and etc. would you do it in other way? if so i would like to hear how.
remember, my mission is to build python cashier without any outside libraries\db's.
Code:
print('welcome to python cashier')
money = int(input('enter cashier current money\n'))
prices = {"small popcorn" : 3, "medium popcorn" : 5,
"large popcorn" : 8, "small drink" : 1,
"medium drink" : 2, "large drink" : 3 }
def calc(money, prices):
try:
itemchange = pay - itemvalue
except Exception as e:
print(e)
finally:
print(f'change: {itemchange}')
def show(money, prices):
try:
for key,value in prices.items():
print (f'{key}: {value}')
except Exception as e:
print(e)
finally:
return False
show(money, prices)
def start(money, prices):
try:
for key, value in prices.items():
global item
item = str(input('enter what you want to buy:\n'))
if (item in prices.keys()):
global itemvalue
itemvalue = prices.get(item)
print(f'{item}: {itemvalue}')
global pay
pay = float(input('how much costumer pay?\n'))
return calc(money, prices)
else:
print('item not found')
except Exception as e:
print(e)
finally:
moneybalance = money + itemvalue
print(f'moneybalance: {moneybalance}')
start(money, prices)
The real question is if someone asked you to save the information. As you said they asked in the mission to do this without a database I don't think you need to do that. Btw if it's without an actual database you can use just .txt files to save information. Check about the "open" function in Python.
Quick Example:
with open('save.txt', 'w') as file:
file.write(str(cash)) # cash is an integer.
It's an example of writing a file. Btw I don't think they wanted you to save the information. Check it before you're applying something to don't ruin your work :).
I would suggest to put all the declarations of variables at the start of the script and add a "main" function. Call the main function from name == 'main'. Too I recommend you to do a more detailed function names. Try not to use global variables in this small code (you can declare it at the start, it'll be global and more easy to read your code and understand it faster). I think in your first question you thought about making your program run more than one time in any program run? Read about Python Loops if I'm correct :). Too you wrapped your input return with str() and it useless, because the returned value from input is already str (String type).
Quick Example:
# Variables Declaration (Just an example for the structure).
ABC = 0
cash = 1000
# Functions Declaration
def calculate_price(money, prices):
# Code Here.
pass
def main():
# The code of your start function, you don't need and arguments for this.
pass
if __name__ == '__main__':
main() # Call main here.
In this way your code will be readable and comfort to analyze :).
This sounds more like an assignment than an actual problem.
Here is the thing, and this is fundamental software architecture. You only need a database when it is time to persist something. So, what you need os a persistence service. Call it a database if you want, it can be as simple as a text file, as someone sugested.
Ironically the same set of rules that made your life easier as a developer, are making you confused. And questions arise. Specifically the question of what exactly happens when I close the program?
But the answer is dead simple. The requirements made it simple. If you close the application its gone. There is no persistence mechanism. This is actually super safe. No record, no proof. More importantly, its a requirement to be like so.
The problem, therefore, is you. You feel the need for persistence. You see an account and immediately place it in a database somewhere. You do a transaction, you feel the need to store it.
So what exactly happens when you persist something in a program? The program keeps all the information it needs to run. It may have gotten that information from a database, a user, or it may have randomly generated it. Who cares? All the program needs is the information in objects it knows how to use, in classes types and whatever more; in other words, in RAM. So persistence is taking things out of RAM into a less volatile storage.
But the real question is, how exactly does that affect the program? Does that cause any fundamental change?
For one the cashier operations are still the same. Prices must still be calculated, transactions must still be made, and all the rules associated must still be observed.
If something, there is less code involved. There is no solving the persistence problem. The information is volatile. Its a requirement to be so.
This is fundamentally the evolution process of a system under development, or even to some extent in production.
During development I care very little to store anything. What is actually usefull is to have reproucible state of the data. One I can interact with to make tests and verify that the software works (ie. it does not let me buy something if I have no money). This is easier done without any database system, specially the repoducible part.
Sure I can do backups of the data, but is it not easier to just restart the app? And where is the bug exactly if I need a specific database to see it? Is it in the code? Or is it a consistency issue in the database system? Probably the latter. How does that affect the cashier? It doesn't. Its a persistence detail. It has nothing to do with the business of a cashier.
I feel I have way more I could say, but I'm not sure where exactly you expect to get with this question. So I will stop here. Feel free to further inquire on this matter.
I have a script that is operating a physical device. There is a physical malfunction on the device that occurs sometimes and when it does, I want to reset the device and continue with the script. I have this:
while True:
do_device_control()
device_status = get_device_status()
if device_status == 'Fault':
reset_device()
It seems to me that a neater / more Pythonic approach would raise an Exception:
try:
while True:
do_device_control()
device_status = get_device_status()
if device_status == 'Fault':
raise DeviceFaultException()
except DeviceFaultException:
reset_device()
But as far as I can see, there is no way to resume the script after resetting the device. Is there a way to make Exception handling work for this situation, or a more Pythonic approach than what I'm currently using?
A common Python idiom is "Ask forgiveness rather than permission", which applies very well to your question. An implementation like the following would suffice:
while True:
try:
do_device_control()
except DeviceFaultException:
reset_device()
This should get similar behavior as to what is in your original block of code using if statements.
However, you probably noticed that I did not check the device_status in my code. This is because you should allow the do_device_control function to raise the exception if it is unable to complete its task. This allows you to handle exceptions at the time the actions are being executed.
In an environment where you are working with a device that is running asynchronously to your code, you may check the device status and it be fine. It then might fail between the check and your do_device_control function. This is one example of why the "ask forgiveness rather than permission" paradigm exists.
Of course, this example only works if you can expect the do_device_control function to throw some sort of exception or change it so that it does. If you cannot expect/change this behavior, your first code block with the if statement would be preferred over explicitly raising an exception just to immediately catch it within the same code block.
I've made a map editor in Python2.7.9 for a small project and I'm looking for ways to preserve the data I edit in the event of some unhandled exception. My editor already has a method for saving out data, and my current solution is to have the main loop wrapped in a try..finally block, similar to this example:
import os, datetime #..and others.
if __name__ == '__main__':
DataMgr = DataManager() # initializes the editor.
save_note = None
try:
MainLoop() # unsurprisingly, this calls the main loop.
except Exception as e: # I am of the impression this will catch every type of exception.
save_note = "Exception dump: %s : %s." % (type(e).__name__, e) # A memo appended to the comments in the save file.
finally:
exception_fp = DataMgr.cwd + "dump_%s.kmap" % str(datetime.datetime.now())
DataMgr.saveFile(exception_fp, memo = save_note) # saves out to a dump file using a familiar method with a note outlining what happened.
This seems like the best way to make sure that, no matter what happens, an attempt is made to preserve the editor's current state (to the extent that saveFile() is equipped to do so) in the event that it should crash. But I wonder if encapsulating my entire main loop in a try block is actually safe and efficient and good form. Is it? Are there risks or problems? Is there a better or more conventional way?
Wrapping the main loop in a try...finally block is the accepted pattern when you need something to happen no matter what. In some cases it's logging and continuing, in others it's saving everything possible and quitting.
So you're code is fine.
If your file isn't that big, I would suggest maybe reading the entire input file into memory, closing the file, then doing your data processing on the copy in memory, this will solve any problems you have with not corrupting your data at the cost of potentially slowing down your runtime.
Alternatively, take a look at the atexit python module. This allows you to register a function(s) for a automatic callback function when the program exits.
That being said what you have should work reasonably well.
I find I've been confused by the problem that when I needn't to use try..except.For last few days it was used in almost every function I defined which I think maybe a bad practice.For example:
class mongodb(object):
def getRecords(self,tname,conditions=''):
try:
col = eval("self.db.%s" %tname)
recs = col.find(condition)
return recs
except Exception,e:
#here make some error log with e.message
What I thought is ,exceptions may be raised everywhere and I have to use try to get them.
And my question is,is it a good practice to use it everywhere when defining functions?If not are there any principles for it?Help would be appreciated!
Regards
That may not be the best thing to do. Whole point of exceptions is that you can catch them on very different level than it's raised. It's best to handle them in the place where you have enough information to make something useful with them (that is very application and context dependent).
For example code below can throw IOError("[Errno 2] No such file or directory"):
def read_data(filename):
return open(filename).read()
In that function you don't have enough information to do something with it, but in place where you actually using this function, in case of such exception, you may decide to try different filename or display error to the user, or something else:
try:
data = read_data('data-file.txt')
except IOError:
data = read_data('another-data-file.txt')
# or
show_error_message("Data file was not found.")
# or something else
This (catching all possible exceptions very broadly) is indeed considered bad practice. You'll mask the real reason for the exception.
Catch only 'explicitely named' types of exceptions (which you expect to happen and you can/will handle gracefully). Let the rest (unexpected ones) bubble as they should.
You can log these (uncaught) exceptions (globally) by overriding sys.excepthook:
import sys
import traceback
# ...
def my_uncaught_exception_hook(exc_type, exc_value, exc_traceback):
msg_exc = "".join( \
traceback.format_exception(exc_type, exc_value, exc_traceback) )
# ... log here...
sys.excepthook = my_uncaught_exception_hook # our uncaught exception hook
You must find a balance between several goals:
An application should recover from as many errors as possible by itself.
An application should report all unrecoverable errors with enough detail to fix the cause of the problem.
Errors can happen everywhere but you don't want to pollute your code with all the error handling code.
Applications shouldn't crash
To solve #3, you can use an exception hook. All unhandled exceptions will cause the current transaction to abort. Catch them at the highest level, roll back the transaction (so the database doesn't become inconsistent) and either throw them again or swallow them (so the app doesn't crash). You should use decorators for this. This solves #4 and #1.
The solution for #2 is experience. You will learn with time what information you need to solve problems. The hard part is to still have the information when an error happens. One solution is to add debug logging calls in the low level methods.
Another solution is a dictionary per thread in which you can store some bits and which you dump when an error happens.
another option is to wrap a large section of code in a try: except: (for instance in a web application, one specific GUI page) and then use sys.exc_info() to print out the error and also the stack where it occurred
import sys
import traceback
try:
#some buggy code
x = ??
except:
print sys.exc_info()[0] #prints the exception class
print sys.exc_info()[1] #prints the error message
print repr(traceback.format_tb(sys.exc_info()[2])) #prints the stack
I can't understand what sort of exceptions I should handle 'here and now', and what sort of exceptions I should re-raise or just don't handle here, and what to do with them later (on higher tier). For example: I wrote client/server application using python3 with ssl communication. Client is supposed to verify files on any differences on them, and if diff exists then it should send this 'updated' file to server.
class BasicConnection:
#blablabla
def sendMessage(self, sock, url, port, fileToSend, buffSize):
try:
sock.connect((url, port))
while True:
data = fileToSend.read(buffSize)
if not data: break
sock.send(data)
return True
except socket.timeout as toErr:
raise ConnectionError("TimeOutError trying to send File to remote socket: %s:%d"
% (url,port)) from toErr
except socket.error as sErr:
raise ConnectionError("Error trying to send File to remote socket: %s:%d"
% (url,port)) from sErr
except ssl.SSLError as sslErr:
raise ConnectionError("SSLError trying to send File to remote socket: %s:%d"
% (url,port)) from sslErr
finally:
sock.close()
Is it right way to use exceptions in python? The problem is: what if file.read() throws IOError? Should I handle it here, or just do nothing and catch it later? And many other possible exceptions?
Client use this class (BasicConnection) to send updated files to server:
class PClient():
def __init__(self, DATA):
'''DATA = { 'sendTo' : {'host':'','port':''},
'use_ssl' : {'use_ssl':'', 'fileKey':'', 'fileCert':'', 'fileCaCert':''},
'dirToCheck' : '',
'localStorage': '',
'timeToCheck' : '',
'buffSize' : '',
'logFile' : ''} '''
self._DATA = DATA
self._running = False
self.configureLogging()
def configureLogging(self):
#blablabla
def isRun(self):
return self._running
def initPClient(self):
try:
#blablabla
return True
except ConnectionError as conErr:
self._mainLogger.exception(conErr)
return False
except FileCheckingError as fcErr:
self._mainLogger.exception(fcErr)
return False
except IOError as ioErr:
self._mainLogger.exception(ioErr)
return False
except OSError as osErr:
self._mainLogger.exception(osErr)
return False
def startPClient(self):
try:
self._running = True
while self.isRun():
try :
self._mainLogger.debug("Checking differences")
diffFiles = FileChecker().checkDictionary(self._dict)
if len(diffFiles) != 0:
for fileName in diffFiles:
try:
self._mainLogger.info("Sending updated file: %s to remote socket: %s:%d"
% (fileName,self._DATA['sendTo']['host'],self._DATA['sendTo']['port']))
fileToSend = io.open(fileName, "rb")
result = False
result = BasicConnection().sendMessage(self._sock, self._DATA['sendTo']['host'],
self._DATA['sendTo']['port'], fileToSend, self._DATA['buffSize'])
if result:
self._mainLogger.info("Updated file: %s was successfully delivered to remote socket: %s:%d"
% (fileName,self._DATA['sendTo']['host'],self._DATA['sendTo']['port']))
except ConnectionError as conErr:
self._mainLogger.exception(conErr)
except IOError as ioErr:
self._mainLogger.exception(ioErr)
except OSError as osErr:
self._mainLogger.exception(osErr)
self._mainLogger.debug("Updating localStorage %s from %s " %(self._DATA['localStorage'], self._DATA['dirToCheck']))
FileChecker().updateLocalStorage(self._DATA['dirToCheck'],
self._DATA['localStorage'])
self._mainLogger.info("Directory %s were checked" %(self._DATA['dirToCheck']))
time.sleep(self._DATA['timeToCheck'])
except FileCheckingError as fcErr:
self._mainLogger.exception(fcErr)
except IOError as ioErr:
self._mainLogger.exception(ioErr)
except OSError as osErr:
self._mainLogger.exception(osErr)
except KeyboardInterrupt:
self._mainLogger.info("Shutting down...")
self.stopPClient()
except Exception as exc:
self._mainLogger.exception(exc)
self.stopPClient()
raise RuntimeError("Something goes wrong...") from exc
def stopPClient(self):
self._running = False
Is it correct? May be someone spend his own time and just help me to understand pythonic style of handling exceptions? I can't understand what to do with such exceptions as NameError, TypeError, KeyError, ValueError...and so on.......They could be thrown at any statement, at any time... and what to do with them, if I want to logged everything.
And what information should people usually log? If error occurs, what info about it I should log? All traceback, or just relevant message about it or something else?
I hope somebody helps me.
Thanks a lot.
In general, you should "catch" the exceptions that you expect to happen (because they may be caused by user error, or other environmental problems outside of your program's control), especially if you know what your code might be able to do about them. Just giving more details in an error report is a marginal issue, though some programs' specs may require doing that (e.g. a long-running server that's not supposed to crash due to such problems, but rather log a lot of state information, give the user a summary explanation, and just keep working for future queries).
NameError, TypeError, KeyError, ValueError, SyntaxError, AttributeError, and so on, can be thought of as due to errors in the program -- bugs, not problems outside of the programmer's control. If you're releasing a library or framework, so that your code is going to be called by other code outside of your control, then such bugs may quite likely be in that other code; you should normally let the exception propagate to help the other programmer debug their own bugs. If you're releasing an application, you own the bugs, and you must pick the strategy that helps you find them.
If your bugs show up while an end-user is running the program, you should log a lot of state information, and give the user a summary explanation and apologies (perhaps with a request to send you the log info, if you can't automate that -- or, at least, ask permission before you send anything from the user's machine to yours). You may be able to save some of the user's work so far, but often (in a program that's known to be buggy) that may not work anyway.
Most bugs should show up during your own testing of course; in that case, propagating the exception is useful as you can hook it up to a debugger and explore the bug's details.
Sometimes some exceptions like these show up just because "it's easier to ask forgiveness than permission" (EAFP) -- a perfectly acceptable programming technique in Python. In that case of course you should handle them at once. For example:
try:
return mylist[theindex]
except IndexError:
return None
here you might expect that theindex is generally a valid index into mylist, but occasionally outside of mylist's bounds -- and the latter case, by the semantics of the hypothetic app in which this snippet belongs, is not an error, just a little anomaly to be fixed by considering the list to be conceptually extended on both sides with infinite numbers of Nones. It's easier to just try/except than to properly check for positive and negative values of the index (and faster, if being out of bounds is a truly rare occurrence).
Similarly appropriate cases for KeyError and AttributeError happen less frequently, thanks to the getattr builtin and get method of dicts (which let you provide a default value), collections.defaultdict, etc; but lists have no direct equivalent of those, so the try/except is seen more frequently for IndexError.
Trying to catch syntax errors, type errors, value errors, name errors, etc, is a bit rarer and more controversial -- though it would surely be appropriate if the error was diagnosed in a "plug-in", third-party code outside your control which your framework/application is trying to load and execute dynamically (indeed that's the case where you're supplying a library or the like and need to coexist peacefully with code out of your control which might well be buggy). Type and value errors may sometimes occur within an EAFP pattern -- e.g. when you try to overload a function to accept either a string or a number and behave slightly differently in each case, catching such errors may be better than trying to check types -- but the very concept of functions thus overloaded is more often than not quite dubious.
Back to "user and environmental errors", users will inevitably make mistakes when they give you input, indicate a filename that's not actually around (or that you don't have permission to read, or to write if that's what you're supposed to be doing), and so on: all such errors should of course be caught and result in a clear explanation to the user about what's gone wrong, and another chance to get the input right. Networks sometime go down, databases or other external servers may not respond as expected, and so forth -- sometimes it's worth catching such problems and retrying (maybe after a little wait -- maybe with an indication to the user about what's wrong, e.g. they may have accidentally unplugged a cable and you want to give them a chance to fix things and tell you when to try again), sometimes (especially in unattended long-running programs) there's nothing much you can do except an ordered shutdown (and detailed logging of every possibly-relevant aspect of the environment).
So, in brief, the answer to your Q's title is, "it depends";-). I hope I have been of use in listing many of the situations and aspects on which it can depend, and recommending what's generally the most useful attitude to take towards such issues.
To start with, you don't need any _mainLogger.
If you want to catch any exceptions, maybe to log or send them by email or whatever, do that at the highest possible level -- certainly not inside this class.
Also, you definitely don't want to convert every Exception to a RuntimeError. Let it emerge. The stopClient() method has no purpose right now. When it has, we'll look at it..
You could basically wrap the ConnectionError, IOError and OSError together (like, re-raise as something else), but not much more than that...