I'm currently doing some Python automation of Excel with com. It's fully functional, and does what I want, but I've discovered something surprising. Sometimes, some of the Excel commands I use will fail with an exception for no apparent reason. Other times, they will work.
In the VB equivalent code for what I'm doing, this problem is apparently considered normal, and is plastered over with a On Error Resume Next statement. Python does not have said statement, of course.
I can't wrap up the whole set in a try except loop, because it could "fail" halfway through and not complete properly. So, what would be a pythonic way to wrap several independent statements into a try except block? Specifically, something cleaner than:
try:
statement
except:
pass
try:
statement
except:
pass
The relevant code is the excel.Selection.Borders bit.
def addGridlines(self, infile, outfile):
"""convert csv to excel, and add gridlines"""
# set constants for excel
xlDiagonalDown = 5
xlDiagonalUp = 6
xlNone = -4142
xlContinuous = 1
xlThin = 2
xlAutomatic = -4105
xlEdgeLeft = 7
xlEdgeTop = 8
xlEdgeBottom = 9
xlEdgeRight = 10
xlInsideVertical = 11
xlInsideHorizontal = 12
# open file
excel = win32com.client.Dispatch('Excel.Application')
workbook = excel.Workbooks.Open(infile)
worksheet = workbook.Worksheets(1)
# select all cells
worksheet.Range("A1").CurrentRegion.Select()
# add gridlines, sometimes some of these fail, so we have to wrap each in a try catch block
excel.Selection.Borders(xlDiagonalDown).LineStyle = xlNone
excel.Selection.Borders(xlDiagonalUp).LineStyle = xlNone
excel.Selection.Borders(xlDiagonalUp).LineStyle = xlNone
excel.Selection.Borders(xlEdgeLeft).LineStyle = xlContinuous
excel.Selection.Borders(xlEdgeLeft).Weight = xlThin
excel.Selection.Borders(xlEdgeLeft).ColorIndex = xlAutomatic
excel.Selection.Borders(xlEdgeTop).LineStyle = xlContinuous
excel.Selection.Borders(xlEdgeTop).Weight = xlThin
excel.Selection.Borders(xlEdgeTop).ColorIndex = xlAutomatic
excel.Selection.Borders(xlEdgeBottom).LineStyle = xlContinuous
excel.Selection.Borders(xlEdgeBottom).Weight = xlThin
excel.Selection.Borders(xlEdgeBottom).ColorIndex = xlAutomatic
excel.Selection.Borders(xlEdgeRight).LineStyle = xlContinuous
excel.Selection.Borders(xlEdgeRight).Weight = xlThin
excel.Selection.Borders(xlEdgeRight).ColorIndex = xlAutomatic
excel.Selection.Borders(xlInsideVertical).LineStyle = xlContinuous
excel.Selection.Borders(xlInsideVertical).Weight = xlThin
excel.Selection.Borders(xlInsideVertical).ColorIndex = xlAutomatic
excel.Selection.Borders(xlInsideHorizontal).LineStyle = xlContinuous
excel.Selection.Borders(xlInsideHorizontal).Weight = xlThin
excel.Selection.Borders(xlInsideHorizontal).ColorIndex = xlAutomatic
# refit data into columns
excel.Cells.Select()
excel.Cells.EntireColumn.AutoFit()
# save new file in excel format
workbook.SaveAs(outfile, FileFormat=1)
workbook.Close(False)
excel.Quit()
del excel
Update:
Perhaps a bit of explanation on the error bit is required. Two identical runs on my test machine, with identical code, on the same file, produce the same result. One run throws exceptions for every xlInsideVertical line. The other throws exceptions for every xlInsideHorizontal. Finally, a third run completes with no exceptions at all.
As far as I can tell Excel considers this normal behavior, because I'm cloning the VB code built by excel's macro generator, not VB code produced by a person. This might be an erroneous assumption, of course.
It will function with each line wrapped in a try except block I just wanted something shorter and more obvious, because 20 lines wrapped in their own try catch loops is just asking for trouble later.
Update2:
This is a scrubbed CSV file for testing: gist file
Conclusion:
The answer provided by Vsekhar is perfect. It abstracts away the exception suppression, so that later, if and when I have time, I can actually deal with the exceptions as they occur. It also allows for logging the exceptions so they don't disappear, not stopping other exceptions, and is small enough to be easily manageable six months from now.
Consider abstracting away the suppression. And to Aaron's point, do not swallow exceptions generally.
class Suppressor:
def __init__(self, exception_type):
self._exception_type = exception_type
def __call__(self, expression):
try:
exec expression
except self._exception_type as e:
print 'Suppressor: suppressed exception %s with content \'%s\'' % (type(self._exception_type), e)
# or log.msg('...')
Then, note in the traceback of your current code exactly what exception is raised, and create a Suppressor for just that exception:
s = Suppressor(excel.WhateverError) # TODO: put your exception type here
s('excel.Selection.Borders(xlDiagonalDown).LineStyle = xlNone')
This way you get line-by-line execution (so your tracebacks will still be helpful), and you are suppressing only the exceptions you explicitly intended. Other exceptions propagate as usual.
Exceptions never happen "for no apparent reason". There is always a reason and that reason needs to be fixed. Otherwise, your program will start to produce "random" data where "random" is at the mercy of the bug that you're hiding.
But of course, you need a solution for your problem. Here is my suggestion:
Create a wrapper class that implements all the methods that you need and delegates them to the real Excel instance.
Add a decorator before each method which wraps the method in a try except block and log the exception. Never swallow exceptions
Now the code works for your customer which buys you some time to find out the cause of the problem. My guess is that a) Excel doesn't produce a useful error message or b) the wrapper code swallows the real exception leaving you in the dark or c) the Excel method returns an error code (like "false" for "failed") and you need to call another Excel method to determine what the cause of the problem is.
[EDIT] Based on the comment below which boil down to "My boss doesn't care and there is nothing I can do": You're missing a crucial point: It's your bosses duty to make the decision but it your duty to give her a list of options along with pros/cons so that she can make a sound decision. Just sitting there saying "I can't do anything" will get you into the trouble that you're trying to avoid.
Example:
Solution 1: Ignore the errors
Pro: Least amount of work
Con: There is a chance that the resulting data is wrong or random. If important business decisions are based on it, there is a high risk that those decisions will be wrong.
Solution 2: Log the errors
Pro: Little amount of work, users can start to use the results quickly, buys time to figure out the source of the problem
Con: "If you can't fix it today, what makes you think you will have time to fix it tomorrow?" Also, it might take you a long time to find the source of the problem because you're no expert
Solution 3: Ask an expert
Find an expert in the field and help him/her have a look/improve the solution.
Pro: Will get a solution much more quickly than learning the ins and outs of COM yourself
Con: Expensive but high chance of success. Will also find problems that we don't even know about.
...
I think you see the pattern. Bosses make wrong decisions because we (willingly) let them. Any boss in the world is happy for hard facts and input when they have to make a decision (well, those who don't shouldn't be bosses, so this is a surefire way to know when to start looking for a new job).
If you select solution #2, go for the wrapper approach. See the docs how to write a decorator (example from IBM). It's just a few minutes of work to wrap all the methods and it will give you something to work with.
The next step is to create a smaller example which sometimes fails and then post specific questions about Python, Excel and the COM wrapper here to figure out the reason for the problems.
[EDIT2] Here is some code that wraps the "dangerous" parts in a helper class and makes updating the styles more simple:
class BorderHelper(object):
def __init__(self, excel):
self.excel = excel
def set( type, LineStyle = None, Weight = None, Color = None ):
border = self.excel.Selection.Borders( type )
try:
if LineStyle is not None:
border.LineStyle = LineStyle
except:
pass # Ignore if a style can't be set
try:
if Weight is not None:
border.Weight = Weight
except:
pass # Ignore if a style can't be set
try:
if Color is not None:
border.Color = Color
except:
pass # Ignore if a style can't be set
Usage:
borders = BorderHelper( excel )
borders.set( xlDiagonalDown, LineStyle = xlNone )
borders.set( xlDiagonalUp, LineStyle = xlNone )
borders.set( xlEdgeLeft, LineStyle = xlContinuous, Weight = xlThin, Color = xlAutomatic )
...
This just wraps functions calls, but you can extend it to handle attribute access as well, and to proxy the results of nested attribute accesses, finally just wrapping the __setattr__ in your try:except block.
It might be sensible to swallow only some specific exception types in your case (as #vsekhar says).
def onErrorResumeNext(wrapped):
class Proxy(object):
def __init__(self, fn):
self.__fn = fn
def __call__(self, *args, **kwargs):
try:
return self.__fn(*args, **kwargs)
except:
print "swallowed exception"
class VBWrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def __getattr__(self, name):
return Proxy(eval('self.wrapped.'+name))
return VBWrapper(wrapped)
Example:
exceptionProofBorders = onErrorResumeNext(excel.Selection.Borders)
exceptionProofBorders(xlDiagonalDown).LineStyle = xlNone
exceptionProofBorders(xlDiagonalup).LineStyle = xlNone
You can zip arguments from three list, and do the following:
for border, attr, value in myArgs:
while True:
i = 0
try:
setattr(excel.Selection.Borders(border), attr, value)
except:
if i>100:
break
else:
break
If your exceptions are trully random, this will try until success (with a limit of 100 tries). I don't recommend this.
Related
I need to detect when the minutes of the clock/time change and do something,
This is mi code so far, for the clock but still can figuruate out in python how to detect the value has change and do action after. Any help will be apreciated i come from a c++ backgorund my implementations seems so far not working.
while True:
now = datetime.now()
print(now.strftime("%M), end = " ", flush = true)
time.sleep(1)
currentMin = now.srtftime("%M")
that worked for me:
from datetime import datetime
import time
past_min = None
while True:
#current min
now_min = int(datetime.now().strftime("%M"))
#first iteration
if not past_min:
past_min = now_min
if now_min != past_min:
#call your function here
print("Min change detected")
past_min = now_min
#print the seconds
print(datetime.now().strftime("%S"))
time.sleep(1.5)
I think you can create a class (in the below example Minute) with a property currenMin to store the current minute value. By using #<property>.setter function, when the property <property> is changed, it will trigger the setter function
from datetime import datetime
import time
class Minute(object):
def __init__(self):
self._currenMin = ''
#property
def currentMin(self):
return self._currenMin
#currentMin.setter
def currentMin(self, value):
if value != self._currenMin:
# ACTION CODE BELOW
print('Minute changed')
self._currenMin = value
minute = Minute()
while True:
now = datetime.now()
print(now.strftime("%M"), end=" ", flush = True)
time.sleep(1)
minute.currentMin = now.strftime("%M")
Well, for the general case with simple variables, you can't simply do it. There are two simple options to do something similar:
if you control EVERYTHING that writes it, make them trigger that action
write code that regularly checks it and triggers the action when it changes
use language tools like a custom setter (see #user696969's answer)
The first case needs you to control everything that could modify that value. At that point, you might not even need a variable, and just pass the new value (and you can reverse this by having a variable that is always updated). This is a very common pattern, called Event-driven programming, and heavily used for example in UIs, websites (client-side, see a list of DOM events for example) and game frameworks (see pygame's documentation on events)
The second-case of writing a loop or checking it regularly can also work, however, there are some downsides to it as well. You probably don't want to write an infinite loop waiting for it to change, especially not in a way that also blocks the changing of that variable, and thus dead-locking the entire program as it's preventing something it's waiting for. If you just check it regularly between other, it might be hard to ensure it will be checked regardless of what else is the program doing. You might use multiple threads for it, but that brings it's own set of problems. You also have to store and update the previous value, so you can compare it. This might be slow or memory-consuming if the variable holds too much data.
You can also use language tools with custom setters. This is clean, but can not be used for any variable, just for class attributes, so you still need some control over the rest of the program.
Generally I'd use the event-driven approach or the setter one, depending on the wider context. However, for such simple cases, the checking is also fine. The simplest solution might event be to remove the need for this entirely.
My code is working fine, but PyCharm warns about the except clause in the following being too broad. And honestly, it also smells to me as the wrong implementation.
When scraping HTML<tr>s in bulk, which I also have to add to the database or update, I don't know in advance if a game is finished or not. Also, a game that hasn't been finished has different HTML tags in some <td>s, that require different handling.
Basically, if we look at match_score_string and match_relevant_bit we are telling BeautifulSoup to find a <td> with a certain class. If the game is finished already, this <td> will have a score-time sc class, if not, it will have a score-time st class. If it has one, it won't have the other.
It's been a while since I wrote the try-except clauses, but if I recall correctly, the reason (I felt) I had to use them is because BS would throw an error and halt all operations when row.find fails to locate the HTML object.
Is PyCharm complaining justifiably?
# Get Match Score
try:
match_score_string = row.find("td", class_="score-time sc").get_text()
match_string_split = match_score_string.split(" - ")
team_a_score = int(match_string_split[0])
team_b_score = int(match_string_split[1])
print(team_a_score)
print(team_b_score)
except:
team_a_score = None
team_b_score = None
# Get Match URL
try:
match_relevant_bit = row.find("td", class_="score-time sc")
match_url = match_relevant_bit.find("a").get("href")
match_url_done = match_url.rsplit('?JKLMN')
match_url = match_url_done[0]
match_finished = True
except:
match_relevant_bit = row.find("td", class_="score-time st")
match_url = match_relevant_bit.find("a").get("href")
match_url_done = match_url.rsplit('?JKLMN')
match_url = match_url_done[0]
match_finished = False
Your except is certainly too broad. I won't dispute whether or not you need exception handling in these cases -- Exceptions are quite pythonic so use them wherever you feel they are appropriate. However, you haven't argued that you need your code to handle every exception imaginable. A bare except can mask all sorts of bugs. e.g. If you misspell a variable name at a later revision:
try:
match_score_string = row.find("td", class_="score-time sc").get_text()
match_string_split = match_score_string.split(" - ")
team_a_score = int(match_string_spilt[0]) # oops
team_b_score = int(match_string_split[1])
print(team_a_score)
print(team_b_score)
except:
team_a_score = None
team_b_score = None
Bare excepts also prevent a user from sending KeyboardInterrupt to terminate the program ...
These are just 2 examples why you don't want a bare except -- Since the number of programming errors that could get masked by this are near infinite, I don't have enough time to demonstrate all of them ;-).
Instead, you should specify exactly what exceptions you expect to encounter:
try:
match_score_string = row.find("td", class_="score-time sc").get_text()
match_string_split = match_score_string.split(" - ")
team_a_score = int(match_string_spilt[0]) # oops
team_b_score = int(match_string_split[1])
print(team_a_score)
print(team_b_score)
except AttributeError: # If no row is found, accessing get_text fails.
team_a_score = None
team_b_score = None
This way, the code executes as you expect and you don't mask (too many) programming bugs. The next thing you want to do to prevent hiding bugs is to limit amount of code in your exception handler as much as possible.
Try/except clauses are great. It sounds like you don't use them too often.
try/except blocks can help you avoid complicated if statements, or code that checks for a condition before proceeding which can sometimes make it unclear what you are trying to do (see what I did there :P).
There are a few exceptions that you could possibly drag out of your code, and some of them might be more likely than the others. See below:
try:
# This first line could have an AttributeError in two places
# Maybe the row didn't parse quite correctly? find() won't work then.
# Maybe you didn't find any elements? get_text() won't work then.
match_score_string = row.find("td", class_="score-time sc").get_text()
^ ^
# Same as here, you could get an AttributeError if match_score_string isn't a string.
# I would say that isn't very likely in this case,
# Especially if you handle the above line safely.
match_string_split = match_score_string.split(" - ")
# If either of these split strings contains no matches, you'll get an IndexError
# If either of the strings at the index contain letters, you may get a ValueError
team_a_score = int(match_string_split[0])
team_b_score = int(match_string_split[1])
# Theoretically, if you caught the exception from the above two lines
# you might get a NameError here, if the variables above never ended up being defined.
print(team_a_score)
print(team_b_score)
except:
# Can't really imagine much going wrong here though.
team_a_score = None
team_b_score = None
As you might imagine, it may be more useful to catch several types of exceptions. It depends on the complexity of your inputs.
Multiple try/except blocks would also be of benefit here, specifically to handle the case where you catch an error and need to continue executing the rest of the code inside that block.
I've made a map editor in Python2.7.9 for a small project and I'm looking for ways to preserve the data I edit in the event of some unhandled exception. My editor already has a method for saving out data, and my current solution is to have the main loop wrapped in a try..finally block, similar to this example:
import os, datetime #..and others.
if __name__ == '__main__':
DataMgr = DataManager() # initializes the editor.
save_note = None
try:
MainLoop() # unsurprisingly, this calls the main loop.
except Exception as e: # I am of the impression this will catch every type of exception.
save_note = "Exception dump: %s : %s." % (type(e).__name__, e) # A memo appended to the comments in the save file.
finally:
exception_fp = DataMgr.cwd + "dump_%s.kmap" % str(datetime.datetime.now())
DataMgr.saveFile(exception_fp, memo = save_note) # saves out to a dump file using a familiar method with a note outlining what happened.
This seems like the best way to make sure that, no matter what happens, an attempt is made to preserve the editor's current state (to the extent that saveFile() is equipped to do so) in the event that it should crash. But I wonder if encapsulating my entire main loop in a try block is actually safe and efficient and good form. Is it? Are there risks or problems? Is there a better or more conventional way?
Wrapping the main loop in a try...finally block is the accepted pattern when you need something to happen no matter what. In some cases it's logging and continuing, in others it's saving everything possible and quitting.
So you're code is fine.
If your file isn't that big, I would suggest maybe reading the entire input file into memory, closing the file, then doing your data processing on the copy in memory, this will solve any problems you have with not corrupting your data at the cost of potentially slowing down your runtime.
Alternatively, take a look at the atexit python module. This allows you to register a function(s) for a automatic callback function when the program exits.
That being said what you have should work reasonably well.
I want to automatically indent the next line of a console app but the user needs to be able to remove it. sys.stdout.write and print make undeletable characters and I can't write to sys.stdin (as far as I know). I'm essentially trying to get smart indenting but I can only nest deeper and deeper. Any ideas on how to climb back out?
Edit: I should have noted that this is part of a Windows program that uses IronPython. While I could do something much fancier (and might in the future), I am hoping to quickly get a reasonably pleasant experience with very little effort as a starting point.
The cmd module provides a very simple interface for creating a command line interface to your program. It might not be able to put some buffer characters in front of the next line but if you're looking for an obvious way to let your users know that the command has returned, it can provide a shell-like prompt at the beginning of each line. If you already have functions defined for your program, integrating them into a processor would be a matter of writing a handler that access the function:
import cmd
import math
def findHpyot(length, height):
return math.sqrt(length **2 + height **2)
class MathProcessor(cmd.Cmd):
prompt = "Math>"
def do_hypot(self, line):
x = raw_input("Length:")
y = raw_input("Height:")
if x and y:
try:
hypot = findHypot(float(x), float(y))
print "Hypot:: %.2f" %hypot
except ValueError:
print "Length and Height must be numbers"
def do_EOF(self, line):
return True
def do_exit(self, line):
return True
def do_quit(self, line):
return True
if __name__ == "__main__":
cmdProcessor = MathProcessor()
cmdProcessor.cmdloop()
Things to consider when writing an interactive shell using cmd
The name after do_ is the command that your users will use so that in this example, the available commands will be hypot, exit, quit, and help.
Without overriding do_help, calling help will give you a list of available commands
Any call that you want to quit the program should return True
If you want to process entries from the function call, say you wanted to be able to handle a call like "hypot 3 4" you can use the local line variable in the function call
I find I've been confused by the problem that when I needn't to use try..except.For last few days it was used in almost every function I defined which I think maybe a bad practice.For example:
class mongodb(object):
def getRecords(self,tname,conditions=''):
try:
col = eval("self.db.%s" %tname)
recs = col.find(condition)
return recs
except Exception,e:
#here make some error log with e.message
What I thought is ,exceptions may be raised everywhere and I have to use try to get them.
And my question is,is it a good practice to use it everywhere when defining functions?If not are there any principles for it?Help would be appreciated!
Regards
That may not be the best thing to do. Whole point of exceptions is that you can catch them on very different level than it's raised. It's best to handle them in the place where you have enough information to make something useful with them (that is very application and context dependent).
For example code below can throw IOError("[Errno 2] No such file or directory"):
def read_data(filename):
return open(filename).read()
In that function you don't have enough information to do something with it, but in place where you actually using this function, in case of such exception, you may decide to try different filename or display error to the user, or something else:
try:
data = read_data('data-file.txt')
except IOError:
data = read_data('another-data-file.txt')
# or
show_error_message("Data file was not found.")
# or something else
This (catching all possible exceptions very broadly) is indeed considered bad practice. You'll mask the real reason for the exception.
Catch only 'explicitely named' types of exceptions (which you expect to happen and you can/will handle gracefully). Let the rest (unexpected ones) bubble as they should.
You can log these (uncaught) exceptions (globally) by overriding sys.excepthook:
import sys
import traceback
# ...
def my_uncaught_exception_hook(exc_type, exc_value, exc_traceback):
msg_exc = "".join( \
traceback.format_exception(exc_type, exc_value, exc_traceback) )
# ... log here...
sys.excepthook = my_uncaught_exception_hook # our uncaught exception hook
You must find a balance between several goals:
An application should recover from as many errors as possible by itself.
An application should report all unrecoverable errors with enough detail to fix the cause of the problem.
Errors can happen everywhere but you don't want to pollute your code with all the error handling code.
Applications shouldn't crash
To solve #3, you can use an exception hook. All unhandled exceptions will cause the current transaction to abort. Catch them at the highest level, roll back the transaction (so the database doesn't become inconsistent) and either throw them again or swallow them (so the app doesn't crash). You should use decorators for this. This solves #4 and #1.
The solution for #2 is experience. You will learn with time what information you need to solve problems. The hard part is to still have the information when an error happens. One solution is to add debug logging calls in the low level methods.
Another solution is a dictionary per thread in which you can store some bits and which you dump when an error happens.
another option is to wrap a large section of code in a try: except: (for instance in a web application, one specific GUI page) and then use sys.exc_info() to print out the error and also the stack where it occurred
import sys
import traceback
try:
#some buggy code
x = ??
except:
print sys.exc_info()[0] #prints the exception class
print sys.exc_info()[1] #prints the error message
print repr(traceback.format_tb(sys.exc_info()[2])) #prints the stack