I am trying to develop something with xlwings because I need to manipulate a xls file with macros etc. Although it is always good to close connections, Excel is notorious in that it blocks access if more than one instance is running. Therefore I need to make sure that the app closes even though my code fails somewhere upstream.
I am currently doing this with a try statement that spans the whole script and when it fails calls app.quit(). But this suppresses my error messages, which makes debugging hard. So I feel there must be something better.
In another context I have seen with being used. And I have the feeling it would apply here too, but I do not understand how it works, nor how it would work in this specific case.
import xlwings as xw
def myexcel():
try:
#connect to Excel app in the background
excel = xw.App(visible=False)
# open excel book
wb = excel.books.open(str(file))
# asign the active app so it can be closed later
app = xw.apps.active
# more code goes here
except:
app.quit()
How could one make sure that the excel connection gets always closed no-matter the most efficient way?
If with is the solution, I would also appreciate a pointer to a good source to learn more about that concept.
As you mentioned, you can use a with statement and build your own contextmanager. Here's a converted example based on your code:
import xlwings as xw
class MyExcelApp:
def __init__(self):
self.excel = xw.App(visible=False)
def __enter__(self):
return self.excel
def __exit__(self, exc, value, traceback):
# Handle your app-specific exceptions (exc) here
self.excel.quit()
return True
# ^ return True only if you intend to catch all errors in here.
# Otherwise, leave as is and use try... except on the outside.
class MyExcelWorkbook:
def __init__(self, xlapp, bookname):
self.workbook = xlapp.books.open(bookname)
def __enter__(self):
return self.workbook
def __exit__(self, exc, value, traceback):
# Handle your workbook specific exceptions (exc) here
# self.workbook.save() # depends what you want to do here
self.workbook.close()
return True
# ^ return True only if you intend to catch all errors in here.
# Otherwise, leave as is and use try... except on the outside.
With this set up you can simply call it like this:
with MyExcelApp() as app:
with MyExcelWorkbook(filename) as wb:
# do something with wb
You can also implement it with a generator, which will be quite similar to the other answer.
Here's a simplified version:
import xlwings as xw
from contextlib import contextmanager
#contextmanager
def my_excel_app():
app = xw.App(visible=False)
try:
yield app
except: # <-- Add SPECIFIC app exceptions
# Handle the errors
finally:
app.quit()
Usage:
with my_excel() as app:
wb = app.books.open(some_file)
# do something...
you do it right - using try block in this case is the way to go. With statement is good when you need to open file, but not for your use case when you use library which is opening excel file using its own way.
To show details of exception you can change your code as follows:
import xlwings as xw
def myexcel():
try:
#connect to Excel app in the background
excel = xw.App(visible=False)
# open excel book
wb = excel.books.open(str(file))
# asign the active app so it can be closed later
app = xw.apps.active
# more code goes here
finally:
app.quit()
except Exception as e:
print('exception catched: {}'.format(e))
app.quit()
Preferred solution
xlwings added a solution in v0.24.3 to this problem:
xlwings.App() can now be used as context manager, making sure that there are no zombie processes left over on Windows, even if you use a hidden instance and your code fails. It is therefore recommended to use it whenever you can, like so:
import xlwings as xw
with xw.App(visible=False) as app:
wb = xw.Book("test.xlsx")
sheet = wb.sheets['sheet1']
# To evoke an error, I try to call an non-exisiting sheet here.
nonexistent_sheet["A1"]
Solution before v24.0.3
You can use the library traceback, which makes debugging easier, because the error is displayed in red color. See this example:
import xlwings as xw
import traceback
filename = "test.xlsx"
try:
# Do what you want here in the try block. For example, the following lines.
app = xw.App(visible=False)
wb = xw.Book(filename)
sheet = wb.sheets['sheet1']
# To evoke an error, I try to call an nonexistent sheet here.
nonexistent_sheet["A1"]
# Use BaseException because it catches all possible exceptions: https://stackoverflow.com/a/31609619/13968392
except BaseException:
# This prints the actual error in a verbose way.
print(traceback.print_exc())
app.quit()
The error displays with print(traceback.print_exc()) as follows:
Related
When running xlwings 0.26.1 (latest for Anaconda 3.83) or 0.10.0 (using for compatibility reasons) with the latest version of Office 365 Excel, I get an error after moving a sheet when running app.quit():
import xlwings as xw
import pythoncom
pythoncom.CoInitialize()
app = xw.apps.add()
app.display_alerts = False
app.screen_updating = False
wbSource = app.books.open('pathSourceTemp')
wsSource = wbSource.sheets['sourceSheet']
wbDestination = app.books.open('pathDestinationTemp')
wsDestination = None
#Grabs first sheet in destination
wsDestination = wbDestination.sheets[0]
#Copy sheet "before" destination sheet (which should be 1 sheet after the destination sheet)
wsSource.api.Copy(Before=wsDestination.api)
wbDestination.save()
#Close workbooks and app
wbDestination.close()
wbSource.close()
app.screen_updating = True
app.quit()
The final line causes Excel to throw an error that I have to click out of for the process to continue.
The solution I found which works with both xlwings 0.10.0 and 0.26.1 is to simply brute force with the app.kill() method:
#Close workbooks and app
wbDestination.close()
wbSource.close()
app.screen_updating = True
#app.quit() <- throws error
app.kill() <- no error
Not sure what unintended side effects this might have, but apparently the .kill() command was introduced in version 0.9.0. As long as you close the workbooks first, I dont see how it can cause any problems with data loss or corruption.
Since version 0.24.3, the idiomatic way is using with xw.App() as app.
The advantage of this is that there won't be any hidden excel processes left over in the background, if you use a hidden instance (visible=False) and your code fails.
import xlwings as xw
# You could also omit .screen_updating (and .display_alerts)
# and use xw.App(visible=False) instead, if appropriate.
with xw.App() as app:
app.display_alerts = False
app.screen_updating = False
wbSource = xw.Book()
wbDestination = xw.Book()
# Do your stuff here.
app.screen_updating = True
# Save as needed, for example: wbDestination.save()
wbSource.close()
wbDestination.close()
I'm trying to open password protected excel workbooks and found the code I've posted below on this page but when I try to impliment it I'm getting a sytaxError and an indentationEorror
xlpassword.py
import xlwings as xw
from autoit.autoit import AutoItError
import autoit
import threading
class _WB(object):
def __init__(self, path, password=None):
self.path = path
self.password = password
self.name = path
#staticmethod
def _handlepassword(password):#this line is giving the error
if password:
autoit.win_wait_active("[TITLE:Excel]", 5)
autoit.send(password)
autoit.send("{ENTER}")
def op(self):
try: # If already opened
autoit.win_activate("%s - Excel"%self.name)
self.book = xw.Book(self.path)
except AutoItError: # Else
t = threading.Thread(target=self._handlepassword, args=(self.password,))
t.start()
self.book = xw.Book(self.path)
t.join()
finally:
return self
def _wait(self):
autoit.win_wait_active("%s - Excel"%self.name, 1)
def close(self):
self._wait()
self.book.close()
autoit.win_close("Excel")
when I get to the def _handlepassword line I get the output
SyntaxError: unexpected EOF while parsing (<string>, line 1)
IndentationError: unexpected indent (<string>, line 1)
Which means when I import xlpassword.py into another python script, that new script fails to run
test_run.py
import pandas as pd
from xlpassword import * #I know this isn't best practice
PATH = "C:\\Path\\to\\my\\file.xlsx"
print(PATH)
wb = _WB(path=PATH, password='MyP8ssw0rd')
I'm using python 3.8.1 on a windows 10 machine, and I have tried to run the code in spyder, sublime, and Rstudio (I normally work in Rstudio but I thought that might be what's causing the problem.)
I have read up on classes, class methods, and static methods and I can't see what I'm doing wrong here so if anyone could provide assistance it would help a lot.
Since v0.16.1, xlwings supports opening of password protected workbooks out of the box:
import xlwings as xw
wb = xw.Book(password='mypassword')
See also the API Reference: https://docs.xlwings.org/en/stable/api.html#xlwings.Book
When I'm trying to read data from sqlalchemy df=pd.read_sql_table(table, con, schema) getting runtime error :
runtime/cgo: could not obtain pthread_keys
tried 0x115 0x116 0x117 0x118 0x119 0x11a 0x11b 0x11c 0x11d 0x11e 0x11f 0x120 0x121 0x122 0x123 0x124 0x125 0x126 0x127 0x128 0x129 0x12a 0x12b 0x12c 0x12d 0x12e 0x12f 0x130 0x131 0x132 0x133 0x134 0x135 0x136 0x137 0x138 0x139 0x13a 0x13b 0x13c 0x13d 0x13e 0x13f 0x140 0x141 0x142 0x143 0x144 0x145 0x146 0x147 0x148 0x149 0x14a 0x14b 0x14c 0x14d 0x14e 0x14f 0x150 0x151 0x152 0x153 0x154 0x155 0x156 0x157 0x158 0x159 0x15a 0x15b 0x15c 0x15d 0x15e 0x15f 0x160 0x161 0x162 0x163 0x164 0x165 0x166 0x167 0x168 0x169 0x16a 0x16b 0x16c 0x16d 0x16e 0x16f 0x170 0x171 0x172 0x173 0x174 0x175 0x176 0x177 0x178 0x179 0x17a 0x17b 0x17c 0x17d 0x17e 0x17f 0x180 0x181 0x182 0x183 0x184 0x185 0x186 0x187 0x188 0x189 0x18a 0x18b 0x18c 0x18d 0x18e 0x18f 0x190 0x191 0x192 0x193 0x194
Below is the code:
class TeradataWriter:
def __init__(self):
print("in init")
def read_data_from_teradata(self):
try:
print('Create main')
import pdb;pdb.set_trace()
eng = self.create_connection_engine()
df = pd.read_sql_table("table_name", eng, schema="schema")
print(df)
except Exception as ex:
print('Exception: %s', ex.with_traceback())
def create_connection_engine(self):
try:
return create_engine('teradatasql://' + constants.TERADATA_HOST + '/?user='+ constants.TERADATA_USER_NAME + '&password=' + constants.TERADATA_PWD, echo=False)
except Exception as ex:
LOGGER.error('Exception: %s', ex)
raise Exception(message_constants.ERROR_WHILE_CREATING_CONNECTION_WITH_TERADATA)
if __name__ == "__main__":
p = TeradataWriter()
p.write_dataframe_to_teradata()
Edit: This is fixed. I was finally able to get their support and engineering team to reproduce the issue. They now build the driver with a newer version of go. Upgrade to >= 17.0.3, and you shouldn't see anymore segfaults.
I think I finally figured out why this happens. According to this Go issue, it happens if "If the host process spawns threads prior to loading the shared library, the offset will have changed."
In my case, I was importing matplotlib.pyplot in IPython before calling code that loads the shared library. This starts an event loop and causes the conditions that lead to the segfault.
I changed my code to import matplotlib.pyplot after configuring the teradata driver, and it went away.
According to the Go issue, they just need to recompile the library with a newer version of Go, which I've asked them to do. We'll see what they say.
I have run in to same issue -
So to fix the problem, I moved connect statement to main and it kind of fixed. Its worth trying in your case.
I have the strangest problem I have ever met in my life.
I have a part of my code that looks like this:
class AzureDevOpsServiceError(Exception):
pass
skip = ["auto"]
def retrieve_results():
print(variable_not_defined)
... # some useful implementation
if not "results" in skip:
try:
print("before")
retrieve_results()
print("after")
except AzureDevOpsServiceError as e:
print(f"Error raised: {e}")
Obviously, this shall raise an error because variable_not_defined is, well, not defined.
However, for some strange reasons, the code executes correctly and prints
before
after
I have tried to call the function with an argument (retrieve_results(1234)) or adding an argument in the function (def retrieve_results(arg1) and retrieve_results()): both modifications will trigger an exception, so obviously the function is called.
Anyone has got a similar issue and knows what happens?
FYI: this is actually what my implementation looks like:
from azure.devops.exceptions import AzureDevOpsServiceError
import logging
def _retrieve_manual_results(connect: Connectivity, data: DataForPickle) -> None:
"""Retrieve the list of Test Results"""
print("G" + ggggggggggggggggggggggggggggggggggggg)
logger = connect.logger
data.run_in_progress = [165644]
if __name__ == "__main__":
p = ...
connect = ...
data = ...
if not "results" in p.options.skip:
try:
print("........B.........")
_retrieve_manual_results(connect, data)
print("........A.........")
except AzureDevOpsServiceError as e:
logging.error(f"E004: Error while retrieving Test Results: {e}")
logging.debug("More details below...", exc_info=True)
As highlighted by #gmds, it was a problem of cache.
Deleting the .pyc file didn't do much.
However, I have found a solution:
Renaming the function (e.g. adding _)
Running the program
Renaming back (i.e. removing _ in the previous example)
Now, the issue is solved.
If anyone knows what is going behind the scene I am very interested.
So I am new to parallel processing, but I was starting to get it working for parsing multiple Excel files simultaneously. It works well with when I only use openpyxl, but that is a basic XML parser as I understand it. When I include the part that uses XLWings (I like to take advantage of its ability to evaluate equations in Excel for file verification purposes), I get the following error:
pywintypes.com_error: (-2147221008, 'CoInitialize has not been called.', None, None)
This is roughly the code I use to initialize a new XLWings instance and load a workbook:
def openWorkbook(self, filePath):
app = xw.apps.add()
app.display_alerts = False
app.screen_updating = False
wb = self.app.books(filePath) #Note that this is called only once for each workbook.
app.screen_updating = True
app.quit()
Is there some way to get XLWings to open several simultaneous instances of Excel? Should I try doing something like this? If so, I am not sure how the initialization would work with giving threads over to XLWings.
So I figured out the solution, it was actually surprisingly easy. I just added pythoncom.CoInitialize() from pythoncom package before my xw.apps.add() call:
ParallelProcessController.py
from multiprocessing.dummy import Pool
from LoadWorkbook import openWorkbook
def callOpenWorkbookInParallel(self, lsExcelFiles):
pool = Pool(processes=3)
pool.map(openWorkbook, lsExcelFiles)
LoadWorkbook.py
import xlwings as xw
import pythoncom
def openWorkbook(self, filePath):
pythoncom.CoInitialize()
app = xw.apps.add()
wb = app.books(filePath)
app.quit()