Develop Reference

How to ensure xlwings connection is closed if script fails

I am trying to develop something with xlwings because I need to manipulate a xls file with macros etc. Although it is always good to close connections, Excel is notorious in that it blocks access if more than one instance is running. Therefore I need to make sure that the app closes even though my code fails somewhere upstream.
I am currently doing this with a try statement that spans the whole script and when it fails calls app.quit(). But this suppresses my error messages, which makes debugging hard. So I feel there must be something better.
In another context I have seen with being used. And I have the feeling it would apply here too, but I do not understand how it works, nor how it would work in this specific case.
import xlwings as xw
def myexcel():
try:
#connect to Excel app in the background
excel = xw.App(visible=False)
# open excel book
wb = excel.books.open(str(file))
# asign the active app so it can be closed later
app = xw.apps.active
# more code goes here
except:
app.quit()
How could one make sure that the excel connection gets always closed no-matter the most efficient way?
If with is the solution, I would also appreciate a pointer to a good source to learn more about that concept.

As you mentioned, you can use a with statement and build your own contextmanager. Here's a converted example based on your code:
import xlwings as xw
class MyExcelApp:
def __init__(self):
self.excel = xw.App(visible=False)
def __enter__(self):
return self.excel
def __exit__(self, exc, value, traceback):
# Handle your app-specific exceptions (exc) here
self.excel.quit()
return True
# ^ return True only if you intend to catch all errors in here.
# Otherwise, leave as is and use try... except on the outside.
class MyExcelWorkbook:
def __init__(self, xlapp, bookname):
self.workbook = xlapp.books.open(bookname)
def __enter__(self):
return self.workbook
def __exit__(self, exc, value, traceback):
# Handle your workbook specific exceptions (exc) here
# self.workbook.save() # depends what you want to do here
self.workbook.close()
return True
# ^ return True only if you intend to catch all errors in here.
# Otherwise, leave as is and use try... except on the outside.
With this set up you can simply call it like this:
with MyExcelApp() as app:
with MyExcelWorkbook(filename) as wb:
# do something with wb
You can also implement it with a generator, which will be quite similar to the other answer.
Here's a simplified version:
import xlwings as xw
from contextlib import contextmanager
#contextmanager
def my_excel_app():
app = xw.App(visible=False)
try:
yield app
except: # <-- Add SPECIFIC app exceptions
# Handle the errors
finally:
app.quit()
Usage:
with my_excel() as app:
wb = app.books.open(some_file)
# do something...

you do it right - using try block in this case is the way to go. With statement is good when you need to open file, but not for your use case when you use library which is opening excel file using its own way.
To show details of exception you can change your code as follows:
import xlwings as xw
def myexcel():
try:
#connect to Excel app in the background
excel = xw.App(visible=False)
# open excel book
wb = excel.books.open(str(file))
# asign the active app so it can be closed later
app = xw.apps.active
# more code goes here
finally:
app.quit()
except Exception as e:
print('exception catched: {}'.format(e))
app.quit()

Preferred solution
xlwings added a solution in v0.24.3 to this problem:
xlwings.App() can now be used as context manager, making sure that there are no zombie processes left over on Windows, even if you use a hidden instance and your code fails. It is therefore recommended to use it whenever you can, like so:
import xlwings as xw
with xw.App(visible=False) as app:
wb = xw.Book("test.xlsx")
sheet = wb.sheets['sheet1']
# To evoke an error, I try to call an non-exisiting sheet here.
nonexistent_sheet["A1"]
Solution before v24.0.3
You can use the library traceback, which makes debugging easier, because the error is displayed in red color. See this example:
import xlwings as xw
import traceback
filename = "test.xlsx"
try:
# Do what you want here in the try block. For example, the following lines.
app = xw.App(visible=False)
wb = xw.Book(filename)
sheet = wb.sheets['sheet1']
# To evoke an error, I try to call an nonexistent sheet here.
nonexistent_sheet["A1"]
# Use BaseException because it catches all possible exceptions: https://stackoverflow.com/a/31609619/13968392
except BaseException:
# This prints the actual error in a verbose way.
print(traceback.print_exc())
app.quit()
The error displays with print(traceback.print_exc()) as follows:

Unable to lock file in python

I have written two scripts in python that will be working with data in the same directory. One script will be set to run every 5 minutes and will save data to the directory, and then once a day the other script will zip all of that data, archive it and delete the original files. To avoid the archiving script deleting files which may be being saved by the worker script, I want to create a system-wide mutex so that the worker script knows not to run while the archiver is doing its thing.
I've done some searching and seen that on unix-based systems, the generally accepted method of doing this is to attempt to lock a file. If you get the lock then great, go ahead and run, if you can't get it then you know the other process is already running. I've written the following code:
import fcntl
import traceback
class ProcessLock:
def __init__(self, path_to_file, block):
self.file_path = path_to_file
try:
options = fcntl.LOCK_EX
if not block:
options = options | fcntl.LOCK_NB
self.file = open(path_to_file, 'w+')
self.lock = fcntl.flock(file, options)
except:
print 'caught something: {}'.format(traceback.format_exc())
self.file = None
self.lock = None
def is_locked(self):
return self.lock is not None
def unlock(self):
self.lock = None
self.file = None
def aquire_lock(lock_name):
path = '/tmp/{}.lock'.format(lock_name)
return ProcessLock(path, False)
def aquire_lock_blocking(lock_name):
path = '/tmp/{}.lock'.format(lock_name)
return ProcessLock(path, True)
However for the life of me I can't get it to actually work. I have searched and all of the samples I've seen and other questions posted on here seem to work using the code that I have got. I've also tried both flock and lockf, but neither work. The call to open works correctly, but I get the following logged out to the console:
self.lock = fcntl.flock(file, options)
TypeError: descriptor 'fileno' of 'file' object needs an argument
I don't know enough about Python to be able to know what this error means. Hopefully someone can see if I'm doing anything wrong. I'm running this in Pycharm on macOS

Using XLWings with parallel processing

So I am new to parallel processing, but I was starting to get it working for parsing multiple Excel files simultaneously. It works well with when I only use openpyxl, but that is a basic XML parser as I understand it. When I include the part that uses XLWings (I like to take advantage of its ability to evaluate equations in Excel for file verification purposes), I get the following error:
pywintypes.com_error: (-2147221008, 'CoInitialize has not been called.', None, None)
This is roughly the code I use to initialize a new XLWings instance and load a workbook:
def openWorkbook(self, filePath):
app = xw.apps.add()
app.display_alerts = False
app.screen_updating = False
wb = self.app.books(filePath) #Note that this is called only once for each workbook.
app.screen_updating = True
app.quit()
Is there some way to get XLWings to open several simultaneous instances of Excel? Should I try doing something like this? If so, I am not sure how the initialization would work with giving threads over to XLWings.

So I figured out the solution, it was actually surprisingly easy. I just added pythoncom.CoInitialize() from pythoncom package before my xw.apps.add() call:
ParallelProcessController.py
from multiprocessing.dummy import Pool
from LoadWorkbook import openWorkbook
def callOpenWorkbookInParallel(self, lsExcelFiles):
pool = Pool(processes=3)
pool.map(openWorkbook, lsExcelFiles)
LoadWorkbook.py
import xlwings as xw
import pythoncom
def openWorkbook(self, filePath):
pythoncom.CoInitialize()
app = xw.apps.add()
wb = app.books(filePath)
app.quit()

how to unittest and mock for open funtion

I have read many article over the last 6 hours and i still don't understand mocking and unit-testing. I want to unit test a open function, how can i do this correctly?
i am also concerned as the bulk of my code is using external files for data import and manipulation. I understand that i need to mock them for testing, but I am struggling to understand how to move forward.
Some advice please. Thank you in advance
prototype5.py
import os
import sys
import io
import pandas
pandas.set_option('display.width', None)
def openSetupConfig (a):
"""
SUMMARY
Read setup file
setup file will ONLY hold the file path of the working directory
:param a: str
:return: contents of the file stored as str
"""
try:
setupConfig = open(a, "r")
return setupConfig.read()
except Exception as ve:
ve = (str(ve) + "\n\nPlease ensure setup file " + str(a) + " is available")
sys.exit(ve)
dirPath = openSetupConfig("Setup.dat")
test_prototype5.py
import prototype5
import unittest
class TEST_openSetupConfig (unittest.TestCase):
"""
Test the openSetupConfig function from the prototype 5 library
"""
def test_open_correct_file(self):
result = prototype5.openSetupConfig("Setup.dat")
self.assertTrue(result)
if __name__ == '__main__':
unittest.main()

So the rule of thumb is to mock, stub or fake all external dependencies to the method/function under test. The point is to test the logic in isolation. So in your case you want to test that it can open a file or log an error message if it can't be opened.
import unittest
from mock import patch
from prototype5 import openSetupConfig # you don't want to run the whole file
import __builtin__ # needed to mock open
def test_openSetupConfig_with_valid_file(self):
"""
It should return file contents when passed a valid file.
"""
expect = 'fake_contents'
with patch('__builtin__.open', return_value=expect) as mock_open:
actual = openSetupConfig("Setup.dat")
self.assertEqual(expect, actual)
mock_open.assert_called()
#patch('prototype5.sys.exit')
def test_openSetupConfig_with_invalid_file(self, mock_exit):
"""
It should log an error and exit when passed an invalid file.
"""
with patch('__builtin__.open', side_effect=FileNotFoundError) as mock_open:
openSetupConfig('foo')
mock_exit.assert_called()

'Could not open the ID file' when using win32com.client in python

I have an issue connecting to Lotus Notes from Python using win32com.client.
I am using the following code:
import win32com.client
import csv # imports the csv module
import sys # imports the sys module
import re
notesServer = "AALMBX01/Server/..."
notesPass = "PASS"
#Connect to notes database on server
notesSession = win32com.client.Dispatch('Lotus.NotesSession')
notesSession.Initialize(notesPass)
db_name = 'mail\iizs.nsf'
db = notesSession.getDatabase(notesServer, db_name)
view = db.GetView("($All)")
doc = view.getFirstDocument()
And I get the following error:
(-2147352567, 'Exception occurred.', (0, u'NotesSession', u'Notes
error: Wrong Password. (Passwords are case sensitive - be sure to use
correct upper and lower case.)'
Also tried leaving password blank and disabling 'request password for LN applications' in the interface. With a blank password, I am getting the following error message:
(-2147352567, 'Exception occurred.', (0, u'NotesDatabase', u'Database
AALMBX01/Server/...!!mail\iizs.nsf has not been opened yet'
I have tried the following:
Using lnlib and get_session function.
Checking that notus.ini file is in place (C:\Users\iizs\NotesData in my case) and includes the reference to userid (tried adding the full path to userid file, which is located in C:\Users\iizs\NotesData\data).
Adding a value to HKEY_CURRENT_USER\Software\Lotus\Notes(optional-version)\NotesIniPath
Added the folder containing notes.ini file (C:\Users\iizs\NotesData) and user.id file (C:\Users\iizs\NotesData\data) to the PATH environment variable.
The error is still the same. Tried copying user.id to one of system folders too (system32) - did not help either.
Any suggestions?

import win32com.client
import pywintypes
from win32com.client import Dispatch
from win32com.client import constants
notesSession = Dispatch('Lotus.NotesSession')
dir(constants)
dir(notesSession)
Password = 'S3cretP455w0rd'
Server = 'yourserver/yourapp' # yourserver = '' if local
scPath = 'view.nsf'
notesSession.Initialize(Password)
HTH!!
Also, one 'gotcha' that happened for me was network drives - if your NOTES.INI file contains a network path, try deleting and adding it in the Python code:
[code]
import os
os.system('net use w: /delete')
os.system('net use w: \\\\apps\\NotesFolder')
[/code]