Develop Reference

Running TCL code (on an existing TCL shell) from Python

Introduction
I recently started working on a legacy product, under Linux, which incorporates a builtin TCL shell. Due to company limitations, I can't gain control to "behind the scenes" and all of the code I can write must be run under this TCL shell, handling the pre-defined clumsy TCL API of the product.
I found myself wondering a few times whether it will be possible to patch some Python into this setup, as some solutions seem more legitimate in Python than in TCL. By patching Python I mean: either calling the Python code from the TCL code itself (which can be done with Elmer, for example), or to use Python from outside of the product to wrap the TCL API (for which I found no "classic" solution).
The Problem
Given that the product already has an existing TCL shell in it, most solutions I browsed through (e.g. Tkinter) can't be used to run the TCL code from Python. I need to "inject" the code to the existing shell.
Solutions Considered
As a solution, I thought about bringing up a simple server on the TCL side which runs simple commands it receives from the Python side. I wrote a small demo and it worked. This enabled me to write a nice, class based, wrapper in Python for the clumsy TCL API, also manage a command queue, etc.
Another two solutions I thought of is forking the software from Python and playing with the read/write file descriptors or connecting through FIFO rather than a socket.
However, I wondered whether I'm actually doing it right or can you suggest a better solution?
Thanks in advance.

First:
If you just want a class based OO system to write your code in, you don't need python, Tcl can do OO just fine. (built in with 8.6, but there are quite a few options to get OO features, classes etc. in older versions too, e.g. Tcllibs SNIT or STOOOP
If you still feel Python is the superior tool for the task at hand (e.g. due to better library support for some tasks), you can 'remote control' the Tcl interpreter using the Tcllib comm package. This needs a working event loop in the Tcl shell you want to control, but otherwise it is pretty simple to do.
In your Tcl shell, install the Tcllib comm package.
(ask again if you need help with that)
Once you have that, start the comm server in your Tcl shell.
package require comm
set id [::comm::comm self]
# write ID to a file
set fd [open idfile.txt w]
puts $fd $id
close $fd
proc stop_server {} {set ::forever 1 }
# enter the event loop
vwait forever
On the python side, you do nearly the same, just in Tkinter code.
Basically like this:
import Tkinter
interp = Tkinter.Tcl()
interp.eval('package require comm')
# load the id
with open('idfile.txt') as fd:
comm_id = fd.read().strip()
result = interp.eval(
'comm::comm send {0!s} {1!s}'.format(comm_id, '{puts "Hello World"}')
Using that python code and your shell, you should see Hello World printed in your shell.
Read the comm manual for more details, how to secure things, get callbacks etc. comm
If you don't like the tkinter burden, you can implement the wire procotol for comm too, should not be too hard in Twisted or with the new async support in Python 3.x.
The on the wire protocol is documented in:
comm wire protocol

Can I use msilib or other Python libraries to extract one file from an .msi file?

What I really want to do is determine whether a particular file in the MSI exists and contains a particular string.
My current idea is to run:
db = msilib.OpenDatabase('c:\Temp\myfile.msi',1)
query = "select * from File"
view = db.OpenView(query)
view.Execute(None)
cur_record = view.Fetch() # do this until I get the record I want
print cur_record.GetString(3) # do stuff with this value
And then if it's there, extract all the files using
msiexec /a c:\Temp\myfile.msi /qn TARGETDIR=c:\foo
and use whatever parser to see whether my string is there. But I'm hoping a less clunky way exists.

Note that, as the docs for msilib say, "Support for reading .cab files is currently not implemented". And. more generally, the library is designed for building .msi files, not reading them. And there is nothing else in the stdlib that will do what you want.
So, there are a few possibilities:
Find and install another library, like pycabinet. I know nothing about this particular library; it's just the first search hit I got; you probably want to search on your own. But it claims to provide a zipfile-like API for CAB files, which sounds like exactly the part you're missing.
Use win32com (if you've got pywin32) or ctypes (if you're a masochist) to talk to the underlying COM interfaces and/or the classic Cabinet API (which I think is now deprecated, but still works).
Use IronPython instead of CPython, so you can use the simpler .NET interfaces.
Since I don't have a Windows box here, I can't test this, but here's a sketch of Christopher Painter's .NET solution written in IronPython instead of C#:
import clr
clr.AddReference('Microsoft.Deployment.WindowsInstaller')
clr.AddReference('Microsoft.Deployment.WindowsInstaller.Package')
from Microsoft.Deployment.WindowsInstaller import *
from Microsoft.Deployment.WindowsInstaller.Package import *
def FindAndExtractFiles(packagePath, longFileName):
with InstallPackage(packagePath, DatabaseOpenMode.ReadOnly) as installPackage:
if installPackage.FindFiles(longFileName).Count() > 0:
installPackage.ExtractFiles()

Realize that in using Python you have to deal with the Windows Installer (COM) Automation interface. This means you have to do all the database connections, querying and processing yourself.
If you could move to C# ( or say PowerShell ) you could leverage some higher level classes that exist in Windows Installer XML (WiX) Deployment Tools Foundation (DTF).
using Microsoft.Deployment.WindowsInstaller;
using Microsoft.Deployment.WindowsInstaller.Package;
static void FindAndExtractFiles(string packagePath, string longFileName)
{
using (var installPackage = new InstallPackage(packagePath, DatabaseOpenMode.ReadOnly))
{
if(installPackage.FindFiles(longFileName).Count() > 0 )
installPackage.ExtractFiles();
}
}
You could also write this as ComVisible(True) and call it from Python.

The MSI APIs are inherently clunky, so it's only a matter of where the abstraction lies. Bear in mind that if you just need this a couple times, it may be easier to browse the cab file(s) manually in Explorer. (Files are stored by file key instead of file name).

Using a TLB-defined interface with Python and COM

I will try to keep this question as tight as possible, but if it seems that I am saying insane things, it is almost certainly because I am ignorant of some key point, so please do correct me.
I am writing a program, in a Windows environment, that will interface with an existing application that has a COM interface to allow 3rd-party software to interact with it.
I have read all of the documentation for this application, and it says that there is a TLB file that defines the functions and data available via COM.
How do I use the TLB file with python? How do I discover the progID of the application so that I can interface with it (this isn't given in the documentation).
I'm pretty lost. I have a fair amount of experience with Python, but I am completely new to developing in a Windows environment. Any help would be enormously helpful. I have been reading all the documentation on win32com, but I still have no clue what to do, as no one addresses -- as far as I have seen -- bringing in a TLB file.

The questions asked is to link the custom TLB file with COM client to be developed in python. I have done a small example code for my COM server developed in C# and same is accessed by python client using "comtypes" package.
The below snippet of code gives:
import comtypes.client as CC
import comtypes
ccHandle = CC.CreateObject("CSharpServer.InterfaceImplementation")
print (ccHandle)
import comtypes.gen.CSharpServer as CS
InterfaceHandle = ccHandle.QueryInterface(CS.IManagedInterface)
print ("output of PrintHi function = ", InterfaceHandle.PrintHi("World"))
The above python script is for the C# COM server code available at http://msdn.microsoft.com/en-us/library/aa645738(v=vs.71).aspx (refer to the File 1: CSharpServer.cs).

OK, it's been a while since I've done this, and I'm not a COM expert by any means. Read the COM chapter from Python Programming on Windows to see how to do this. Follow along with the examples (trying things out against Excel) to get a feel of how things work.
First off, install the PyWin32 Extensions if you haven't already. This is the package that gives you pythonwin.exe and the COM interface modules. Get it from here.
Then you are going to open the "COM Makepy Utility" from PythonWin's Tools menu. Browse through the list of registered COM components (some will be typelibs, others DLLs) until you identify the one you have (you have to do a bit of detective work). Click OK to generate the Python glue code. You will then need to run it again with the -i command-line argmument to generate the boilerplate code so your python script can use this glue. Here's a paraphrase of the O'Reilly example for the Microsoft Excel Object library:
import win32com.client
from win32com.client import gencache
gencache.EnsureModule('{00020813-0000-0000-C000-000000000046}', 0, 1, 2)
earlyBound = win32com.client.Dispatch("Excel.Application")
lateBound = win32com.client.dynamic.Dispatch("Excel.Application")
print earlyBound.ActiveCell()
Using early-bound objects is optional, but it does improve performance.
To find the ProgID is again a bit of detective work, although this answer seems to imply it's going to be hard. Try poking around the HKEY_CLASSES_ROOT hive of the registry with RegEdit to see if you see a ProgID that looks promising.

Accessing a Panatone Huey via Python

I have a Panatone Huey, a monitor calibration probe (device you attach to the monitor, and it gives you colour readings) - I want to get readings from the device in Python.
Having never written such a device driver before, I'm not sure where to start.
I've found are two open-source C/C++ projects that interface with the Heuy - ArgyllCMS and mcalib.
ArgyllCMS comes with a spotread command which returns readings from the device, although it only functions as an interactive command line tool, so running it via subprocess will not (easily) work.
The code ArgyllCMS uses to communicate with the device is in spectro/huey.c
Not tried it (only just found it while writing this question), but mcalib contains much less code, mainly just heuy.cpp - however it has a worrying number of FIXME comments and incomplete methods, and the code appears to have been automatically generated (unhelpful variable names)
There seems to be three options:
Modify spotread to work without any interactive prompts, call it via subprocess
Create a C-based Python module around huey.c or huey.cpp
Re-implement the interface using something like PyUSB
Being much more familiar with Python, I'm tempted to use PyUSB, but will this be substantially more work than wrapping existing code with the Python C API? Is there anything obvious in either of the C implementations that will not be easily doable in PyUSB?

Given the existence of spotread the easiest (though perhaps not the best) way to proceed would be to use pexpect. It allows you to interact with other command-line programs.

IronPython: What kind of jobs you ever done with IronPython instead of standard .NET languages (e.g., C#)

I am learning IronPython along wiht Python. I'm curious what kinds of tasks you tend to use IronPython to tackle more often than standard .NET languages.
Thanks for any example.

One example that is a perfect match for IronPython is when you want to include scriptability in your application. I have been working on systems where the user can interact with the whole application class model directly through the in-app python scripting layer and it allows for a great deal of flexibility for advanced users.
A tangible example is when you want to expose "hooks" in the application where the user can customize the business rules (say, do a custom broker fee calculation when a new trade is created in a trading system for instance).
Edit per request:
here is a simple (mocked-upped) example. When a user creates a new trade in the system, the system checks if the following Python function is defined, and if present, the trade gets the result of the function as a fee added to it before it is committed to the database:
def calculate_broker_fee(trade):
fee = 0.043 # default value
if trade.exchange in ["SWX", "EURONEXT"] and \
trade.instrument.instrument_type == "Bond":
fee = trade.quantity*0.00234
return fee

I've just deployed my IronPython Point-of-Sales service application at the beginning of this month. The service application is a RESTful http server serving query & transactional request to .NET WinForms client. With help of some remoting library, the service application is purely implemented in IronPython.
In my opinion, Python is undoubtly the best language to code complex business logics. Here are my reasons.
The language is very expressive. I could come up with endless ideas for internal DSL that make my business logic shorter and simpler to understand.
It's interactive. Trouble-shooting and logic testing can be done interactively.
It's dynamic. It's freedom. No xml configuration. No plumbing. No compile.
I can work in my favorite editor.
Most skeptics alway mention "code-completion" and "debugger". Well I miss them sometimes. However, I'm aware that I deliberately gave up those conveniences in favor to, a much more important factor, comprehensibility. With proper unit-testing and logging, I'd pick IronPython over any languages for my business logics.
updated:
Sometimes, I experiment and log user issues with Ironpython Script that act as if being a client like this:
>> from boon.service import client
>> CASH_PAYMENT_TYPE = '000000011'
>> cl = client.Client('http://pos-server/bin?posB2K')
>> cl.connect('user', 'password')
>> order = cl.workspace('pos.Order')
>> order.load('1312')
>> payments = order.dataset.Tables['POS_PAYMENTS']
>> payments.Rows[0]['PAYMENT_TYPE_ID'] = CASH_PAYMENT_TYPE
>> order.save()
Sometimes, I investigate bugs by scripting server object like this:
>> from boon import pos
>> pos.register_pos_service(debug=True)
>> from boon.service import get_instance
>> possvc = get_instance('pos')
>> print possvc.store['POS_PAYMENTS'] \
.. .where(lambda r: r.POS_HD_ID == 1312) \
.. .include('PAYMENT_TYPE_ID', 'PAY_AMT') \
.. .list()
[('000000011', 1520)]
You may find the code not too elegance since I prefer to based my work on ADO.NET DataSet. It's simpler for Windoows Forms client, however.

In the day job, it's my standard language for those little bits of build process that are too much for .bat files and not heavyweight enough to demand a separate executable; this includes anything that could use a little bit of XML processing or reflection -- generating Wix files with systematic handling of 32 and 64 bit installs, for example. It beats out PowerShell in this role because IronPython is an XCOPY install onto build machines.
It's also very useful for prototyping fragments of code against unfamiliar or complex APIs (WMI and Active Directory being the usual ones for me), or diagnosing problems in code using those APIs (like sniffing out the oddities that happen when you're on the domain controller, rather than elsewhere).

Created a load tool for a MS Group Chat Server plugin. The GC API is in C#. I wrapped that into a dll and had FePy load it. The main application, configuration scripts etc are all in FePy.

I use IronPython for a few different purposes:
An alternative to Powershell when I need to script something and invoke a .NET library, or when the script is complicated enough to warrant a real programming language.
Embedding in a .NET app for scriptable plugins.
Prototyping and testing .NET libs in immediate mode. This is way easier than making a test project in C#

We use it a lot for small administrative tools against SharePoint. In particular it is fantastic for exploring the API against real data (with all its real life quirks). Development iterations are faster and you can't always install Visual Studio on production servers.