Develop Reference

Control Flow issue: Python function called but not executed

I have the strangest problem I have ever met in my life.
I have a part of my code that looks like this:
class AzureDevOpsServiceError(Exception):
pass
skip = ["auto"]
def retrieve_results():
print(variable_not_defined)
... # some useful implementation
if not "results" in skip:
try:
print("before")
retrieve_results()
print("after")
except AzureDevOpsServiceError as e:
print(f"Error raised: {e}")
Obviously, this shall raise an error because variable_not_defined is, well, not defined.
However, for some strange reasons, the code executes correctly and prints
before
after
I have tried to call the function with an argument (retrieve_results(1234)) or adding an argument in the function (def retrieve_results(arg1) and retrieve_results()): both modifications will trigger an exception, so obviously the function is called.
Anyone has got a similar issue and knows what happens?
FYI: this is actually what my implementation looks like:
from azure.devops.exceptions import AzureDevOpsServiceError
import logging
def _retrieve_manual_results(connect: Connectivity, data: DataForPickle) -> None:
"""Retrieve the list of Test Results"""
print("G" + ggggggggggggggggggggggggggggggggggggg)
logger = connect.logger
data.run_in_progress = [165644]
if __name__ == "__main__":
p = ...
connect = ...
data = ...
if not "results" in p.options.skip:
try:
print("........B.........")
_retrieve_manual_results(connect, data)
print("........A.........")
except AzureDevOpsServiceError as e:
logging.error(f"E004: Error while retrieving Test Results: {e}")
logging.debug("More details below...", exc_info=True)

As highlighted by #gmds, it was a problem of cache.
Deleting the .pyc file didn't do much.
However, I have found a solution:
Renaming the function (e.g. adding _)
Running the program
Renaming back (i.e. removing _ in the previous example)
Now, the issue is solved.
If anyone knows what is going behind the scene I am very interested.

Implementing retry decorator one method higher than exception

I am trying to implement the retry decorator on a serial query. A general idea of my code is shown below. I am struggling to get it to retry when the decorator is one method up in the hierarchy. How can I have the method be retried when it's one method up from the method that throws the exception?
One complication that is frustrating is my increment time per retry depends on the actual command. Some commands require more time than others. That's why I have the extra_time_per_retry passed in, and couldn't implement the retry decorator using the traditional #retry style.
FYI the _serial is created in the class on init via pySerial.
I got it to work with the retry decorator directly above the method that throws the exception. I would like it to be two above to keep my code clean.
I have tried feeding the retry decorator the exact exception type, but couldn't get it to work.
def _query_with_retries(self, cmd, extra_time_per_retry):
_retriable_query = retry(stop_max_attempt_number=5,
wait_incrementing_start=self._serial.timeout + extra_time_per_retry,
wait_incrementing_increment=10)(self._query)
return _retriable_query(cmd)
def _query(self, cmd):
cmd_msg = cmd + '\r'
self._serial.reset_input_buffer()
self._serial.reset_output_buffer()
self._serial.write(cmd_msg)
return self._readlines()
def _readlines(self):
response_str = self._serial.read_until('\r', 256) # Max 256 bytes
# Parse response here, if a bad one set bad_response = true
if bad_response:
raise ResponseError("Response had custom error xyz")

I guess you could pack your call to _readlines() into an exception handling block, reraising the error:
python 3.x
#retry
def _query(self, cmd):
cmd_msg = cmd + '\r'
self._serial.reset_input_buffer()
self._serial.reset_output_buffer()
self._serial.write(cmd_msg)
try:
answer = self._readlines()
except Exception as e:
raise e
return answer
python 2.x
#retry
def _query(self, cmd):
cmd_msg = cmd + '\r'
self._serial.reset_input_buffer()
self._serial.reset_output_buffer()
self._serial.write(cmd_msg)
try:
answer = self._readlines()
except Exception:
t, v, tb = sys.exc_info()
raise t, v, tb
return answer
This way, you catch the exception directly when it occurs, and raise it inside the method which will be retried. I am not sure whether this declutters enough for you, however it should work.
Some might complain about using a blank except Exception, however since I am reraising it always immediately, I do not see any harm.

How to use COM from python win32com or comtypes to access an IRTDServer?

Environment
Windows 10 + python 3.6.3 64 bit (also tried 32 bit). I am a python developer trying to use COM for (nearly) the first time and hit this huge blocker.
Problem
I have had various errors when trying to use an IRTDServer implemented in a dll (not written by me), via either win32com or comtypes. Using win32com turned out to be more difficult. I have an included an example unittest for both libraries below.
Accessing the server from Excel 2016 works as expected; this returns the expected value:
=RTD("foo.bar", , "STAT1", "METRIC1")
Code using win32com library
Here is a simple test case which should connect to the server but doesn't. (This is just one version, as I have changed it many times trying to debug the problem.)
from unittest import TestCase
class COMtest(TestCase):
def test_win32com(self):
import win32com.client
from win32com.server.util import wrap
class RTDclient:
# are these only required when implementing the server?
_com_interfaces_ = ["IRTDUpdateEvent"]
_public_methods_ = ["Disconnect", "UpdateNotify"]
_public_attrs_ = ["HeartbeatInterval"]
def __init__(self, *args, **kwargs):
self._comObj = win32com.client.Dispatch(*args, **kwargs)
def connect(self):
self._rtd = win32com.client.CastTo(self._comObj, 'IRtdServer')
result = self._rtd.ServerStart(wrap(self))
assert result > 0
def UpdateNotify(self):
print("UpdateNotify() callback")
def Disconnect(self):
print("Disconnect() called")
HeartbeatInterval = -1
_rtd = RTDclient("foo.bar")
_rtd.connect()
Result:
Traceback (most recent call last):
File "env\lib\site-packages\win32com\client\gencache.py", line 532, in EnsureDispatch
ti = disp._oleobj_.GetTypeInfo()
pywintypes.com_error: (-2147467263, 'Not implemented', None, None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test\test.py", line 23, in test_win32com
_rtd.connect()
File "test\test.py", line 16, in connect
self._rtd = win32com.client.CastTo(dispatch, 'IRtdServer')
File "env\lib\site-packages\win32com\client\__init__.py", line 134, in CastTo
ob = gencache.EnsureDispatch(ob)
File "env\lib\site-packages\win32com\client\gencache.py", line 543, in EnsureDispatch
raise TypeError("This COM object can not automate the makepy process - please run makepy manually for this object")
TypeError: This COM object can not automate the makepy process - please run makepy manually for this object
Following those directions, I ran the makepy script successfully:
> env\Scripts\python.exe env\lib\site-packages\win32com\client\makepy.py "foo.bar"
Generating to C:\Users\user1\AppData\Local\Temp\gen_py\3.5\longuuid1x0x1x0.py
Building definitions from type library...
Generating...
Importing module
(I replaced the UUID on stackoverflow for privacy. This UUID is the same as the typelib UUID for "foo.bar".)
The generated file contains various the function and type definitions of both IRtdServer and IRTDUpdateEvent. But in this file, both interfaces are subclasses of win32com.client.DispatchBaseClass, while according to OleViewDotNet, they should be subclasses of IUnknown?
However, when I attempted to run the unittest again, I received the exact same error as before. It is as if the lookup mechanism is not finding the generated module?
Also, GetTypeInfo returning Not implemented is alarming me. From my understanding, win32com uses that method (part of IDispatch COM interface) to determine the argument and return types for all other functions in other interfaces, including IRtdServer. If it's not implemented, it would be unable to determine the types correctly. Yet, the generated file seems to include this information, which is also perplexing.
Code using comtypes library
from unittest import TestCase
class COMtest(TestCase):
def test_comtypes(self):
import comtypes.client
class RTDclient:
# are these for win32com only?
_com_interfaces_ = ["IRTDUpdateEvent"]
_public_methods_ = ["Disconnect", "UpdateNotify"]
_public_attrs_ = ["HeartbeatInterval"]
def __init__(self, clsid):
self._comObj = comtypes.client.CreateObject(clsid)
def connect(self):
self._rtd = self._comObj.IRtdServer()
result = self._rtd.ServerStart(self)
assert result > 0
def UpdateNotify(self):
print("UpdateNotify() callback")
def Disconnect(self):
print("Disconnect() called")
HeartbeatInterval = -1
_rtd = RTDclient("foo.bar")
_rtd.connect()
Result:
File "test\test.py", line 27, in test_comtypes
_rtd.connect()
File "test\test.py", line 16, in connect
self._rtd = self._comObj.IRTDServer()
File "env\lib\site-packages\comtypes\client\dynamic.py", line 110, in __getattr__
dispid = self._comobj.GetIDsOfNames(name)[0]
File "env\lib\site-packages\comtypes\automation.py", line 708, in GetIDsOfNames
self.__com_GetIDsOfNames(riid_null, arr, len(names), lcid, ids)
_ctypes.COMError: (-2147352570, 'Unknown name.', (None, None, None, 0, None))
Some other solutions I've tried
(Based on googling and answers in the comments below)
(Re-)Registered the DLL
Registered the 32 bit version of the DLL and tried python 32 bit
Set compatibility mode of python.exe to Windows XP SP3
Tried not instantiating IRtdServer, that is, replacing these two lines:
self._rtd = self._comObj.IRtdServer()
result = self._rtd.ServerStart(self)
with:
result = self._comObj.ServerStart(self)
The error this time is:
TypeError: 'NoneType' object is not callable
That would seem to indicate that the ServerStart function exists, but is undefined? (Seems really weird. There must be more to this mystery.)
Tried passing interface="IRtdServer" parameter to CreateObject:
def __init__(self, clsid):
self._comObj = comtypes.client.CreateObject(clsid, interface="IRtdServer")
def connect(self):
result = self._comObj.ServerStart(self)
...
The error received is:
File "test\test.py", line 13, in __init__
self._comObj = comtypes.client.CreateObject(clsid, interface="IRtdServer")
File "env\lib\site-packages\comtypes\client\__init__.py", line 238, in CreateObject
obj = comtypes.CoCreateInstance(clsid, clsctx=clsctx, interface=interface)
File "env\lib\site-packages\comtypes\__init__.py", line 1223, in CoCreateInstance
p = POINTER(interface)()
TypeError: Cannot create instance: has no _type_
Tracing code in the comtypes library, that would seem to indicate that the interface parameter wants an interface class, not a string. I found various interfaces defined in the comtypes library: IDispatch, IPersist, IServiceProvider. All are subclasses of IUnknown. According to OleViewDotNet, IRtdServer is also a subclass of IUnknown. This leads me to believe that I need to similarly write an IRtdServer class in python in order to use the interface, but I don't know how to do that.
I noticed the dynamic parameter of CreateObject. The code indicates this is mutually exclusive to the interface parameter, so I tried that:
def __init__(self, clsid):
self._comObj = comtypes.client.CreateObject(clsid, dynamic=True)
def connect(self):
self._rtd = self._comObj.IRtdServer()
result = self._rtd.ServerStart(self)
But the error is the same as my original error: IRtdServer has _ctypes.COMError: (-2147352570, 'Unknown name.', (None, None, None, 0, None))
Any help or clues would be greatly be appreciated. Thank you in advance.
(Not really knowing what I'm doing,) I tried to use OleViewDotNet to look at the DLL:

I ran into same problem.
I also tried using win32com to get excel run that for me, that's a bit unstable to be honest...I cannot even touch my Excel.
Therefore I spent some time looking into this. The problem lies with CastTo. Think that COM object you (and I) loaded just does not contain enough information to be casted (some methods like GetTypeInfo are not implemented etc...)
Therefore I created a wrapper that makes methods of those COM objects callable...not obvious. And this seems working for me.
Client code is modified from a project called pyrtd, which didn't work for various reasons (think due to change of RTD model...return of RefreshData is just completely different now).
import functools
import pythoncom
import win32com.client
from win32com import universal
from win32com.client import gencache
from win32com.server.util import wrap
EXCEL_TLB_GUID = '{00020813-0000-0000-C000-000000000046}'
EXCEL_TLB_LCID = 0
EXCEL_TLB_MAJOR = 1
EXCEL_TLB_MINOR = 4
gencache.EnsureModule(EXCEL_TLB_GUID, EXCEL_TLB_LCID, EXCEL_TLB_MAJOR, EXCEL_TLB_MINOR)
universal.RegisterInterfaces(EXCEL_TLB_GUID,
EXCEL_TLB_LCID, EXCEL_TLB_MAJOR, EXCEL_TLB_MINOR,
['IRtdServer', 'IRTDUpdateEvent'])
# noinspection PyProtectedMember
class ObjectWrapperCOM:
"""
This object can act as a wrapper for an object dispatched using win32com.client.Dispatch
Sometimes the object written by 3rd party is not well constructed that win32com will not be able to obtain
type information etc in order to cast the object to a certain interface. win32com.client.CastTo will fail.
This wrapper class will enable the object to call its methods in this case, even if we do not know what exactly
the wrapped object is.
"""
LCID = 0x0
def __init__(self, obj):
self._impl = obj # type: win32com.client.CDispatch
def __getattr__(self, item):
flags, dispid = self._impl._find_dispatch_type_(item)
if dispid is None:
raise AttributeError("{} is not a valid property or method for this object.".format(item))
return functools.partial(self._impl._oleobj_.Invoke, dispid, self.LCID, flags, True)
# noinspection PyPep8Naming
class RTDUpdateEvent:
"""
Implements interface IRTDUpdateEvent from COM imports
"""
_com_interfaces_ = ['IRTDUpdateEvent']
_public_methods_ = ['Disconnect', 'UpdateNotify']
_public_attrs_ = ['HeartbeatInterval']
# Implementation of IRTDUpdateEvent.
HeartbeatInterval = -1
def __init__(self, event_driven=True):
self.ready = False
self._event_driven = event_driven
def UpdateNotify(self):
if self._event_driven:
self.ready = True
def Disconnect(self):
pass
class RTDClient:
"""
Implements a Real-Time-Data (RTD) client for accessing COM data sources that provide an IRtdServer interface.
"""
MAX_REGISTERED_TOPICS = 1024
def __init__(self, class_id):
"""
:param classid: can either be class ID or program ID
"""
self._class_id = class_id
self._rtd = None
self._update_event = None
self._topic_to_id = {}
self._id_to_topic = {}
self._topic_values = {}
self._last_topic_id = 0
def connect(self, event_driven=True):
"""
Connects to the RTD server.
Set event_driven to false if you to disable update notifications.
In this case you'll need to call refresh_data manually.
"""
dispatch = win32com.client.Dispatch(self._class_id)
self._update_event = RTDUpdateEvent(event_driven)
try:
self._rtd = win32com.client.CastTo(dispatch, 'IRtdServer')
except TypeError:
# Automated makepy failed...no detailed construction available for the class
self._rtd = ObjectWrapperCOM(dispatch)
self._rtd.ServerStart(wrap(self._update_event))
def update(self):
"""
Check if there is data waiting and call RefreshData if necessary. Returns True if new data has been received.
Note that you should call this following a call to pythoncom.PumpWaitingMessages(). If you neglect to
pump the message loop you'll never receive UpdateNotify callbacks.
"""
# noinspection PyUnresolvedReferences
pythoncom.PumpWaitingMessages()
if self._update_event.ready:
self._update_event.ready = False
self.refresh_data()
return True
else:
return False
def refresh_data(self):
"""
Grabs new data from the RTD server.
"""
(ids, values) = self._rtd.RefreshData(self.MAX_REGISTERED_TOPICS)
for id_, value in zip(ids, values):
if id_ is None and value is None:
# This is probably the end of message
continue
assert id_ in self._id_to_topic, "Topic ID {} is not registered.".format(id_)
topic = self._id_to_topic[id_]
self._topic_values[topic] = value
def get(self, topic: tuple):
"""
Gets the value of a registered topic. Returns None if no value is available. Throws an exception if
the topic isn't registered.
"""
assert topic in self._topic_to_id, 'Topic %s not registered.' % (topic,)
return self._topic_values.get(topic)
def register_topic(self, topic: tuple):
"""
Registers a topic with the RTD server. The topic's value will be updated in subsequent data refreshes.
"""
if topic not in self._topic_to_id:
id_ = self._last_topic_id
self._last_topic_id += 1
self._topic_to_id[topic] = id_
self._id_to_topic[id_] = topic
self._rtd.ConnectData(id_, topic, True)
def unregister_topic(self, topic: tuple):
"""
Un-register topic so that it will not get updated.
:param topic:
:return:
"""
assert topic in self._topic_to_id, 'Topic %s not registered.' % (topic,)
self._rtd.DisconnectData(self._topic_to_id[topic])
def disconnect(self):
"""
Closes RTD server connection.
:return:
"""
self._rtd.ServerTerminate()

There seems to already be both server/client for Excel 2002.
pyrtd
Looking at that source, once you create a dispatch object, then it seems to be cast to IRtdServer.
Extract related parts, it becomes below.
from win32com import client, universal
from win32com.server.util import wrap
def __init__(self, classid):
self._classid = classid
self._rtd = None
def connect(self, event_driven=True):
dispatch = client.Dispatch(self._classid)
self._rtd = client.CastTo(dispatch, 'IRtdServer')
if event_driven:
self._rtd.ServerStart(wrap(self))
else:
self._rtd.ServerStart(None)
Please refer to client.py and examples/rtdtime.py of the following sources.
pyrtd - default
pyrtd/rtd/client.py
pyrtd/examples/rtdtime.py

Handling Exceptions in Python Behave Testing framework

I've been thinking about switching from nose to behave for testing (mocha/chai etc have spoiled me). So far so good, but I can't seem to figure out any way of testing for exceptions besides:
#then("It throws a KeyError exception")
def step_impl(context):
try:
konfigure.load_env_mapping("baz", context.configs)
except KeyError, e:
assert (e.message == "No baz configuration found")
With nose I can annotate a test with
#raises(KeyError)
I can't find anything like this in behave (not in the source, not in the examples, not here). It sure would be grand to be able to specify exceptions that might be thrown in the scenario outlines.
Anyone been down this path?

I'm pretty new to BDD myself, but generally, the idea would be that the tests document what behaves the client can expect - not the step implementations. So I'd expect the canonical way to test this would be something like:
When I try to load config baz
Then it throws a KeyError with message "No baz configuration found"
With steps defined like:
#when('...')
def step(context):
try:
# do some loading here
context.exc = None
except Exception, e:
context.exc = e
#then('it throws a {type} with message "{msg}"')
def step(context, type, msg):
assert isinstance(context.exc, eval(type)), "Invalid exception - expected " + type
assert context.exc.message == msg, "Invalid message - expected " + msg
If that's a common pattern, you could just write your own decorator:
def catch_all(func):
def wrapper(context, *args, **kwargs):
try:
func(context, *args, **kwargs)
context.exc = None
except Exception, e:
context.exc = e
return wrapper
#when('... ...')
#catch_all
def step(context):
# do some loading here - same as before

This try/catch approach by Barry works, but I see some issues:
Adding a try/except to your steps means that errors will be hidden.
Adding an extra decorator is inelegant. I would like my decorator to be a modified #where
My suggestion is to
have the expect exception before the failing statement
in the try/catch, raise if the error was not expected
in the after_scenario, raise error if expected error not found.
use the modified given/when/then everywhere
Code:
def given(regexp):
return _wrapped_step(behave.given, regexp) #pylint: disable=no-member
def then(regexp):
return _wrapped_step(behave.then, regexp) #pylint: disable=no-member
def when(regexp):
return _wrapped_step(behave.when, regexp) #pylint: disable=no-member
def _wrapped_step(step_function, regexp):
def wrapper(func):
"""
This corresponds to, for step_function=given
#given(regexp)
#accept_expected_exception
def a_given_step_function(context, ...
"""
return step_function(regexp)(_accept_expected_exception(func))
return wrapper
def _accept_expected_exception(func):
"""
If an error is expected, check if it matches the error.
Otherwise raise it again.
"""
def wrapper(context, *args, **kwargs):
try:
func(context, *args, **kwargs)
except Exception, e: #pylint: disable=W0703
expected_fail = context.expected_fail
# Reset expected fail, only try matching once.
context.expected_fail = None
if expected_fail:
expected_fail.assert_exception(e)
else:
raise
return wrapper
class ErrorExpected(object):
def __init__(self, message):
self.message = message
def get_message_from_exception(self, exception):
return str(exception)
def assert_exception(self, exception):
actual_msg = self.get_message_from_exception(exception)
assert self.message == actual_msg, self.failmessage(exception)
def failmessage(self, exception):
msg = "Not getting expected error: {0}\nInstead got{1}"
msg = msg.format(self.message, self.get_message_from_exception(exception))
return msg
#given('the next step shall fail with')
def expect_fail(context):
if context.expected_fail:
msg = 'Already expecting failure:\n {0}'.format(context.expected_fail.message)
context.expected_fail = None
util.show_gherkin_error(msg)
context.expected_fail = ErrorExpected(context.text)
I import my modified given/then/when instead of behave, and add to my environment.py initiating context.expected fail before scenario and checking it after:
def after_scenario(context, scenario):
if context.expected_fail:
msg = "Expected failure not found: %s" % (context.expected_fail.message)
util.show_gherkin_error(msg)

The try / except approach you show is actually completely correct because it shows the way that you would actually use the code in real life. However, there's a reason that you don't completely like it. It leads to ugly problems with things like the following:
Scenario: correct password accepted
Given that I have a correct password
When I attempt to log in
Then I should get a prompt
Scenario: incorrect password rejected
Given that I have an incorrect password
When I attempt to log in
Then I should get an exception
If I write the step definition without try/except then the second scenario will fail. If I write it with try/except then the first scenario risks hiding an exception, especially if the exception happens after the prompt has already been printed.
Instead those scenarios should, IMHO, be written as something like
Scenario: correct password accepted
Given that I have a correct password
When I log in
Then I should get a prompt
Scenario: correct password accepted
Given that I have a correct password
When I try to log in
Then I should get an exception
The "I log in" step should not use try; The "I try to log in" matches neatly to try and gives away the fact that there might not be success.
Then there comes the question about code reuse between the two almost, but not quite identical steps. Probably we don't want to have two functions which both login. Apart from simply having a common other function you call, you could also do something like this near the end of your step file.
#when(u'{who} try to {what}')
def step_impl(context):
try:
context.execute_steps("when" + who + " " + what)
context.exception=None
except Exception as e:
context.exception=e
This will automatically convert all steps containing the word "try to" into steps with the same name but with try to deleted and then protect them with a try/except.
There are some questions about when you actually should deal with exceptions in BDD since they aren't user visible. It's not part of the answer to this question though so I've put them in a separate posting.

Behave is not in the assertion matcher business. Therefore, it does not provide a solution for this. There are already enough Python packages that solve this problem.
SEE ALSO: behave.example: Select an assertion matcher library

Hang in Python script using SQLAlchemy and multiprocessing

Consider the following Python script, which uses SQLAlchemy and the Python multiprocessing module.
This is with Python 2.6.6-8+b1(default) and SQLAlchemy 0.6.3-3 (default) on Debian squeeze.
This is a simplified version of some actual code.
import multiprocessing
from sqlalchemy import *
from sqlalchemy.orm import *
dbuser = ...
password = ...
dbname = ...
dbstring = "postgresql://%s:%s#localhost:5432/%s"%(dbuser, password, dbname)
db = create_engine(dbstring)
m = MetaData(db)
def make_foo(i):
t1 = Table('foo%s'%i, m, Column('a', Integer, primary_key=True))
conn = db.connect()
for i in range(10):
conn.execute("DROP TABLE IF EXISTS foo%s"%i)
conn.close()
db.dispose()
for i in range(10):
make_foo(i)
m.create_all()
def do(kwargs):
i, dbstring = kwargs['i'], kwargs['dbstring']
db = create_engine(dbstring)
Session = scoped_session(sessionmaker())
Session.configure(bind=db)
Session.execute("COMMIT; BEGIN; TRUNCATE foo%s; COMMIT;")
Session.commit()
db.dispose()
pool = multiprocessing.Pool(processes=5) # start 4 worker processes
results = []
arglist = []
for i in range(10):
arglist.append({'i':i, 'dbstring':dbstring})
r = pool.map_async(do, arglist, callback=results.append) # evaluate "f(10)" asynchronously
r.get()
r.wait()
pool.close()
pool.join()
This script hangs with the following error message.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
task = get()
TypeError: ('__init__() takes at least 4 arguments (2 given)', <class 'sqlalchemy.exc.ProgrammingError'>, ('(ProgrammingError) syntax error at or near "%"\nLINE 1: COMMIT; BEGIN; TRUNCATE foo%s; COMMIT;\n ^\n',))
Of course, the syntax error here is TRUNCATE foo%s;. My question is, why is the process hanging, and can I persuade it to exit with an error instead, without doing major surgery to my code? This behavior is very similar to that of my actual code.
Note that the hang does not occur if the statement is replaced by something like print foobarbaz. Also, the hang still happens if we replace
Session.execute("COMMIT; BEGIN; TRUNCATE foo%s; COMMIT;")
Session.commit()
db.dispose()
by just Session.execute("TRUNCATE foo%s;")
I'm using the former version because it is closer to what my actual code is doing.
Also, removing multiprocessing from the picture and looping over the tables serially makes the hang go away, and it just exits with an error.
I'm also kind of puzzled by the form of the error, particularly the TypeError: ('__init__() takes at least 4 arguments (2 given)' bit. Where is this error coming from? It seems likely it is from somewhere in the multiprocessing code.
The PostgreSQL logs aren't helpful. I see lots of lines like
2012-01-09 14:16:34.174 IST [7810] 4f0aa96a.1e82/1 12/583 0 ERROR: syntax error at or near "%" at character 28
2012-01-09 14:16:34.175 IST [7810] 4f0aa96a.1e82/2 12/583 0 STATEMENT: COMMIT; BEGIN; TRUNCATE foo%s; COMMIT;
but nothing else that seems relevant.
UPDATE 1: Thanks to lbolla and his insightful analysis, I was able to file a Python bug report about this.
See sbt's analysis in that report, and also here. See also the Python bug report Fix exception pickling. So, following sbt's explanation, we can reproduce the original error with
import sqlalchemy.exc
e = sqlalchemy.exc.ProgrammingError("", {}, None)
type(e)(*e.args)
which gives
Traceback (most recent call last):
File "<stdin>", line 9, in <module>
TypeError: __init__() takes at least 4 arguments (2 given)
UPDATE 2: This has been fixed, at least for SQLAlchemy, by Mike Bayer, see the bug report StatementError Exceptions un-pickable.. Per Mike's suggestion, I also reported a similar bug to psycopg2, though I didn't (and don't) have an actual example of breakage. Regardless, they have apparently fixed it, though they gave no details of the fix. See psycopg exceptions cannot be pickled. For good measure, I also reported a Python bug ConfigParser exceptions are not pickleable corresponding to the SO question lbolla mentioned. It seems they want a test for this.
Anyway, this looks like it will continue to be a problem in the foreseeable future, since, by and large, Python developers don't seem to be aware of this issue and so don't guard against it. Surprisingly, it seems that there are not enough people using multiprocessing for this to be a well known issue, or maybe they just put up with it. I hope the Python developers get around to fixing it at least for Python 3, because it is annoying.
I accepted lbolla's answer, as without his explanation of how the problem was related to exception handling, I would likely have gone nowhere in understanding this. I also want to thank sbt, who explained that Python not being able to pickle exceptions was the problem. I'm very grateful to both of them, and please vote their answers up. Thanks.
UPDATE 3: I posted a followup question: Catching unpickleable exceptions and re-raising.

I believe the TypeError comes from multiprocessing's get.
I've stripped out all the DB code from your script. Take a look at this:
import multiprocessing
import sqlalchemy.exc
def do(kwargs):
i = kwargs['i']
print i
raise sqlalchemy.exc.ProgrammingError("", {}, None)
return i
pool = multiprocessing.Pool(processes=5) # start 4 worker processes
results = []
arglist = []
for i in range(10):
arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append) # evaluate "f(10)" asynchronously
# Use get or wait?
# r.get()
r.wait()
pool.close()
pool.join()
print results
Using r.wait returns the result expected, but using r.get raises TypeError. As describe in python's docs, use r.wait after a map_async.
Edit: I have to amend my previous answer. I now believe the TypeError comes from SQLAlchemy. I've amended my script to reproduce the error.
Edit 2: It looks like the problem is that multiprocessing.pool does not play well if any worker raises an Exception whose constructor requires a parameter (see also here).
I've amended my script to highlight this.
import multiprocessing
class BadExc(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
self.a = a
class GoodExc(Exception):
def __init__(self, a=None):
'''Optional param in the constructor.'''
self.a = a
def do(kwargs):
i = kwargs['i']
print i
raise BadExc('a')
# raise GoodExc('a')
return i
pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
# set a timeout in order to be able to catch C-c
r.get(1e100)
except KeyboardInterrupt:
pass
print results
In your case, given that your code raises an SQLAlchemy exception, the only solution I can think of is to catch all the exceptions in the do function and re-raise a normal Exception instead. Something like this:
import multiprocessing
class BadExc(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
self.a = a
def do(kwargs):
try:
i = kwargs['i']
print i
raise BadExc('a')
return i
except Exception as e:
raise Exception(repr(e))
pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
# set a timeout in order to be able to catch C-c
r.get(1e100)
except KeyboardInterrupt:
pass
print results
Edit 3: so, it seems to be a bug with Python, but proper exceptions in SQLAlchemy would workaround it: hence, I've raised the issue with SQLAlchemy, too.
As a workaround the problem, I think the solution at the end of Edit 2 would do (wrapping callbacks in try-except and re-raise).

The TypeError: ('__init__() takes at least 4 arguments (2 given) error isn't related to the sql you're trying to execute, it has to do with how you're using SqlAlchemy's API.
The trouble is that you're trying to call execute on the session class rather than an instance of that session.
Try this:
session = Session()
session.execute("COMMIT; BEGIN; TRUNCATE foo%s; COMMIT;")
session.commit()
From the docs:
It is intended that the sessionmaker() function be called within the
global scope of an application, and the returned class be made
available to the rest of the application as the single class used to
instantiate sessions.
So Session = sessionmaker() returns a new session class and session = Session() returns an instance of that class which you can then call execute on.

I don't know about the cause of the original exception. However, multiprocessing's problems with "bad" exceptions is really down to how pickling works. I think the sqlachemy exception class is broken.
If an exception class has an __init__() method which does not call BaseException.__init__() (directly or indirectly) then self.args probably will not be set properly. BaseException.__reduce__() (which is used by the pickle protocol) assumes that a copy of an exception e can be recreated by just doing
type(e)(*e.args)
For example
>>> e = ValueError("bad value")
>>> e
ValueError('bad value',)
>>> type(e)(*e.args)
ValueError('bad value',)
If this invariant does not hold then pickling/unpickling will fail. So instances of
class BadExc(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
self.a = a
can be pickled, but the result cannot be unpickled:
>>> from cPickle import loads, dumps
>>> class BadExc(Exception):
... def __init__(self, a):
... '''Non-optional param in the constructor.'''
... self.a = a
...
>>> loads(dumps(BadExc(1)))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.BadExc'>, ())
But instances of
class GoodExc1(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
Exception.__init__(self, a)
self.a = a
or
class GoodExc2(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
self.args = (a,)
self.a = a
can be successfully pickled/unpickled.
So you should ask the developers of sqlalchemy to fix their exception classes. In the mean time you can probably use copy_reg.pickle() to override BaseException.__reduce__() for the troublesome classes.

(This is in answer to Faheem Mitha's question in a comment about how to use copy_reg to work around the broken exception classes.)
The __init__() methods of SQLAlchemy's exception classes seem to call their base class's __init__() methods, but with different arguments. This mucks up pickling.
To customise the pickling of sqlalchemy's exception classes you can use copy_reg to register your own reduce functions for those classes.
A reduce function takes an argument obj and returns a pair (callable_obj, args) such that a copy of obj can be created by doing callable_obj(*args). For example
class StatementError(SQLAlchemyError):
def __init__(self, message, statement, params, orig):
SQLAlchemyError.__init__(self, message)
self.statement = statement
self.params = params
self.orig = orig
...
can be "fixed" by doing
import copy_reg, sqlalchemy.exc
def reduce_StatementError(e):
message = e.args[0]
args = (message, e.statement, e.params, e.orig)
return (type(e), args)
copy_reg.pickle(sqlalchemy.exc.StatementError, reduce_StatementError)
There are several other classes in sqlalchemy.exc which need to be fixed similarly. But hopefully you get the idea.
On second thoughts, rather than fixing each class individually, you can probably just monkey patch the __reduce__() method of the base exception class:
import sqlalchemy.exc
def rebuild_exc(cls, args, dic):
e = Exception.__new__(cls)
e.args = args
e.__dict__.update(dic)
return e
def __reduce__(e):
return (rebuild_exc, (type(e), e.args, e.__dict__))
sqlalchemy.exc.SQLAlchemyError.__reduce__ = __reduce__