I'm playing with contract.py, Terrence Way's reference implementation of design-by-contract for Python. The implementation throws an exception when a contract (precondition/postcondition/invariant) is violated, but it doesn't provide you a quick way of identifying which specific contract has failed if there are multiple ones associated with a method.
For example, if I take the circbuf.py example, and violate the precondition by passing in a negative argument, like so:
circbuf(-5)
Then I get a traceback that looks like this:
Traceback (most recent call last):
File "circbuf.py", line 115, in <module>
circbuf(-5)
File "<string>", line 3, in __assert_circbuf___init___chk
File "build/bdist.macosx-10.5-i386/egg/contract.py", line 1204, in call_constructor_all
File "build/bdist.macosx-10.5-i386/egg/contract.py", line 1293, in _method_call_all
File "build/bdist.macosx-10.5-i386/egg/contract.py", line 1332, in _call_all
File "build/bdist.macosx-10.5-i386/egg/contract.py", line 1371, in _check_preconditions
contract.PreconditionViolationError: ('__main__.circbuf.__init__', 4)
My hunch is that the second argument in the PreconditionViolationError (4) refers to the line number in the circbuf.init docstring that contains the assertion:
def __init__(self, leng):
"""Construct an empty circular buffer.
pre::
leng > 0
post[self]::
self.is_empty() and len(self.buf) == leng
"""
However, it's a pain to have to open the file and count the docstring line numbers. Does anybody have a quicker solution for identifying which contract has failed?
(Note that in this example, there's a single precondition, so it's obvious, but multiple preconditions are possible).
This is an old question but I may as well answer it. I added some output, you'll see it at the comment # jlr001. Add the line below to your contract.py and when it raises an exception it will show the doc line number and the statement that triggered it. Nothing more than that, but it will at least stop you from needing to guess which condition triggered it.
def _define_checker(name, args, contract, path):
"""Define a function that does contract assertion checking.
args is a string argument declaration (ex: 'a, b, c = 1, *va, **ka')
contract is an element of the contracts list returned by parse_docstring
module is the containing module (not parent class)
Returns the newly-defined function.
pre::
isstring(name)
isstring(args)
contract[0] in _CONTRACTS
len(contract[2]) > 0
post::
isinstance(__return__, FunctionType)
__return__.__name__ == name
"""
output = StringIO()
output.write('def %s(%s):\n' % (name, args))
# ttw001... raise new exception classes
ex = _EXCEPTIONS.get(contract[0], 'ContractViolationError')
output.write('\tfrom %s import forall, exists, implies, %s\n' % \
(MODULE, ex))
loc = '.'.join([x.__name__ for x in path])
for c in contract[2]:
output.write('\tif not (')
output.write(c[0])
# jlr001: adding conidition statement to output message, easier debugging
output.write('): raise %s("%s", %u, "%s")\n' % (ex, loc, c[1], c[0]))
# ...ttw001
# ttw016: return True for superclasses to use in preconditions
output.write('\treturn True')
# ...ttw016
return _define(name, output.getvalue(), path[0])
Without modifying his code, I don't think you can, but since this is python...
If you look for where he raises the exception to the user, it I think is possible to push the info you're looking for into it... I wouldn't expect you to be able to get the trace-back to be any better though because the code is actually contained in a comment block and then processed.
The code is pretty complicated, but this might be a block to look at - maybe if you dump out some of the args you can figure out whats going on...
def _check_preconditions(a, func, va, ka):
# ttw006: correctly weaken pre-conditions...
# ab002: Avoid generating AttributeError exceptions...
if hasattr(func, '__assert_pre'):
try:
func.__assert_pre(*va, **ka)
except PreconditionViolationError, args:
# if the pre-conditions fail, *all* super-preconditions
# must fail too, otherwise
for f in a:
if f is not func and hasattr(f, '__assert_pre'):
f.__assert_pre(*va, **ka)
raise InvalidPreconditionError(args)
# rr001: raise original PreconditionViolationError, not
# inner AttributeError...
# raise
raise args
# ...rr001
# ...ab002
# ...ttw006
Related
Problem
For a Markdown document I want to filter out all sections whose header titles are not in the list to_keep. A section consists of a header and the body until the next section or the end of the document. For simplicity lets assume that the document only has level 1 headers.
When I make a simple case distinction on whether the current element has been preceeded by a header in to_keep and do either return None or return [] I get an error. That is, for pandoc --filter filter.py -o output.pdf input.md I get TypeError: panflute.dump needs input of type "panflute.Doc" but received one of type "list" (code, example file and complete error message at the end).
I use Python 3.7.4 and panflute 1.12.5 and pandoc 2.2.3.2.
Question
If make a more fine grained distinction on when to do return [], it works (function action_working). My question is, why is this more fine grained distinction neccesary? My solution seems to work, but it might well be accidental... How can I get this to work properly?
Files
error
Traceback (most recent call last):
File "filter.py", line 42, in <module>
main()
File "filter.py", line 39, in main
return run_filter(action_not_working, doc=doc)
File "C:\Users\ody_he\AppData\Local\Continuum\anaconda3\lib\site-packages\panflute\io.py", line 266, in run_filter
return run_filters([action], *args, **kwargs)
File "C:\Users\ody_he\AppData\Local\Continuum\anaconda3\lib\site-packages\panflute\io.py", line 253, in run_filters
dump(doc, output_stream=output_stream)
File "C:\Users\ody_he\AppData\Local\Continuum\anaconda3\lib\site-packages\panflute\io.py", line 132, in dump
raise TypeError(msg)
TypeError: panflute.dump needs input of type "panflute.Doc" but received one of type "list"
Error running filter filter.py:
Filter returned error status 1
input.md
# English
Some cool english text this is!
# Deutsch
Dies ist die deutsche Übersetzung!
# Sources
Some source.
# Priority
**Medium** *[Low | Medium | High]*
# Status
**Open for Discussion** *\[Draft | Open for Discussion | Final\]*
# Interested Persons (mailing list)
- Franz, Heinz, Karl
fiter.py
from panflute import *
to_keep = ['Deutsch', 'Status']
keep_current = False
def action_not_working(elem, doc):
'''For every element we check if it occurs in a section we wish to keep.
If it is, we keep it and return None (indicating to keep the element unchanged).
Otherwise we remove the element (return []).'''
global to_keep, keep_current
update_keep(elem)
if keep_current:
return None
else:
return []
def action_working(elem, doc):
global to_keep, keep_current
update_keep(elem)
if keep_current:
return None
else:
if isinstance(elem, Header):
return []
elif isinstance(elem, Para):
return []
elif isinstance(elem, BulletList):
return []
def update_keep(elem):
'''if the element is a header we update to_keep.'''
global to_keep, keep_current
if isinstance(elem, Header):
# Keep if the title of a section is in too keep
keep_current = stringify(elem) in to_keep
def main(doc=None):
return run_filter(action_not_working, doc=doc)
if __name__ == '__main__':
main()
I think what happens is that panflute call the action on all elements, including the Doc root element. If keep_current is False when walking the Doc element, it will be replaced by a list. This leads to the error message you are seeing, as panflute expectes the root node to always be there.
The updated filter only acts on Header, Para, and BulletList elements, so the Doc root node will be left untouched. You'll probably want to use something more generic like isinstance(elem, Block) instead.
An alternative approach could be to use panflute's load and dump elements directly: load the document into a Doc element, manually iterate over all blocks in args and remove all that are unwanted, then dump the resulting doc back into the output stream.
from panflute import *
to_keep = ['Deutsch', 'Status']
keep_current = False
doc = load()
for top_level_block in doc.args:
# do things, remove unwanted blocks
dump(doc)
Apologies in advance, but I am unable to post a fully working example (too much overhead in this code to distill to a runnable snippet). I will post as much explanatory detail as I can, and please do let me know if anything critical seems missing.
Running Python 2.7.5 through IDLE
I am writing a program to compare two text files. Since the files can be large (~500MB) and each row comparison is independent, I would like to implement multiprocessing to speed up the comparison. This is working pretty well, but I am getting stuck on a pseudo-random Bad file descriptor error. I am new to multiprocessing, so I guess there is a technical problem with my implementation. Can anyone point me in the right direction?
Here is the code causing the trouble (specifically the pool.map):
# openfiles
csvReaderTest = csv.reader(open(testpath, 'r'))
csvReaderProd = csv.reader(open(prodpath, 'r'))
compwriter = csv.writer(open(outpath, 'wb'))
pool = Pool()
num_chunks = 3
chunksTest = itertools.groupby(csvReaderTest, keyfunc)
chunksProd = itertools.groupby(csvReaderProd, keyfunc)
while True:
# make a list of num_chunks chunks
groupsTest = [list(chunk) for key, chunk in itertools.islice(chunksTest, num_chunks)]
groupsProd = [list(chunk) for key, chunk in itertools.islice(chunksProd, num_chunks)]
# merge the two lists (pair off comparison rows)
groups_combined = zip(groupsTest,groupsProd)
if groups_combined:
# http://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments
a_args = groups_combined # a list - set of combinations to be tested
second_arg = True
worker_result = pool.map(worker_mini_star, itertools.izip(itertools.repeat(second_arg),a_args))
Here is the full error output. (This error sometimes occurs, and other times the comparison runs to finish without problems):
Traceback (most recent call last):
File "H:/<PATH_SNIP>/python_csv_compare_multiprocessing_rev02_test2.py", line 407, in <module>
main(fileTest, fileProd, fileout, stringFields, checkFileLengths)
File "H:/<PATH_SNIP>/python_csv_compare_multiprocessing_rev02_test2.py", line 306, in main
worker_result = pool.map(worker_mini_star, itertools.izip(itertools.repeat(second_arg),a_args))
File "C:\Python27\lib\multiprocessing\pool.py", line 250, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Python27\lib\multiprocessing\pool.py", line 554, in get
raise self._value
IOError: [Errno 9] Bad file descriptor
If it helps, here are the functions called by pool.map:
def worker_mini(flag, chunk):
row_comp = []
for entry, entry2 in zip(chunk[0][0], chunk[1][0]):
if entry == entry2:
temp_comp = entry
else:
temp_comp = '%s|%s' % (entry, entry2)
row_comp.append(temp_comp)
return True, row_comp
#takes a single tuple argument and unpacks the tuple to multiple arguments
def worker_mini_star(flag_chunk):
"""Convert `f([1,2])` to `f(1,2)` call."""
return worker_mini(*flag_chunk)
def main():
I need to test a instance method that calls open. In the first test case, I set mock.mock_open to return a string, as expected. This works wonderfully.
However, I also need to test the case in which an IOError is thrown from this function. How can I make mock.mock_open raise an arbitrary exception?
This is my approach so far:
#mock.patch.object(somemodule, 'generateDefaultKey')
def test_load_privatekey(self, genkey)
mo = mock.mock_open(read_data=self.key)
mo.side_effect = IOError
with mock.patch('__main__.open', mo, create=True):
self.controller.loadPrivkey()
self.assertTrue(genkey.called, 'Key failed to regenerate')
Assign the exception to mock.mock_open.side_effect:
mock.mock_open.side_effect = IOError
From the mock.Mock.side_effect documentation:
This can either be a function to be called when the mock is called, or an exception (class or instance) to be raised.
Demo:
>>> mock = MagicMock()
>>> mock.mock_open.side_effect = IOError()
>>> mock.mock_open()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/mock.py", line 955, in __call__
return _mock_self._mock_call(*args, **kwargs)
File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/mock.py", line 1010, in _mock_call
raise effect
IOError
When using patch() as a context manager, a new mock object is produced; assign to that mock object:
with mock.patch('__main__.open', mo, create=True) as mocked_open:
mocked_open.side_effect = IOError()
self.controller.loadPrivkey()
I found Martijn's answer a little hard to generalize, so here is what I hope is a simpler answer less tied to the OPs initial code:
from unittest.mock import patch, mock_open
def func_to_test():
try:
with open('path/to/open', 'r') as file:
return 1
except IOError:
return 2
def test_open_error():
with patch("builtins.open", mock_open()) as mock_file:
mock_file.side_effect = IOError()
assert func_to_test() == 2
mock_file.assert_called_with('path/to/open', 'r')
Sorry if this question is stupid. I created an unittest class which needs to take given inputs and outputs from outside. Thus, I guess these values should be initiated. However, I met some errors in the following code:
CODE:
import unittest
from StringIO import StringIO
##########Inputs and outputs from outside#######
a=[1,2]
b=[2,3]
out=[3,4]
####################################
def func1(a,b):
return a+b
class MyTestCase(unittest.TestCase):
def __init__(self,a,b,out):
self.a=a
self.b=b
self.out=out
def testMsed(self):
for i in range(self.tot_iter):
print i
fun = func1(self.a[i],self.b[i])
value = self.out[i]
testFailureMessage = "Test of function name: %s iteration: %i expected: %i != calculated: %i" % ("func1",i,value,fun)
self.assertEqual(round(fun,3),round(value,3),testFailureMessage)
if __name__ == '__main__':
f = MyTestCase(a,b,out)
from pprint import pprint
stream = StringIO()
runner = unittest.TextTestRunner(stream=stream, verbosity=2)
result = runner.run(unittest.makeSuite(MyTestCase(a,b,out)))
print 'Tests run', result.testsRun
However, I got the following error
Traceback (most recent call last):
File "C:testing.py", line 33, in <module>
result = runner.run(unittest.makeSuite(MyTestCase(a,b,out)))
File "C:\Python27\lib\unittest\loader.py", line 310, in makeSuite
return _makeLoader(prefix, sortUsing, suiteClass).loadTestsFromTestCase(testCaseClass)
File "C:\Python27\lib\unittest\loader.py", line 50, in loadTestsFromTestCase
if issubclass(testCaseClass, suite.TestSuite):
TypeError: issubclass() arg 1 must be a class
Can anyone give me some suggestions? Thanks!
The root of the problem is this line,
result = runner.run(unittest.makeSuite(MyTestCase(a,b,out)))
unittest.makeSuite expects a class, not an instance of a class. So just MyTestCase, not MyTestCase(a, b, out). This means that you can't pass parameters to your test case in the manner you are attempting to. You should probably move the code from init to a setUp function. Either access a, b, and out as globals inside setUp or take a look at this link for information regarding passing parameters to a unit test.
By the way, here is the source file within python where the problem originated. Might be informative to read.
I'm trying to use the ghmm python module on mac osx with Python 2.7. I've managed to get everything installed, and I can import ghmm in the python environment, but there are errors when I run this (from the ghmm 'tutorial') (UnfairCasino can be found here http://ghmm.sourceforge.net/UnfairCasino.py):
from ghmm import *
from UnfairCasino import test_seq
sigma = IntegerRange(1,7)
A = [[0.9, 0.1], [0.3, 0.7]]
efair = [1.0 / 6] * 6
eloaded = [3.0 / 13, 3.0 / 13, 2.0 / 13, 2.0 / 13, 2.0 / 13, 1.0 / 13]
B = [efair, eloaded]
pi = [0.5] * 2
m = HMMFromMatrices(sigma, DiscreteDistribution(sigma), A, B, pi)
v = m.viterbi(test_seq)
Specifically I get this error:
GHMM ghmm.py:148 - sequence.c:ghmm_dseq_free(1199): Attempted m_free on NULL pointer. Bad program, BAD! No cookie for you.
python(52313,0x7fff70940cc0) malloc: * error for object 0x74706d6574744120: pointer being freed was not allocated
* set a breakpoint in malloc_error_break to debug
Abort trap
and when I set the ghmm.py logger to "DEBUG", the log prints out the following just before:
GHMM ghmm.py:2333 - HMM.viterbi() -- begin
GHMM ghmm.py:849 - EmissionSequence.asSequenceSet() -- begin >
GHMM ghmm.py:862 - EmissionSequence.asSequenceSet() -- end >
Traceback (most recent call last):
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 842, in emit
msg = self.format(record)
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 719, in format
return fmt.format(record)
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 464, in format
record.message = record.getMessage()
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 328, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file ghmm.py, line 1159
Traceback (most recent call last):
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 842, in emit
msg = self.format(record)
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 719, in format
return fmt.format(record)
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 464, in format
record.message = record.getMessage()
File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/logging/init.py", line 328, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file ghmm.py, line 949
GHMM ghmm.py:2354 - HMM.viterbi() -- end
GHMM ghmm.py:1167 - del SequenceSubSet >
So I suspect it has something to do with the way Sequences are deleted once the Viterbi function is completed, but I'm not sure if this means I need to modify the Python code, the C code, or if I need to compile ghmm and the wrappers differently. Any help/suggestions would be appreciated greatly as I have been trying to get this library to work the last 4 days.
Given the age of this question, you've probably moved onto something else, but this seemed to be the only related result I found. The issue is that a double-free is happening, due to some weirdness in how the python function 'EmissionSequence::asSequenceSet' is executed. If you look at how ghmm.py is implemented (~lines 845 - 863)
def asSequenceSet(self):
"""
#returns this EmissionSequence as a one element SequenceSet
"""
log.debug("EmissionSequence.asSequenceSet() -- begin " + repr(self.cseq))
seq = self.sequenceAllocationFunction(1)
# checking for state labels in the source C sequence struct
if self.emissionDomain.CDataType == "int" and self.cseq.state_labels is not None:
log.debug("EmissionSequence.asSequenceSet() -- found labels !")
seq.calloc_state_labels()
self.cseq.copyStateLabel(0, seq, 0)
seq.setLength(0, self.cseq.getLength(0))
seq.setSequence(0, self.cseq.getSequence(0))
seq.setWeight(0, self.cseq.getWeight(0))
log.debug("EmissionSequence.asSequenceSet() -- end " + repr(seq))
return SequenceSetSubset(self.emissionDomain, seq, self)
This should probably raise some red flags, since it seems to be reaching into the C a bit much (not that I know for sure, I haven't looked to far into it).
Anyways, if you look a little above this function, there is another function called 'sequenceSet':
def sequenceSet(self):
"""
#return a one-element SequenceSet with this sequence.
"""
# in order to copy the sequence in 'self', we first create an empty SequenceSet and then
# add 'self'
seqSet = SequenceSet(self.emissionDomain, [])
seqSet.cseq.add(self.cseq)
return seqSet
It seems that it has the same purpose, but is implemented differently. Anyways, if you replace the body of 'EmissionSequence::asSequenceSet' in ghmm.py, with just:
def asSequenceSet(self):
"""
#returns this EmissionSequence as a one element SequenceSet
"""
return self.sequenceSet();
And then rebuild/reinstall the ghmm module, the code will work without crashing, and you should be able to go on your merry way. I'm not sure if this can be submitted as a fix, since the ghmm project looks a little dead, but hopefully this is simple enough to help anyone in dire straights using this library.