Due to this, I need to use python memoryview.cast('I') to access a FPGA avoiding double read/write strobe. No panic, you wont need an FPGA to answer the question below...
So here comes a python sample which fails ('testfile' can be any file here -just longer than 20 bytes-, but for me, eventually it will be a IO mapped FPGA HW):
#!/usr/bin/python
import struct
import mmap
with open('testfile', "r+b") as f:
mm=memoryview(mmap.mmap(f.fileno(), 20)).cast('I')
# now try to assign the 2 first U32 of the file new values 1 and 2
# mm[0]=1; mm[1]=2 would work, but the following fails:
mm[0:1]=memoryview(struct.pack('II',1,2)).cast('I') #assignement error
The error is:
./test.py
Traceback (most recent call last):
File "./test.py", line 8, in <module>
mm[0:1]=memoryview(struct.pack('II',1,2)).cast('I')
ValueError: memoryview assignment: lvalue and rvalue have different structures
I don't undestand the error... what "different structures" are we talking about??
How can I rewrite the right hand-side of the assignment expression so it works??
Changing the left-hand-side will fail for the FPGA... as it seems anything else generates wrong signal towards the hardware...
More generally, how should I rework my array of 32 bit integers to fit the left-hand-side of the assignment...?
Yes #Monica is right: The error was simpler than I though. The slice on the left hand side is wrong indeed.
Related
I'm working on some machine learning code and today I've lost about 6 hours because simple typo.
It was this:
numpy.empty(100,100)
instead of
numpy.empty([100,100])
As I'm not really used to numpy, so I forgot the brackets. The code happily crunched the numbers and at the end, just before saving results to disk, it crashed on that line.
Just to put things in perspective I code on remote machine in shell, so IDE is not really an option. Also I doubt IDE would catch this.
Here's what I already tried:
running pylint - well pylint kinda works. After I've disabled everything apart of errors and warnings, it even seem to be usefull. But pylint have serious issue with imported modules. As seen on official bug tracker devs know about it, but cannot/won't do anything about it. There is suggested workaround, but ignoring whole module, would not help in my case.
running pychecker - if I create code snippet with the mistake I made, the pychecker reports error - same error as python interpreter. However if I run pychecker on the actual source file (~100 LOC) it reported other errors (unused vars, unused imports, etc.); but the faulty numpy line was skipped.
At last I have tried pyflakes but it does even less checking than pychecker/pylint combo.
So is there any reliable method which can check code in advance? Without actually running it.
A language with stronger type checking would have been able to save you from this particular error, but not from errors in general. There are plenty of ways to go wrong that pass static type checking. So if you have computations that takes a long time, it makes sense to adopt the following strategies:
Test the code from end to end on small examples (that run in a few seconds or minutes) before running it on big data that will consume hours.
Structure long-running computations so that intermediate results are saved to files on disk at appropriate points in the computation. This means that when something breaks, you can fix the problem and restart the computation from the last save point.
Run the code from the interactive interpreter, so that in the event of an exception you are returned to the interactive session, giving you a chance of being able to recover the data using a post-mortem debugging session. For example, suppose I have some long-running computation:
def work(A, C):
B = scipy.linalg.inv(A) # takes a long time when A is big
return B.dot(C)
I run this from the interactive interpreter and it raises an exception:
>>> D = work(A, C)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "q22080243.py", line 6, in work
return B.dot(C)
ValueError: matrices are not aligned
Oh no! I forgot to transpose C! Do I have to do the inversion of A again? Not if I call pdb.pm:
>>> import pdb
>>> pdb.pm()
> q22080243.py(6)work()
-> return B.dot(C)
(Pdb) B
array([[-0.01129249, 0.06886091, ..., 0.08530621, -0.03698717],
[ 0.02586344, -0.04872148, ..., -0.04853373, 0.01089163],
...,
[-0.11463087, 0.15048804, ..., 0.0722889 , -0.12388141],
[-0.00467437, -0.13650975, ..., -0.13894875, 0.02823997]])
Now, unlike in Lisp, I can't just set things right and continue the execution. But at least I can recover the intermediate results:
(Pdb) D = B.dot(C.T)
(Pdb) numpy.savetxt('result.txt', D)
Do you use unit tests? There is really no better way.
I am getting an error (see below) when trying to use cv.CreateHist in Python. I
am also noticing another alarming problem. If I spit out all of the attributes
of the cv module into a file, and then I search them, I find that a ton of
common things are missing.
For example, cv.TermCriteria() is not there; cv.ConnectedComp is not there; and
cv.CvRect is not there.
Everything about my installation, with Open CV 2.2, worked just fine. I can plot
images, make CvScalars, and call plenty of the functions, like cv.CamShift...
but there are a dozen or so of these hit-or-miss functions or data structures
that are simply missing with no explanation.
Here's my code for cv.CreateHist:
import cv
q = cv.CreateHist([1],1,cv.CV_HIST_ARRAY)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: x9y��
The weird wingding stuff is actually what it spits out at the command line, not a copy-paste error. Can anyone help figure this out? It's incredibly puzzling.
Ely
As for CvRect, see the documentation. It says such types are represented as Pythonic tuples.
As for your call to CreateHist, you may be passing the arguments in wrong order. See createhist in the docs for python opencv.
I am getting the error:
Warning: invalid value encountered in log
From Python and I believe the error is thrown by numpy (using version 1.5.0). However, since I am calling the "log" function in several places, I'm not sure where the error is coming from. Is there a way to get numpy to print the line number that generated this error?
I assume the warning is caused by taking the log of a number that is small enough to be rounded to 0 or smaller (negative). Is that right? What is the usual origin of these warnings?
Putting np.seterr(invalid='raise') in your code (before the errant log call)
will cause numpy to raise an exception instead of issuing a warning.
That will give you a traceback error message and tell you the line Python was executing when the error occurred.
If you have access to the numpy source, you should be able to find the line that prints that warning (using grep, etc) and edit the corresponding file to force an error (using an assertion, for example) when an invalid value is passed. That will give you a stack trace pointing to the place in your code that called log with the improper value.
I had a brief look in my numpy source, and couldn't find anything that matches the warning you described though (my version of numpy is older than yours, though).
>>> import numpy
>>> numpy.log(0)
-inf
>>> numpy.__version__
'1.3.0'
Is it possible that you're calling some other log function that isn't in numpy? For example, here is one that actually throws an exception when given invalid input.
>>> import math
>>> math.log(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: math domain error
I'm finding it very difficult to develop with PyClips, because it appears to replace useful error messages thrown by Clips with a generic "syntax error" message. This makes debugging very laborious and practically impossible on large codebases when using PyClips.
Consider the following example. I wrote a very large expression, which contained the multiplication operator, but I mistakenly forgot to add the second argument. Instead of simply telling I was missing an argument, PyClips told me there was a syntax error. What should have taken me 1 second to correct, took me 5 minutes to correct as I hunted through my large expression, looking for the mistake.
Here's a condensed version:
In Clips, with a useful error message:
clips
CLIPS> (defrule myrule "" (myfact 123) => (bind ?prob (* (min 1 2))))
[ARGACCES4] Function * expected at least 2 argument(s)
ERROR:
(defrule MAIN::myrule ""
(myfact 123)
=>
(bind ?prob (* (min 1 2))
And in PyClips, with an unuseful error message:
python
>>> import clips
>>> clips.BuildRule('myrule','(myfact 123)','(bind ?prob (* (min 1 2)))','')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.6/dist-packages/clips/_clips_wrap.py", line 2839, in BuildRule
_c.build(construct)
_clips.ClipsError: C08: syntax error, or unable to parse expression
How can I get PyClips to give me the real error thrown by Clips?
Catch the ClipsError, then read ErrorStream for the details. For example:
engine = clips.Environment()
engine.Reset()
engine.Clear()
try:
engine.Load(os.path.abspath(rule_file))
except clips.ClipsError:
logging.error(clips.ErrorStream.Read())
I am using rss2email for converting a number of RSS feeds into mail for easier consumption. That is, I was using it because it broke in a horrible way today: On every run, it only gives me this backtrace:
Traceback (most recent call last):
File "/usr/share/rss2email/rss2email.py", line 740, in <module>
elif action == "list": list()
File "/usr/share/rss2email/rss2email.py", line 681, in list
feeds, feedfileObject = load(lock=0)
File "/usr/share/rss2email/rss2email.py", line 422, in load
feeds = pickle.load(feedfileObject)
TypeError: ("'str' object is not callable", 'sxOYAAuyzSx0WqN3BVPjE+6pgPU', ((2009, 3, 19, 1, 19, 31, 3, 78, 0), {}))
The only helpful fact that I have been able to construct from this backtrace is that the file ~/.rss2email/feeds.dat in which rss2email keeps all its configuration and runtime state is somehow broken. Apparently, rss2email reads its state and dumps it back using cPickle on every run.
I have even found the line containing that 'sxOYAAuyzSx0WqN3BVPjE+6pgPU'string mentioned above in the giant (>12MB) feeds.dat file. To my untrained eye, the dump does not appear to be truncated or otherwise damaged.
What approaches could I try in order to reconstruct the file?
The Python version is 2.5.4 on a Debian/unstable system.
EDIT
Peter Gibson and J.F. Sebastian have suggested directly loading from the
pickle file and I had tried that before. Apparently, a Feed class
that is defined in rss2email.py is needed, so here's my script:
#!/usr/bin/python
import sys
# import pickle
import cPickle as pickle
sys.path.insert(0,"/usr/share/rss2email")
from rss2email import Feed
feedfile = open("feeds.dat", 'rb')
feeds = pickle.load(feedfile)
The "plain" pickle variant produces the following traceback:
Traceback (most recent call last):
File "./r2e-rescue.py", line 8, in <module>
feeds = pickle.load(feedfile)
File "/usr/lib/python2.5/pickle.py", line 1370, in load
return Unpickler(file).load()
File "/usr/lib/python2.5/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.5/pickle.py", line 1133, in load_reduce
value = func(*args)
TypeError: 'str' object is not callable
The cPickle variant produces essentially the same thing as calling
r2e itself:
Traceback (most recent call last):
File "./r2e-rescue.py", line 10, in <module>
feeds = pickle.load(feedfile)
TypeError: ("'str' object is not callable", 'sxOYAAuyzSx0WqN3BVPjE+6pgPU', ((2009, 3, 19, 1, 19, 31, 3, 78, 0), {}))
EDIT 2
Following J.F. Sebastian's suggestion around putting "printf
debugging" into Feed.__setstate__ into my test script, these are the
last few lines before Python bails out.
u'http:/com/news.ars/post/20080924-everyone-declares-victory-in-smutfree-wireless-broadband-test.html': u'http:/com/news.ars/post/20080924-everyone-declares-victory-in-smutfree-wireless-broadband-test.html'},
'to': None,
'url': 'http://arstechnica.com/'}
Traceback (most recent call last):
File "./r2e-rescue.py", line 23, in ?
feeds = pickle.load(feedfile)
TypeError: ("'str' object is not callable", 'sxOYAAuyzSx0WqN3BVPjE+6pgPU', ((2009, 3, 19, 1, 19, 31, 3, 78, 0), {}))
The same thing happens on a Debian/etch box using python 2.4.4-2.
How I solved my problem
A Perl port of pickle.py
Following J.F. Sebastian's comment about how simple the pickle
format is, I went out to port parts of pickle.py to Perl. A couple
of quick regular expressions would have been a faster way to access my
data, but I felt that the hack value and an opportunity to learn more
about Python would be be worth it. Plus, I still feel much more
comfortable using (and debugging code in) Perl than Python.
Most of the porting effort (simple types, tuples, lists, dictionaries)
went very straightforward. Perl's and Python's different notions of
classes and objects has been the only issue so far where a bit more
than simple translation of idioms was needed. The result is a module
called Pickle::Parse which after a bit of polishing will be
published on CPAN.
A module called Python::Serialise::Pickle existed on CPAN, but I
found its parsing capabilities lacking: It spews debugging output all
over the place and doesn't seem to support classes/objects.
Parsing, transforming data, detecting actual errors in the stream
Based upon Pickle::Parse, I tried to parse the feeds.dat file.
After a few iteration of fixing trivial bugs in my parsing code, I got
an error message that was strikingly similar to pickle.py's original
object not callable error message:
Can't use string ("sxOYAAuyzSx0WqN3BVPjE+6pgPU") as a subroutine
ref while "strict refs" in use at lib/Pickle/Parse.pm line 489,
<STDIN> line 187102.
Ha! Now we're at a point where it's quite likely that the actual data
stream is broken. Plus, we get an idea where it is broken.
It turned out that the first line of the following sequence was wrong:
g7724
((I2009
I3
I19
I1
I19
I31
I3
I78
I0
t(dtRp62457
Position 7724 in the "memo" pointed to that string
"sxOYAAuyzSx0WqN3BVPjE+6pgPU". From similar records earlier in the
stream, it was clear that a time.struct_time object was needed
instead. All later records shared this wrong pointer. With a simple
search/replace operation, it was trivial to fix this.
I find it ironic that I found the source of the error by accident
through Perl's feature that tells the user its position in the input
data stream when it dies.
Conclusion
I will move away from rss2email as soon as I find time to
automatically transform its pickled configuration/state mess to
another tool's format.
pickle.py needs more meaningful error messages that tell the user
about the position of the data stream (not the poision in its own
code) where things go wrong.
Porting parts pickle.py to Perl was fun and, in the end, rewarding.
Have you tried manually loading the feeds.dat file using both cPickle and pickle? If the output differs it might hint at the error.
Something like (from your home directory):
import cPickle, pickle
f = open('.rss2email/feeds.dat', 'r')
obj1 = cPickle.load(f)
obj2 = pickle.load(f)
(you might need to open in binary mode 'rb' if rss2email doesn't pickle in ascii).
Pete
Edit: The fact that cPickle and pickle give the same error suggests that the feeds.dat file is the problem. Probably a change in the Feed class between versions of rss2email as suggested in the Ubuntu bug J.F. Sebastian links to.
Sounds like the internals of cPickle are getting tangled up. This thread (http://bytes.com/groups/python/565085-cpickle-problems) looks like it might have a clue..
'sxOYAAuyzSx0WqN3BVPjE+6pgPU' is most probably unrelated to the pickle's problem
Post an error traceback for (to determine what class defines the attribute that can't be called (the one that leads to the TypeError):
python -c "import pickle; pickle.load(open('feeds.dat'))"
EDIT:
Add the following to your code and run (redirect stderr to file then use 'tail -2' on it to print last 2 lines):
from pprint import pprint
def setstate(self, dict_):
pprint(dict_, stream=sys.stderr, depth=None)
self.__dict__.update(dict_)
Feed.__setstate__ = setstate
If the above doesn't yield an interesting output then use general troubleshooting tactics:
Confirm that 'feeds.dat' is the problem:
backup ~/.rss2email directory
install rss2email into virtualenv/pip sandbox (or use zc.buildout) to isolate the environment (make sure you are using feedparser.py from the trunk).
add couple of feeds, add feeds until 'feeds.dat' size is greater than the current. Run some tests.
try old 'feeds.dat'
try new 'feeds.dat' on existing rss2email installation
See r2e bails out with TypeError bug on Ubuntu.