I'm working on some machine learning code and today I've lost about 6 hours because simple typo.
It was this:
numpy.empty(100,100)
instead of
numpy.empty([100,100])
As I'm not really used to numpy, so I forgot the brackets. The code happily crunched the numbers and at the end, just before saving results to disk, it crashed on that line.
Just to put things in perspective I code on remote machine in shell, so IDE is not really an option. Also I doubt IDE would catch this.
Here's what I already tried:
running pylint - well pylint kinda works. After I've disabled everything apart of errors and warnings, it even seem to be usefull. But pylint have serious issue with imported modules. As seen on official bug tracker devs know about it, but cannot/won't do anything about it. There is suggested workaround, but ignoring whole module, would not help in my case.
running pychecker - if I create code snippet with the mistake I made, the pychecker reports error - same error as python interpreter. However if I run pychecker on the actual source file (~100 LOC) it reported other errors (unused vars, unused imports, etc.); but the faulty numpy line was skipped.
At last I have tried pyflakes but it does even less checking than pychecker/pylint combo.
So is there any reliable method which can check code in advance? Without actually running it.
A language with stronger type checking would have been able to save you from this particular error, but not from errors in general. There are plenty of ways to go wrong that pass static type checking. So if you have computations that takes a long time, it makes sense to adopt the following strategies:
Test the code from end to end on small examples (that run in a few seconds or minutes) before running it on big data that will consume hours.
Structure long-running computations so that intermediate results are saved to files on disk at appropriate points in the computation. This means that when something breaks, you can fix the problem and restart the computation from the last save point.
Run the code from the interactive interpreter, so that in the event of an exception you are returned to the interactive session, giving you a chance of being able to recover the data using a post-mortem debugging session. For example, suppose I have some long-running computation:
def work(A, C):
B = scipy.linalg.inv(A) # takes a long time when A is big
return B.dot(C)
I run this from the interactive interpreter and it raises an exception:
>>> D = work(A, C)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "q22080243.py", line 6, in work
return B.dot(C)
ValueError: matrices are not aligned
Oh no! I forgot to transpose C! Do I have to do the inversion of A again? Not if I call pdb.pm:
>>> import pdb
>>> pdb.pm()
> q22080243.py(6)work()
-> return B.dot(C)
(Pdb) B
array([[-0.01129249, 0.06886091, ..., 0.08530621, -0.03698717],
[ 0.02586344, -0.04872148, ..., -0.04853373, 0.01089163],
...,
[-0.11463087, 0.15048804, ..., 0.0722889 , -0.12388141],
[-0.00467437, -0.13650975, ..., -0.13894875, 0.02823997]])
Now, unlike in Lisp, I can't just set things right and continue the execution. But at least I can recover the intermediate results:
(Pdb) D = B.dot(C.T)
(Pdb) numpy.savetxt('result.txt', D)
Do you use unit tests? There is really no better way.
Related
I'm not sure if it's allowed to seek for help(if not, I don't mind not getting an answer until the competition period is over).
I was solving the Interactive Problem (Dat Bae) on CodeJam. On my local files, I can run the judge (testing_tool.py) and my program (<name>.py) separately and copy paste the I/O manually. However, I assume I need to find a way to make it automatically.
Edit: To make it clear, I want every output of x file to be input in y file and vice versa.
Some details:
I've used sys.stdout.write / sys.stdin.readline instead of print / input throughout my program
I tried running interactive_runner.py, but I don't seem to figure out how to use it.
I tried running it on their server, my program in first tab, the judge file in second. It's always throwing TLE error.
I don't seem to find any tutorial to do the same either, any help will be appreciated! :/
The usage is documented in comments inside the scripts:
interactive_runner.py
# Run this as:
# python interactive_runner.py <cmd_line_judge> -- <cmd_line_solution>
#
# For example:
# python interactive_runner.py python judge.py 0 -- ./my_binary
#
# This will run the first test set of a python judge called "judge.py" that
# receives the test set number (starting from 0) via command line parameter
# with a solution compiled into a binary called "my_binary".
testing_tool.py
# Usage: `testing_tool.py test_number`, where the argument test_number
# is 0 for Test Set 1 or 1 for Test Set 2.
So use them like this:
python interactive_runner.py python testing_tool.py 0 -- ./dat_bae.py
I`m trying to learn the programming on quantum computers.
I have installed qiskit in VS Code (all qiskit extentions available in VS Code market) , python compilator (from Vs Code market "Python" and "Python for VSCode"). I have set up my qikit API for correct working
When I run the exemple I get erros: "Instance of 'QuantumCircuit' has no 'h' member"
What shoud I do?
The code:
from qiskit import ClassicalRegister, QuantumRegister
from qiskit import QuantumCircuit, execute
q = QuantumRegister(2)
c = ClassicalRegister(2)
qc = QuantumCircuit(q)
qc.h(q[0])
qc.cx(q[0], q[1])
qc.measure(q, c)
job_sim = execute(qc, 'local_qasm_simulator')
sim_result = job_sim.result()
print(sim_result.get_counts(qc))
========================
The same error after adding comment # pylint: disable=no-member
The errors in question are coming from pylint, a linter, not from python itself. While pylint is pretty clever, some constructs (particularly those involving dynamically-added properties) are beyond its ability to understand. When you encounter situations like this, the best course of action is twofold:
Check the docs, code, etc. to make absolutely sure the code that you've written is right (i.e. verify that the linter result is a false positive)
Tell the linter that you know what you're doing and it should ignore the false positive
user2357112 took care of the first step in the comments above, demonstrating that the property gets dynamically set by another part of the library.
The second step can be accomplished for pylint by adding a comment after each of the offending lines telling it to turn of that particular check for that particular line:
qc.h(q[0]) # pylint: disable=no-member
This question already has answers here:
What causes a Python segmentation fault?
(8 answers)
Closed 3 years ago.
I'm fairly new to Python and I am having real trouble because I run into it this Segmentation fault: 11 error.
Here is a simple code example that produces this error every time:
import grequests
class Url(object):
pass
a = Url()
a.url = 'http://www.heroku.com'
a.result = 0
b = Url()
b.url = 'http://www.google.com'
b.result = 0
c = Url()
c.url = 'http://www.wordpress.com'
c.result = 0
urls = [a, b, c]
rs = (grequests.get(u.url) for i, u in enumerate(urls))
grequests.map(rs)
What is absolutely bizarre is that if I replace the urls = ... line with this:
urls = [a, b]
Then I get no error, and the script runs fine.
If I change that to just
urls = [c]
Then I also get no error, and the script runs fine.
If I change c.url = ... to
c.url = "http://yahoo.com"
And revert urls = ... back to
urls = [a, b, c]
Then I do get the segmentation fault: 11 error.
Being a memory issue sounds like a possibility though I'm not sure how to fix it.
I've been stuck on this for a number of days, so any help, no matter how small, is greatly appreciated.
For reference, I'm using macOS High Sierra (10.13.5) and installed Python 3.7.0 using Brew.
Segmentation fault (violation) is caused by an invalid memory reference. Trying to access an address that should not be accessible for current process (could also be buffer overrun or entirely bogus or uninitialized pointer). Usually it would be indicative of a bug in the underlying code or a problem during binary build (linking).
This problem lies not in your Python script, even though you may be able to trigger it by modifying your python code. Even if you for instance exhausted buffers used in a module or by the interpreter itself, it should still handle that situation gracefully.
Given your script, either gevent (dependency of grequests) or your Python (and/or bits of its standard library) are likely places where a segfault could have occurred (or a library is being used that causes it). Perhaps try rebuilding them? Where there any substantial changes around them on your system since the time you've built them? Perhaps they are trying to run against libraries other than they've been originally built against?
You can also allow your system to dump cores (I presume MacOS being essentially BSD can do that) and inspect (load it into a debugger such as gdb) the coredump to see what exactly crashed and what was going on at the time.
I'm developing a small game with pyglet. One centerpiece is, of course, drawing coloured rectangels. I initially did this by creating images in memory and blit()ing them, which worked fine. After noticing how ugly, roundabout and inefficent (yes, I profiled - ColorRect.draw() took significant time and became 10x more efficent through this change) this is, I've started creating vertex lists instead, via pyglet.graphics.Batch (I copied most of the code verbatim from one of the examples). Since then, I experience a weird exception in some low-level OpenGL code that I failed to find a cause for or reproduce reliably.
There is no apparent relation to gameplay events -- as in, nothing exceptional happens just before, or I constantly miss it. As the error occurs somewhere deep in the event loop, I cannot easily track down which position update causes it. Honestly, I'm stumped. Thus I'll braindump what I have found out and hope for some kind psychic.
I've tried it out on Windows 7 32 bit (I may get around to try it on Ubuntu 11.10 soon) with Python 3.2.2, with a pyglet revision 043180b64260 (pulled from Goggle Code and built from source, the 1.1.4 release is harder to install as it doesn't run 2to3 automatically, though it appears to be equally py3k-ready). I'll probably update to the latest mercurial version next, but it's only a few commits and the changes seem entirely unrelated.
The full traceback (censored some paths out of principle, but note it's in its own virtualenv):
Traceback (most recent call last):
File "<my main file>", line 152, in <module>
main()
File "<my main file>", line 148, in main
run()
File "<my main file>", line 125, in run
pyglet.app.run()
File "<virtualenv>\Lib\site-packages\pyglet\app\__init__.py", line 123, in run
event_loop.run()
File "<virtualenv>\Lib\site-packages\pyglet\app\base.py", line 135, in run
self._run_estimated()
File "<virtualenv>\Lib\site-packages\pyglet\app\base.py", line 164, in _run_estimated
timeout = self.idle()
File "<virtualenv>\Lib\site-packages\pyglet\app\base.py", line 278, in idle
window.switch_to()
File "<virtualenv>\Lib\site-packages\pyglet\window\win32\__init__.py", line 305, in switch_to
self.context.set_current()
File "<virtualenv>\Lib\site-packages\pyglet\gl\win32.py", line 213, in set_current
super(Win32Context, self).set_current()
File "<virtualenv>\Lib\site-packages\pyglet\gl\base.py", line 320, in set_current
buffers = (gl.GLuint * len(buffers))(*buffers)
IndexError: invalid index
Running with post-mortem (actively stepping through code until it happens used to be infeasible as the FPS went from 60 down to 7) pdb shows:
buffers is a list of ints; I have no idea what these represent or where they come from, but they are pulled from a list called self.object_space._doomed_textures (where self is an window object). The associated comment says this block of code releases texture scheduled for deletion. I don't think I explicitly use textures anywhere, but who knows what pyglet does under the hood. I assume these integers are the IDs or something of the textures to be destroyed.
gl.GLuint is an alias for ctypes.c_ulong; Thus (gl.GLuint * len(buffers))(*buffers) creates an ulong array of the same length and contents
I can evaluate the very same expression at the pdb prompt without errors or data corruption.
Independent experiments (outside the virtualenv and without importing pyglet) with ctypes shows that IndexError is raised if too many arguments are given to the array constructor. This makes no sense, both experimentation and logic suggest the length and argument count must always match.
Are there other cases where this exception may occur? May this be a bug of pyglet, or am I misusing the library and missed the associated warning?
Would the code which creates and maintains the vertex lists be of any use in debugging this? There's probably something wrong with it. I've already stared at it, but since I have little experience with pyglet.graphics, this was of limited use. Just leave a comment if you'd like to see the ColorRect code.
Any other ideas what might cause this?
It is a bit hard to provide a really relevant answer since there is no code provided but from what I can see from the error output.
buffers = (gl.GLuint * len(buffers))(*buffers)
So if I understand correctly, you are multiplying the size of an GLuint (4 bytes) with your actually buffers length (if initialized). Maybe that's why your Index is invalid, because it is too high?
Usually it would be ok since a buffer is in bytes, but you said that it is a list of ints?
Hope it helps
I am getting an error (see below) when trying to use cv.CreateHist in Python. I
am also noticing another alarming problem. If I spit out all of the attributes
of the cv module into a file, and then I search them, I find that a ton of
common things are missing.
For example, cv.TermCriteria() is not there; cv.ConnectedComp is not there; and
cv.CvRect is not there.
Everything about my installation, with Open CV 2.2, worked just fine. I can plot
images, make CvScalars, and call plenty of the functions, like cv.CamShift...
but there are a dozen or so of these hit-or-miss functions or data structures
that are simply missing with no explanation.
Here's my code for cv.CreateHist:
import cv
q = cv.CreateHist([1],1,cv.CV_HIST_ARRAY)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: x9y��
The weird wingding stuff is actually what it spits out at the command line, not a copy-paste error. Can anyone help figure this out? It's incredibly puzzling.
Ely
As for CvRect, see the documentation. It says such types are represented as Pythonic tuples.
As for your call to CreateHist, you may be passing the arguments in wrong order. See createhist in the docs for python opencv.