Develop Reference

Int to Float conversion Python to C++

I have a function written in Python that works perfectly for what I need(it wasn't written by me).
I need to convert it to C++ so that it provides the same outcome. I know that it saves that float into 16-bit texture, so I am guessing this is converting 32-bit int into 16-bit float. All I need to to is to make it work in C++. Here is the python function:
def packTextureBits(index):
index = int(index)
index = index +1024
sigh=index&0x8000
sigh=sigh<<16
exptest=index&0x7fff
if exptest==0:
exp=0
else:
exp=index>>10
exp=exp&0x1f
exp=exp-15
exp=exp+127
exp=exp<<23
mant=index&0x3ff
mant=mant<<13
index=sigh|exp|mant
cp = pointer(c_int(index))
fp = cast(cp, POINTER(c_float))
return fp.contents.value
This was my approach in C++, but it returns completely screwed up values:
float PackIntToFloat(int value)
{
value += 1024;
int sign = (value & 0x8000) << 16;
int exp = value & 0x7fff;
if(exp != 0)
{
exp = value >> 10;
exp = exp & 0x1f;
exp = exp - 15 + 127;
exp = exp << 23;
}
int mant = (value & 0x3fff) << 13;
value = sign | exp | mant;
int* cp = new int(value);
float* fp = reinterpret_cast<float*>(cp);
return *fp;
// Also tried return (float)value; but returns other weird values.
}

So I owe you apologize guys. I was being stupid, not doing enough tests before posting here. My C++ solution is 100% working. I tested separate colors of the texture, and as it turned out, I assigned values to the texture the wrong way. I tried pushing floats into the texture, and it was 16 bit texture. I needed to convert these floats into half-precision floats after this conversion above, and then it started working. Texture flag called PF_FloatRGBA led me into believing that floats were the right thing to assign there, and they werent.
I still need to learn a lot. Thanks for all your help!

Freeing memory when using ctypes

I am using ctypes to try and speed up my code.
My problem is similar to the one in this tutorial : https://cvstuff.wordpress.com/2014/11/27/wraping-c-code-with-python-ctypes-memory-and-pointers/
As pointed out in the tutorial I should free the memory after using the C function. Here is my C code
//C functions
double* getStuff(double *R_list, int items){
double results[items];
double* results_p;
for(int i = 0; i < items; i++){
res = calculation ; \\do some calculation
results[i] = res; }
results_p = results;
printf("C allocated address %p \n", results_p);
return results_p; }
void free_mem(double *a){
printf("freeing address: %p\n", a);
free(a); }
Which I compile with gcc -shared -Wl,-lgsl,-soname, simps -o libsimps.so -fPIC simps.c
And python:
//Python
from ctypes import *
import numpy as np
mydll = CDLL("libsimps.so")
mydll.getStuff.restype = POINTER(c_double)
mydll.getStuff.argtypes = [POINTER(c_double),c_int]
mydll.free_mem.restype = None
mydll.free_mem.argtypes = [POINTER(c_double)]
R = np.logspace(np.log10(0.011),1, 100, dtype = float) #input
tracers = c_int(len(R))
R_c = R.ctypes.data_as(POINTER(c_double))
for_list = mydll.getStuff(R_c,tracers)
print 'Python allocated', hex(for_list)
for_list_py = np.array(np.fromiter(for_list, dtype=np.float64, count=len(R)))
mydll.free_mem(for_list)
Up to the last line the code does what I want it to and the for_list_py values are correct. However, when I try to free the memory, I get a Segmentation fault and on closer inspection the address associated with for_list --> hex(for_list) is different to the one allocated to results_p within C part of the code.
As pointed out in this question, Python ctypes: how to free memory? Getting invalid pointer error , for_list will return the same address if mydll.getStuff.restype is set to c_void_p. But then I struggle to put the actual values I want into for_list_py. This is what I've tried:
cast(for_list, POINTER(c_double) )
for_list_py = np.array(np.fromiter(for_list, dtype=np.float64, count=len(R)))
mydll.free_mem(for_list)
where the cast operation seems to change for_list into an integer. I'm fairly new to C and very confused. Do I need to free that chunk of memory? If so, how do I do that whilst also keeping the output in a numpy array? Thanks!
Edit: It appears that the address allocated in C and the one I'm trying to free are the same, though I still recieve a Segmentation fault.
C allocated address 0x7ffe559a3960
freeing address: 0x7ffe559a3960
Segmentation fault
If I do print for_list I get <__main__.LP_c_double object at 0x7fe2fc93ab00>
Conclusion
Just to let everyone know, I've struggled with c_types for a bit.
I've ended up opting for SWIG instead of c_types. I've found that the code runs faster on the whole (compared to the version presented here). I found this documentation on dealing with memory deallocation in SWIG very useful https://scipy-cookbook.readthedocs.io/items/SWIG_Memory_Deallocation.html as well as the fact that SWIG gives you a very easy way of dealing with numpy n-dimensional arrays.

After getStuff function exits, the memory allocated to results array is not available any more, so when you try to free it, it crashes the program.
Try this instead:
double* getStuff(double *R_list, int items)
{
double* results_p = malloc(sizeof((*results_p) * (items + 1));
if (results_p == NULL)
{
// handle error
}
for(int i = 0; i < items; i++)
{
res = calculation ; \\do some calculation
results_p[i] = res;
}
printf("C allocated address %p \n", results_p);
return results_p;
}

OpenCV+python: HoughLines accumulator access since 3.4.2

In OpenCV 3.4.2 the option to return the number of votes (accumulator value) for each line returned by HoughLines() was added. In python this seems to be supported as well as read in the python docstring of my OpenCV installation:
"Each line is represented by a 2 or 3 element vector (ρ, θ) or (ρ, θ, votes) ."
It is also included in the docs (with some broken formatting).
However I can find no way to return the 3 element option (ρ, θ, votes) in python.
Here is code demonstrating the problem:
import numpy as np
import cv2
print('OpenCV should be at least 3.4.2 to test: ', cv2.__version__)
image = np.eye(10, dtype='uint8')
lines = cv2.HoughLines(image, 1, np.pi/180, 5)
print('(number of lines, 1, output vector dimension): ', lines.shape)
print(lines)
outputs
OpenCV should be at least 3.4.2 to test: 3.4.2
(number of lines, 1, output vector dimension): (3, 1, 2)
[[[ 0. 2.3212879]]
[[ 1. 2.2340214]]
[[-1. 2.4609141]]]
The desired behavior is an extra column with the amount of votes each line received. With the vote values more advanced options than the standard thresholding can be applied, as such it has been often requested and asked about on SE (here, here, here and here) with sometimes the equivalent for HoughCircles(). However both the questions and answers (such as modifying source and recompiling) are from before it was added officially, and therefore do not apply to the current situation.

As of vanilla OpenCV 3.4.3, you can't use this functionality from Python.
How it Works in C++
First of all in the implementation of HoughLines, we can see code that selects the type of the output array lines:
int type = CV_32FC2;
if (lines.fixedType())
{
type = lines.type();
CV_CheckType(type, type == CV_32FC2 || type == CV_32FC3, "Wrong type of output lines");
}
We can then see this parameter used in implementation of HoughLinesStandard when populating lines:
if (type == CV_32FC2)
{
_lines.at<Vec2f>(i) = Vec2f(line.rho, line.angle);
}
else
{
CV_DbgAssert(type == CV_32FC3);
_lines.at<Vec3f>(i) = Vec3f(line.rho, line.angle, (float)accum[idx]);
}
Similar code can be seen in HoughLinesSDiv.
Based on this, we need to pass in an _OutputArray that is fixed type, and stores 32bit floats in 3 channels. How to make a fixed type (but not fixed size, since the algorithm needs to be able to resize it) _OutputArray? Let's look at the implementation again:
A generic cv::Mat is not fixed type, neither is cv::UMat
One option is std::vector<cv::Vec3f>
Another option is cv::Mat3f (that's a cv::Matx<_Tp, m, n>)
Sample Code:
#include <opencv2/opencv.hpp>
int main()
{
cv::Mat image(cv::Mat::eye(10, 10, CV_8UC1) * 255);
cv::Mat2f lines2;
cv::HoughLines(image, lines2, 1, CV_PI / 180, 4); // runs the actual detection
std::cout << lines2 << "\n";
cv::Mat3f lines3;;
cv::HoughLines(image, lines3, 1, CV_PI / 180, 4); // runs the actual detection
std::cout << lines3 << "\n";
return 0;
}
Console Output:
[0, 2.3212879;
1, 2.2340214;
-1, 2.4609141]
[0, 2.3212879, 10;
1, 2.2340214, 6;
-1, 2.4609141, 6]
How the Python Wrapper Works
Let's look at the autogenerated code wrapping the HoughLines function:
static PyObject* pyopencv_cv_HoughLines(PyObject* , PyObject* args, PyObject* kw)
{
using namespace cv;
{
PyObject* pyobj_image = NULL;
Mat image;
PyObject* pyobj_lines = NULL;
Mat lines;
double rho=0;
double theta=0;
int threshold=0;
double srn=0;
double stn=0;
double min_theta=0;
double max_theta=CV_PI;
const char* keywords[] = { "image", "rho", "theta", "threshold", "lines", "srn", "stn", "min_theta", "max_theta", NULL };
if( PyArg_ParseTupleAndKeywords(args, kw, "Oddi|Odddd:HoughLines", (char**)keywords, &pyobj_image, &rho, &theta, &threshold, &pyobj_lines, &srn, &stn, &min_theta, &max_theta) &&
pyopencv_to(pyobj_image, image, ArgInfo("image", 0)) &&
pyopencv_to(pyobj_lines, lines, ArgInfo("lines", 1)) )
{
ERRWRAP2(cv::HoughLines(image, lines, rho, theta, threshold, srn, stn, min_theta, max_theta));
return pyopencv_from(lines);
}
}
PyErr_Clear();
// Similar snippet handling UMat...
return NULL;
}
To summarize this, it tries to convert the object passed in the lines parameter to a cv::Mat, and then it calls cv::HoughLines with the cv::Mat as the output parameter. (If this fails, then it tries the same thing with cv::UMat) Unfortunately, this means that there is no way to give cv::HoughLines a fixed type lines, so as of 3.4.3 this functionality is inaccessible from Python.
Solutions
The only solutions, as far as I can see, involve modifying the OpenCV source code, and rebuilding.
Quick Hack
This is trivial, edit the implementation of cv::HoughLines and change the default type to be CV_32FC3:
int type = CV_32FC3;
However this means that you will always get the votes (which also means that the OpenCL optimization, if present, won't get used).
Better Patch
Add an optional boolean parameter return_votes with default value false. Modify the code such that when return_votes is true, the type is forced to CV_32FC3.
Header:
CV_EXPORTS_W void HoughLines( InputArray image, OutputArray lines,
double rho, double theta, int threshold,
double srn = 0, double stn = 0,
double min_theta = 0, double max_theta = CV_PI,
bool return_votes = false );
Implementation:
void HoughLines( InputArray _image, OutputArray lines,
double rho, double theta, int threshold,
double srn, double stn, double min_theta, double max_theta,
bool return_votes )
{
CV_INSTRUMENT_REGION()
int type = CV_32FC2;
if (return_votes)
{
type = CV_32FC3;
}
else if (lines.fixedType())
{
type = lines.type();
CV_CheckType(type, type == CV_32FC2 || type == CV_32FC3, "Wrong type of output lines");
}
// the rest...

There is a new python binding (opencv 4.5.1)
doc : cv.HoughLinesWithAccumulator

Passing C++ double array to Python Results in a Crash

I'm running into an issue while trying to pass a double array from C++ to Python. I run a script to create a binary file with data, then read that data back into an array and am trying to pass the array to Python. I've followed advice here: how to return array from c function to python using ctypes among other pages I have found through google. I can write a generic example that works fine (like a similar array to the link above), but when I try to pass the array read from a binary file (code below), the program crashes with "Unhandled exception at ADDR (ucrtbase.dll) in python.exe: An invalid parameter was passed to a function that considers invalid parameters fatal." So, I'm wondering if anyone has any insight.
A word on methodology:
Right now, I'm just trying to learn - that's why I'm going through the convoluted process of saving to disk, loading, and passing to Python. Eventaully, I will use this in scientific simulations where the data read from disk needs to be generated by distributed computing/a super computer. I would like to use Python for its ease of plotting (matplotlib) and C++ for its speed (iterative calculations, etc).
So, on to my code. This generates the binary file:
for (int zzz = 0; zzz < arraysize; ++zzz)
{
for (int yyy = 0; yyy < arraysize; ++yyy)
{
for (int xxx = 0; xxx < arraysize; ++xxx)
{//totalBatP returns a 3 element std::vector<double> - dblArray3_t is basically that with a few overloaded operators (+,-,etc)
dblArray3_t BatP = B.totalBatP({ -5 + xxx * stepsize, -5 + yyy * stepsize, -5 + zzz * stepsize }, 37);
for (int bbb = 0; bbb < 3; ++bbb)
{
dataarray[loopind] = BatP[bbb];
++loopind;
...(end braces here)
FILE* binfile;
binfile = fopen("MBdata.bin", "wb");
fwrite(dataarray, 8, 3 * arraysize * arraysize * arraysize, binfile);
The code that reads the file:
DLLEXPORT double* readDblBin(const std::string filename, unsigned int numOfDblsToRead)
{
char* buffer = new char[numOfDblsToRead];
std::ifstream binFile;
binFile.open(filename, std::ios::in | std::ios::binary);
binFile.read(buffer, numOfDblsToRead);
double* dataArray = (double*)buffer;
binFile.close();
return dataArray;
}
And the Python Code that receives the array:
def readBDataWrapper(filename, numDblsToRead):
fileIO = ctypes.CDLL('./fileIO.dll')
fileIO.readDblBin.argtypes = (ctypes.c_char_p, ctypes.c_uint)
fileIO.readDblBin.restype = ctypes.POINTER(ctypes.c_double)
return fileIO.readDblBin(filename, numDblsToRead)

One possible problem is here
char* buffer = new char[numOfDblsToRead];
Here you allocate numOfDblsToRead bytes. You probably want numOfDblsToRead * sizeof(double).
Same with the reading from the file, you only read numOfDblsToRead bytes.

I figured it out - at least it appears to be working. The problem was with the binary files that were generated with the first code block. I swapped the c-style writing with ofstream. My assumption is perhaps I was using the code to write to disk wrong somehow. Anyway, it appears to work now.
Replaced:
FILE* binfile;
binfile = fopen("MBdata.bin", "wb");
fwrite(dataarray, 8, 3 * arraysize * arraysize * arraysize, binfile);
With:
std::ofstream binfile;
binfile.open("MBdata.bin", std::ios::binary | std::ios::out);
binfile.write(reinterpret_cast<const char*>(dataarray), std::streamsize(totaliter * sizeof(double)));
binfile.close();

How do I debug code that segfaults unless run through gdb?

That's a single threaded code.
In particular: ahocorasick Python extension module (easy_install ahocorasick).
I isolated the problem to a trivial example:
import ahocorasick
t = ahocorasick.KeywordTree()
t.add("a")
When I run it in gdb, all is fine, same happens when I enter these instructions into Python CLI. However, when I try to run the script regularily, I get a segfault.
To make it even weirder, the line that causes segfault (identified by core dump analysis) is a regular int incrementation (see the bottom of the function body).
I'm completely stuck by this moment, what can I do?
int
aho_corasick_addstring(aho_corasick_t *in, unsigned char *string, size_t n)
{
aho_corasick_t* g = in;
aho_corasick_state_t *state,*s = NULL;
int j = 0;
state = g->zerostate;
// As long as we have transitions follow them
while( j != n &&
(s = aho_corasick_goto_get(state,*(string+j))) != FAIL )
{
state = s;
++j;
}
if ( j == n ) {
/* dyoo: added so that if a keyword ends up in a prefix
of another, we still mark that as a match.*/
aho_corasick_output(s) = j;
return 0;
}
while( j != n )
{
// Create new state
if ( (s = xalloc(sizeof(aho_corasick_state_t))) == NULL )
return -1;
s->id = g->newstate++;
debug(printf("allocating state %d\n", s->id)); /* debug */
s->depth = state->depth + 1;
/* FIXME: check the error return value of
aho_corasick_goto_initialize. */
aho_corasick_goto_initialize(s);
// Create transition
aho_corasick_goto_set(state,*(string+j), s);
debug(printf("%u -> %c -> %u\n",state->id,*(string+j),s->id));
state = s;
aho_corasick_output(s) = 0;
aho_corasick_fail(s) = NULL;
++j; // <--- HERE!
}
aho_corasick_output(s) = n;
return 0;
}

There are other tools you can use that will find faults that does not necessarily crash the program.
valgrind, electric fence, purify, coverity, and lint-like tools may be able to help you.
You might need to build your own python in some cases for this to be usable. Also, for memory corruption things, there is (or was, haven't built exetensions in a while) a possibility to let python use direct memory allocation instead of pythons own.

Have you tried translating that while loop to a for loop? Maybe there's some subtle misunderstanding with the ++j that will disappear if you use something more intuitive.