Develop Reference

"Obtaining file position failed" while using ndarray.tofile() and numpy.fromfile() with FIFOs

I am writing some dummy code to learn how FIFOs work in python, (and later use them in my ongoing projects). When I am trying to write, or read from it, I am getting the "OSError: obtaining file position failed" message.
I am trying to transport complex datas between two python codes. I am using FIFOs, because I will need more different channels, to communicate between running modules. I am running them with the bash script, that you can see below.
#first.py
import numpy as np
data = np.complex64([1, 2, 3])
fifo = open("fifoka", "wb")
data.tofile(fifo)
fifo.flush()
fifo.close()
#second.py
import numpy as np
fifo = open("fifoka", "rb")
data = np.fromfile(fifo, dtype=np.complex64)
fifo.close()
print(data)
#!/bin/bash
mkfifo fifoka
python3 first.py | \
python3 second.py
rm fifoka
If I use fifo.write(data.tobytes()) instead of the data.tofile(fifo), then it is working fine, but according to the spec it should work the same way.
I have the same problem when I try to read from the same fifo, so I think I am doing the same mistake.
So my question is, how I should use the np.fromfile() and the ndarray.tofile() correctly in this case.

Seems to be a defect in numpy. I just ran into this moving some code from Py2.7, numpy version unknown, to Py3.8.2, numpy 1.17.4. Changing arr.tofile(fout) to fout.write(arr.tobytes()) made it work.
Better solution : if the array you want to write is contiguous, use fout.write(memoryview(arr)), which avoids a copy. The utility below will use memoryview if possible, otherwise tobytes.
def arr_tofile(a,outfile):
if a.flags['C_CONTIGUOUS']:
b = memoryview(a)
else:
b = a.tobytes()
outfile.write(b)
I've been using arr.tofile(f) via pipe to send data to other processes under 2.x for over a decade, so it's a new defect, and maybe specific to python 3.x ?
It's true that the docs say that tofile bypasses the write method and uses the fd - but there's no reason to seek or tell in order to perform tofile, no reason it should not work on a pipe.
UPDATE - it doesn't work in python3 because of a compatibility hack
For python2 operation, the procedure in tofile was to obtain FILE* from the python file object; then to flush that file, obtain the underlying file number, and then write to that using the PyArray_ToFile function.
Apparently this was broken by Python3, due to internal buffering added in the file object.
So now, it works by calling npy_PyFile_Dup2 to make a FILE* from a python file object, Then, the write is done to the fd from that FILE*, using the function PyArray_ToFile. Finally, npy_PyFile_DupClose2(file, fd, orig_pos) is used to close the new file and seek the original one to the same position. For python2, the npy_PyFile_Dup2 and npy_PyFile_DupClose2 are defined as basically stubs; a single underlying FILE* is used as described above.
In Python3 they do a lot more. I didn't go through it all, but npy_PyFile_Dup2 actually calls os.dup using python mechanisms to make a new file handle, and, after the write, npy_PyFile_DupClose2 does a 'tell' on the new file, followed by a 'seek' on the original python file to skip over the data just written on the other handle.
Bottom line - this is a bit of a mess - under python3, it's better to avoid using arr.tofile(file), even when it works, unless the size of the write is large enough to justify fairly substantial amount of overhead. And it won't work on a non-seekable file at all. Use file.write(memoryview(arr)) or file.write(arr.tobytes()) instead, in both cases.
Obvious Next Question - can this be fixed to work on pipes?
Maybe - it would rely on being able to detect that the output is a pipe, and in that case to flush the python file object and proceed to write to its file handle (as in the python 2 approach). When writing to a pipe there's no need to support a subsequent 'tell' on the output file.
https://github.com/numpy/numpy/blob/2f70544179e24b0ebc0263111f36e6450bbccf94/doc/source/release/1.8.1-notes.rst
Deprecations
C-API
The utility function npy_PyFile_Dup and npy_PyFile_DupClose are broken by the internal buffering python 3 applies to its file objects. To fix this two new functions npy_PyFile_Dup2 and npy_PyFile_DupClose2 are declared in npy_3kcompat.h and the old functions are deprecated. Due to the fragile nature of these functions it is recommended to instead use the python API when possible.
https://github.com/numpy/numpy/blob/382758355998951cea2b9f6ad1fb83e7dc4c3a02/numpy/core/src/multiarray/methods.c
PyObject *file = (from param)
FILE *fd;
npy_off_t orig_pos = 0;
fd = npy_PyFile_Dup2(file, "wb", &orig_pos);
if (fd == NULL) {
goto fail;
}
if (PyArray_ToFile(self, fd, sep, format) < 0) {
goto fail;
}
if (npy_PyFile_DupClose2(file, fd, orig_pos) < 0) {
goto fail;
}
if (own && npy_PyFile_CloseFile(file) < 0) {
goto fail;
}
Py_DECREF(file);
Py_RETURN_NONE;
https://github.com/numpy/numpy/blob/64fb290a8cb8fa9201f18015f3de1186e950a137/numpy/core/include/numpy/npy_3kcompat.h
/*
* Get a FILE* handle to the file represented by the Python object
*/
static NPY_INLINE FILE*
npy_PyFile_Dup2(PyObject *file, char *mode, npy_off_t *orig_pos)
{
int fd, fd2, unbuf;
PyObject *ret, *os, *io, *io_raw;
npy_off_t pos;
FILE *handle;
/* For Python 2 PyFileObject, use PyFile_AsFile */
#if !defined(NPY_PY3K)
if (PyFile_Check(file)) {
return PyFile_AsFile(file);
}
#endif
/* Flush first to ensure things end up in the file in the correct order */
[[[... continue to call os.dup via python interface....]]]
}
...
/*
* Close the dup-ed file handle, and seek the Python one to the current position
*/
static NPY_INLINE int
npy_PyFile_DupClose2(PyObject *file, FILE* handle, npy_off_t orig_pos)
{
int fd, unbuf;
PyObject *ret, *io, *io_raw;
npy_off_t position;
/* For Python 2 PyFileObject, do nothing */
#if !defined(NPY_PY3K)
if (PyFile_Check(file)) {
return 0;
}
#endif
position = npy_ftell(handle);
/* Close the FILE* handle */
fclose(handle);
....[[more]]...

I think this is due to how numpy.ndarray.tofile works.
from the docs
When fid is a file object, array contents are directly written to the
file, bypassing the file object’s write method. As a result, tofile
cannot be used with files objects supporting compression (e.g.,
GzipFile) or file-like objects that do not support fileno() (e.g.,
BytesIO).

It is not possible to seek in a FIFO.
On a system-call level, lseek(2) is also used to get the file position. So that doesn't work either.

Stop embedded python

I'm embedding python in a C++ plug-in. The plug-in calls a python algorithm dozens of times during each session, each time sending the algorithm different data. So far so good
But now I have a problem:
The algorithm takes sometimes minutes to solve and to return a solution, and during that time often the conditions change making that solution irrelevant. So, what I want is to stop the running of the algorithm at any moment, and run it immediately after with other set of data.
Here's the C++ code for embedding python that I have so far:
void py_embed (void*data){
counter_thread=false;
PyObject *pName, *pModule, *pDict, *pFunc;
//To inform the interpreter about paths to Python run-time libraries
Py_SetProgramName(arg->argv[0]);
if(!gil_init){
gil_init=1;
PyEval_InitThreads();
PyEval_SaveThread();
}
PyGILState_STATE gstate = PyGILState_Ensure();
// Build the name object
pName = PyString_FromString(arg->argv[1]);
if( !pName ){
textfile3<<"Can't build the object "<<endl;
}
// Load the module object
pModule = PyImport_Import(pName);
if( !pModule ){
textfile3<<"Can't import the module "<<endl;
}
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
if( !pDict ){
textfile3<<"Can't get the dict"<<endl;
}
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, arg->argv[2]);
if( !pFunc || !PyCallable_Check(pFunc) ){
textfile3<<"Can't get the function"<<endl;
}
/*Call the algorithm and treat the data that is returned from it
...
...
*/
// Clean up
Py_XDECREF(pArgs2);
Py_XDECREF(pValue2);
Py_DECREF(pModule);
Py_DECREF(pName);
PyGILState_Release(gstate);
counter_thread=true;
_endthread();
};
Edit: The python's algorithm is not my work and I shouldn't change it

This is based off of a cursory knowledge of python, and reading the python docs quickly.
PyThreadState_SetAsyncExc lets you inject an exception into a running python thread.
Run your python interpreter in some thread. In another thread, PyGILState_STATE then PyThreadState_SetAsyncExc into the main thread. (This may require some precursor work to teach the python interpreter about the 2nd thread).
Unless the python code you are running is full of "catch alls", this should cause it to terminate execution.
You can also look into the code to create python sub-interpreters, which would let you start up a new script while the old one shuts down.
Py_AddPendingCall is also tempting to use, but there are enough warnings around it maybe not.

Sorry, but your choices are short. You can either change the python code (ok, plugin - not an option) or run it on another PROCESS (with some nice ipc between). Then you can use the system api to wipe it out.

So, I finally thought of a solution (more of a workaround really).
Instead of terminating the thread that is running the algorithm - let's call it T1 -, I create another one -T2 - with the set of data that is relevant at that time.
In every thread i do this:
thread_counter+=1; //global variable
int thisthread=thread_counter;
and after the solution from python is given I just verify which is the most "recent", the one from T1 or from T2:
if(thisthread==thread_counter){
/*save the solution and treat it */
}
Is terms of computer effort this is not the best solution obviously, but it serves my purposes.
Thank you for the help guys

I've been thinking about this problem, and I agree that sub interpreters may provide you one possible solution https://docs.python.org/2/c-api/init.html#sub-interpreter-support. It supports calls for creating new interpreters and ending existing ones. The bugs & caveats sections describes some issues that depending on your architecture may or may not present a problem.
Another possible solution is to use the python multiprocessing module, and within your worker thread test a global variable (something like time_to_die). Then from the parent, you grab the GIL, set the variable, release the GIL and wait for the child to finish.
But then another idea ocurred to me. Why not just use fork(), init your python interpreter in the child and when the parent decides it's time for the python thread to end, just kill it. Something like this:
void process() {
int pid = fork();
if (pid) {
// in parent
sleep(60);
kill(pid, 9);
}
else{
// in child
Py_Initialize();
PyRun_SimpleString("# insert long running python calculation");
}
}
(This example assumes *nix, if you're on windows, substitute CreateProcess()/TerminateProcess())

Python C Module - Malloc fails in specific version of Python

I'm writing a Python module to perform IO on a O_DIRECT context. One of the limitations of O_DIRECT is you must read into a buffer aligned on a 4096 byte boundary for 2.4 and 2.5 kernels, and 2.6 and up will accept any multiple of 512.
The obvious memory allocation candidate for this is posix_memalign(void **memptr, size_t alignment, size_t size)
In my code, I allocate an area like so:
char *buffer = NULL;
int mem_ret = posix_memalign((void**)&buffer, alignment, size);
if (!buffer) {
PyErr_NoMemory();
return NULL;
}
/* I do some stuff here */
free(buffer);
When I compile and import the module with python3.2, this (and the rest of the unshown module) work fine.
When I attempt the same with python2.7 (I'd like to preserve compatibility) it throws the PyErr_NoMemory exception, and mem_ret == ENOMEM, indicating it was unable to allocate.
Why would the version of Python I compile against affect how posix_memalign operates?
OS: Ubuntu 12.04 LTS
Compiler: Clang + GCC Show same behaviour
UPDATE
I now have a working piece of code, thanks to user694733
However the fact that it works has me even more confused:
#if PY_MAJOR_VERSION >= 3
char *buffer = NULL;
int mem_ret = posix_memalign((void**)&buffer, alignment, count);
#else
void *mem = NULL;
int mem_ret = posix_memalign(&mem, alignment, count);
char *buffer = (char*)mem;
#endif
Can anyone explain why the incorrect first block works under Python3, but not 2.7, and more importantly why the correct second block does not work under Python3?
UPDATE 2
The plot thickens, having settled on the correct form of the code below, I tested on 4 different version of Python.
void *mem = NULL;
int mem_ret = posix_memalign(&mem, alignment, count);
char *buffer = (char*)mem;
if (!buffer) {
PyErr_NoMemory();
return NULL;
}
/* Do stuff with buffer */
free(buffer);
Under Python 2.7: This code operates as expected.
Under Python 3.1: This code operates as expected.
Under Python 3.2: This code generates mem_ret == ENOMEM and returns NULL for buffer
Under Python 3.3: This code operates as expected.
The Python versions not included in the Ubuntu repositories were installed from the PPA at https://launchpad.net/~fkrull/+archive/deadsnakes
If the version tagged Python binaries are to be believed, the versions I have installed are:
python2.7
python3.1
python3.2mu (--with-pymalloc --with-wide-unicode)
python3.3m (--with-pymalloc)
Could the use of the wide-unicode flag in the default Python3 distribution be causing this error? If so, how is this happening?
For clarity, the ENOMEM failure to allocate will occur with any variant of malloc(), even something as simple as malloc(512).

For a quick work-around, stick to mmap instead of malloc+memalign

posix_memalign may not be the same body of code in one compilation environment as another. You could easily imagine that Python 3 would use different feature test macros to Python 2. That could mean it ends up running different code.
You might have a look at the symbols that are used... often times the output of ldd or nm will have mangled names that indicate what version is actually being used.
Additionally, what does an strace show of the allocation system call? I find that's a good way of seeing if the arguments passed in are incorrect, which can be a reason for getting ENOMEM.

How to send array of bytes from C++ to python with boost python and return it modified?

I am creating small GUI system and I would like to make my rendering with python and cairo and pystacia libraries. For C++/Python interaction I am using Boost Python but I am having troubles with pointers.
I have seen this kind of question asked few times but didn't quite understand how to solve it.
If I have a strcut/class with only pointer for image data:
struct ImageData{
unsigned char* data;
void setData(unsigned char* data) { this->data = data; } // lets assume there is more code here that manages memory
unsigned char* getData() { return data; }
}
how can I make this available for python to do this (C++):
ImageData myimage;
myimage.data = some_image_data;
global["myimage"] = python::ptr(&myimage);
and in python:
import mymodule
from mymodule import ImageData
myimagedata = myimage.GetData()
#manipulate with image data and those manipulations can be read from C++ data pointer that is passed
My code works for calling basic method calling of passed ptr to class. This is probably basic use case but I haven't been able to make it work. I tried shared_ptr but failed. Should it be solved using shared_ptr, proxy class or some other way?

You have a problem here with the locality of your variable: myimage will get deleted when you get out of its scope. To fix this, you can create it with dynamic memory allocation and moving then moving the pointer to python:
ImageData * myimage = new ImageData();
myimage->data = new ImageRawData(some_image_data); // assuming copy of the buffer
global["myimage"] = boost::python::ptr(myimage);
Please notice that this does not take care of memory handling on python. You shall use boost::python::handle<> to correctly state you are transferring memory management to python.

How can I handle IPC between C and Python?

I have a an application with two processes, one in C and one in Python. The C process is where all the heavy lifting is done, while the Python process handles the user interface.
The C program writes to a large-ish buffer 4 times per second, and the Python process reads this data. To this point the communication to the Python process has been done by AMQP. I would much rather setup some for of memory sharing between the two processes to reduce overhead and increase performance.
What are my options here? Ideally I would simply have the Python process read the physical memory straight (preferable from memory and not from disk), and then taking care of race conditions with Semaphores or something similar. This is however something I have little experience with, so I'd appreciate any help I can get.
I am using Linux btw.

This question has been asked for a long time. I believe the questioner already has the answer, so I wrote this answer for people later coming.
/*C code*/
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#define GETEKYDIR ("/tmp")
#define PROJECTID (2333)
#define SHMSIZE (1024)
void err_exit(char *buf) {
fprintf(stderr, "%s\n", buf);
exit(1);
}
int
main(int argc, char **argv)
{
key_t key = ftok(GETEKYDIR, PROJECTID);
if ( key < 0 )
err_exit("ftok error");
int shmid;
shmid = shmget(key, SHMSIZE, IPC_CREAT | IPC_EXCL | 0664);
if ( shmid == -1 ) {
if ( errno == EEXIST ) {
printf("shared memeory already exist\n");
shmid = shmget(key ,0, 0);
printf("reference shmid = %d\n", shmid);
} else {
perror("errno");
err_exit("shmget error");
}
}
char *addr;
/* Do not to specific the address to attach
* and attach for read & write*/
if ( (addr = shmat(shmid, 0, 0) ) == (void*)-1) {
if (shmctl(shmid, IPC_RMID, NULL) == -1)
err_exit("shmctl error");
else {
printf("Attach shared memory failed\n");
printf("remove shared memory identifier successful\n");
}
err_exit("shmat error");
}
strcpy( addr, "Shared memory test\n" );
printf("Enter to exit");
getchar();
if ( shmdt(addr) < 0)
err_exit("shmdt error");
if (shmctl(shmid, IPC_RMID, NULL) == -1)
err_exit("shmctl error");
else {
printf("Finally\n");
printf("remove shared memory identifier successful\n");
}
return 0;
}
#python
# Install sysv_ipc module firstly if you don't have this
import sysv_ipc as ipc
def main():
path = "/tmp"
key = ipc.ftok(path, 2333)
shm = ipc.SharedMemory(key, 0, 0)
#I found if we do not attach ourselves
#it will attach as ReadOnly.
shm.attach(0,0)
buf = shm.read(19)
print(buf)
shm.detach()
pass
if __name__ == '__main__':
main()
The C program need to be executed firstly and do not just stop it before python code executed, it will create the shared memory segment and write something into it. Then Python code attach the same segment and read data from it.
After done the all things, press enter key to stop C program and remove shared memory ID.
We can see more about SharedMemory for python in here:
http://semanchuk.com/philip/sysv_ipc/#shared_memory

Suggestion #1:
The simplest way should be using TCP. You mentioned your data size is large. Unless your data size is too huge, you should be fine using TCP. Ensure you make separate threads in C and Python for transmitting/receiving data over TCP.
Suggestion #2:
Python supports wrappers over C. One popular wrapper is ctypes - http://docs.python.org/2/library/ctypes.html
Assuming you are familiar with IPC between two C programs through shared-memory, you can write a C-wrapper for your python program which reads data from the shared memory.
Also check the following diccussion which talks about IPC between python and C++:
Simple IPC between C++ and Python (cross platform)

How about writing the weight-lifting code as a library in C and then providing a Python module as wrapper around it? That is actually a pretty usual approach, in particular it allows prototyping and profiling in Python and then moving the performance-critical parts to C.
If you really have a reason to need two processes, there is an XMLRPC package in Python that should facilitate such IPC tasks. In any case, use an existing framework instead of inventing your own IPC, unless you can really prove that performance requires it.