I try to log all the output of a program written in Python and C. However, printing from Python causes IOError: [Errno 9] Bad file descriptor
Please, does anyone know what the problem is and how to fix it?
PS: It's on Windows XP, Python 2.6 and MinGW GCC
#include <windows.h>
#include <fcntl.h>
#include "Python.h"
int main()
{
int fds[2];
_pipe(fds, 1024, O_BINARY);
_dup2(fds[1], 1);
setvbuf(stdout, NULL, _IONBF, 0);
/* alternative version: */
// HANDLE hReadPipe, hWritePipe;
// int fd;
// DWORD nr;
// CreatePipe(&hReadPipe, &hWritePipe, NULL, 0);
// fd = _open_osfhandle((intptr_t)hWritePipe, _O_BINARY);
// _dup2(fd, 1);
// setvbuf(stdout, NULL, _IONBF, 0);
write(1, "write\n", 6);
printf("printf\n");
Py_Initialize();
PyRun_SimpleString("print 'print'"); // this breaks
Py_Finalize();
char buffer[1024];
fprintf(stderr, "buffer size: %d\n", read(fds[0], buffer, 1024)); // should always be more than 0
/* alternative version: */
// CloseHandle(hWritePipe);
// char buffer[1024];
// ReadFile(hReadPipe, buffer, 1024, &nr, NULL);
// fprintf(stderr, "buffer size: %d\n", nr); // should always be more than 0
}
I think it could be to do with different C runtimes. I know you can't pass file descriptors between different C runtimes - Python is built with MSVC (you will need to check which version) - so you could try to make MinGW build against the same C runtime - I think there is options to do this in MinGW like -lmsvcrt80 (or whichever is the appropriate versions) but for licensing reasons they can't distribute the libraries so you will have to find them on your system. Sorry I don't have any more details on that for now, but hopefully its a start for some googling.
A simpler way would be to just do it all in Python... just make a class which exposes a write and perhaps flush method and assign it to sys.stdout. Eg for a file you can just pass an open file object - it's probably straightfoward to do a similar thing for your pipe. Then just import it and sys and set sys.stdout in a PyRun_SimpleString.
Related
I am writing a Python extension module in C (actually C++, but this doesn't matter) that performs some calculations in an OpenMP loop, in which it can call a user-provided Python callback function, which operates on numpy arrays. Since the standard CPython does not allow one to use Python API from multiple threads simultaneously, I protect these callbacks by a #pragma omp critical block.
This works well in some cases, but sometimes creates a deadlock, whereby one thread is trying to acquire the openmp critical lock, and the other is waiting for the GIL lock:
Thread 0:
__kmpc_critical_with_hint (in libomp.dylib) + 1109 [0x105f8a6dd]
__kmp_acquire_queuing_lock(kmp_user_lock*, int) (in libomp.dylib) + 9 [0x105fbaec1]
int __kmp_acquire_queuing_lock_timed_template<false>(kmp_queuing_lock*, int) (in libomp.dylib) + 405 [0x105fb6e6c]
__kmp_wait_yield_4 (in libomp.dylib) + 135,128,... [0x105fb0d0e,0x105fb0d07,...]
Thread 1:
PyObject_Call (in Python) + 99 [0x106014202]
??? (in umath.so) load address 0x106657000 + 0x25e51 [0x10667ce51]
??? (in umath.so) load address 0x106657000 + 0x23b0c [0x10667ab0c]
??? (in umath.so) load address 0x106657000 + 0x2117e [0x10667817e]
??? (in umath.so) load address 0x106657000 + 0x21238 [0x106678238]
PyGILState_Ensure (in Python) + 93 [0x1060ab4a7]
PyEval_RestoreThread (in Python) + 62 [0x10608cb0a]
PyThread_acquire_lock (in Python) + 101 [0x1060bc1a4]
_pthread_cond_wait (in libsystem_pthread.dylib) + 767 [0x7fff97d0d728]
__psynch_cvwait (in libsystem_kernel.dylib) + 10 [0x7fff9d464db6]
Curiously, this happens whenever the Python callback function encounters an invalid floating-point value or overflows, printing a warning message to the console. Apparently this upsets some synchronization mutexes and leads to a deadlock shortly after.
Here is a stripped-down but self-contained example code.
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <Python.h>
#include <numpy/arrayobject.h>
#include <omp.h>
const int NUM_THREADS = 2; // number of OpenMP threads
const int NUM_BLOCKS = 100; // number of loop iterations
//const double MAX_EXP = 500.0; // this is a safe value - exp(500) does not overflow
const double MAX_EXP = 1000.0; // exp(1000) overflows and produces a warning message, which then hangs the whole thing
int main(int argc, char *argv[])
{
Py_Initialize();
PyEval_InitThreads();
PyObject* numpy = PyImport_ImportModule("numpy");
if(!numpy) {
printf("Failed to import numpy\n");
return 1;
} else printf("numpy imported\n");
import_array1(1);
PyObject* fnc = PyObject_GetAttrString(numpy, "exp");
if(!fnc || !PyCallable_Check(fnc)) {
printf("Failed to get hold on function\n");
return 1;
} else printf("function loaded\n");
omp_set_num_threads(NUM_THREADS);
#pragma omp parallel for schedule(dynamic)
for(int i=0; i<NUM_BLOCKS; i++) {
int tn = omp_get_thread_num();
printf("Thread %i: block %i\n", tn, i);
#pragma omp critical
{
//PyGILState_STATE state = PyGILState_Ensure(); ///< does not help
npy_intp dims[1] = { random() % 64000 + 1000 };
PyArrayObject* args = (PyArrayObject*) PyArray_ZEROS(1, dims, NPY_DOUBLE, 0);
double* raw_data = (double*) PyArray_DATA(args);
for(npy_intp k=0; k<dims[0]; k++)
raw_data[k] = random()*MAX_EXP / RAND_MAX;
printf("Thread %i: calling fnc for block %i with %li points\n", tn, i, dims[0]);
PyObject* result = PyObject_CallFunctionObjArgs(fnc, args, NULL);
Py_DECREF(args);
printf("Thread %i: result[0] for block %i with %li points is %g\n", tn, i, dims[0],
*((double*)PyArray_GETPTR1((PyArrayObject*)result, 0)));
Py_XDECREF(result);
//PyGILState_Release(state);
}
}
Py_Finalize();
}
When I set MAX_EXP=500 in line 7, everything works without warnings and deadlocks, but if I replace it with MAX_EXP=1000, this produces a warning message,
sys:1: RuntimeWarning: overflow encountered in exp
and the next loop iteration never finishes. This behaviour is seen on both Linux and MacOS, Python 2.7 or 3.6 all the same. I tried to add some PyGILState_Ensure() to the code but this doesn't help, and the documentation on these aspects is unclear.
Okay, so the problem turned out to lurk deep inside numpy, namely in the _error_handler() function, which is called whenever an invalid floating-point value is produces (NaN, overflow to infinity, etc.)
This function has several regimes - from ignoring the error completely to raising an exception, but by default it issues a Python warning. In doing so, it temporarily re-acquires GIL, which was released during bulk computation, and that's where the deadlock occurs.
A very similar situation leading to the same problem is discussed here: https://github.com/numpy/numpy/issues/5856
My workaround solution was to create a lock-type class that disables numpy warnings during the existence of this class instance (which is created during the parallelized computation), and restores the original settings once this instance is destroyed. While not ideal, this seems to suffice in my case, though my feeling that ultimately the culprit is numpy itself. For completeness, here is the code of this class:
/** Lock-type class that temporarily disables warnings that numpy produces on floating-point
overflows or other invalid values.
The reason for doing this is that such warnings involve subtle interference with GIL
when executed in a multi-threading context, leading to deadlocks if a user-defined Python
function is accessed from multiple threads (even after being protected by an OpenMP critical
section). The instance of this class is created (and hence warnings are suppressed)
whenever a user-defined callback function is instantiated, and the previous warning settings
are restored once such a function is deallocated.
*/
class NumpyWarningsDisabler {
PyObject *seterr, *prevSettings; ///< pointer to the function and its previous settings
public:
NumpyWarningsDisabler() : seterr(NULL), prevSettings(NULL)
{
PyObject* numpy = PyImport_AddModule("numpy");
if(!numpy) return;
seterr = PyObject_GetAttrString(numpy, "seterr");
if(!seterr) return;
// store the dictionary corresponding to current settings of numpy warnings subsystem
prevSettings = PyObject_CallFunction(seterr, const_cast<char*>("s"), "ignore");
if(!prevSettings) { printf("Failed to suppress numpy warnings\n"); }
/*else { printf("Ignoring numpy warnings\n"); }*/
}
~NumpyWarningsDisabler()
{
if(!seterr || !prevSettings) return;
// restore the previous settings of numpy warnings subsystem
PyObject* args = PyTuple_New(0);
PyObject* result = PyObject_Call(seterr, args, prevSettings);
Py_DECREF(args);
if(!result) { printf("Failed to restore numpy warnings\n"); }
/*else printf("Restored numpy warnings\n");*/
Py_XDECREF(result);
}
};
I like to forbid my C program certain rights, permissions or capabilities, e.g. to open any files (other than stdin, stdout, stderr) or any sockets, ideally even if run as root. The reason is, that the program embeds a Python interpreter and might run untrusted code. Simplified version:
int main(int argc, char** argv)
{
/* TODO: drop all rights/permissions/capabilites
to open files or sockets here! */
Py_Initialize();
PyRun_SimpleString(argv[1]);
Py_Finalize();
}
This has to work with Python 2.6 on Linux 3.2. Any ideas?
Maybe I found the answer on my own. Comments highly desired!
I'm trying to use the seccomp library to disallow all, but certain syscalls.
It seems to work, i.e. in my naïve tests I can read from stdin, write to stdout, but cannot open files via Python.
#include <stdio.h>
#include <seccomp.h>
#include <python2.7/Python.h>
#define ERR_EXIT(err) do { \
fprintf(stderr, "%s near line %d\n", strerror(-err), __LINE__); \
exit(-1); } while (0);
int main(int argc, char** argv)
{
int i;
scmp_filter_ctx ctx;
int err;
Py_Initialize();
/* return illegal calls with error */
if (!(ctx = seccomp_init(SCMP_ACT_ERRNO(1)))) {
ERR_EXIT(1);
}
/* allow write, but only to stdout */
if ((err = seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write),
1, SCMP_A0(SCMP_CMP_EQ, STDOUT_FILENO)))) {
ERR_EXIT(err);
}
/* allow read, but only from stdin */
if ((err = seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read),
1, SCMP_A0(SCMP_CMP_EQ, STDIN_FILENO)))) {
ERR_EXIT(err);
}
/* brk, exit, exit_group, and rt_sigaction are needed by Python */
if ((err = seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0))) {
ERR_EXIT(err);
}
if ((err = seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit), 0))) {
ERR_EXIT(err);
}
if ((err = seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0))) {
ERR_EXIT(err);
}
if ((err = seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigaction), 0))) {
ERR_EXIT(err);
}
if ((err = seccomp_load(ctx))) {
ERR_EXIT(err);
}
for (i = 1; i < argc; i++) {
PyRun_SimpleString(argv[i]);
}
Py_Finalize();
return 0;
}
I very much appreciate any critique on this approach, thanks!
Maybe have a look at this or search for "restricted Python" elsewhere. You might be able to wrap the untrusted code such that it runs in a restricted environment.
You can use SELinux or AppArmor to restrict rights of an application on Linux.
In Unix/Linux, you can limit many resources using the command limit from the command line.
% limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 10240 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
vmemoryuse unlimited
descriptors 4096
memorylocked 64 kbytes
maxproc 1024
Thus, you can limit the number file descriptors open (which both limits the number of sockets you can open and the number of files; unfortunately, it doesn't distinguish between the two types offile descriptors), or how many processes may be forked (maxproc), or how big are files you can create.
I believe that limits are inherited, so if you start a shell with certain limits restricted, all the limits you change will be inherited by any new processes invoked. I.e., you could limit what a user can do by forcing him into an environment where these limits are set.
This may not quite be what you are looking for, but it is a way to do some coarse limiting from the command line.
Just for completeness: One can certainly use systemd-nspawn, but my target system still runs upstart. At some later point I will certainly look into this solution, maybe combined with seccomp and setrlimit.
I have a C++ and a Python application on Windows (7+). I wish to send a ~50KB array of binary data (int[], float[], or double[]) from the C++ application to a NumPy array in the Python application in real time. I want <100ms latency, but can handle up to 500ms. I'm unsure of the correct way to do this.
I believe NumPy technically stores its arrays as just an array of binary data just like C++ (assuming a reasonable C++ compiler, like modern MSVC or GCC). Therefore technically it should be very easy, but I haven't been able to identify a good way to do this.
My current plan would be to use a memory mapped file, and then handle locking the memory-mapped file with more traditional IPC such as the Win32 message pump or a semaphore.
I'm however not sure whether NumPy can read straight from a memory mapped file. It can create a memory-map to a file on disk with numpy.memmap, but this doesn't seem to work for pure memory mapped file where I just have a name or a handle.
I don't know if this is the right approach. Maybe I can get it to work, but ideally I would also want to do it the right way and not be surprised by nasty consequences of me coding stuff I don't understand.
I would appreciate any help or pointers to material that might help me figure out the correct way to do this.
UPDATE:
My C++ code (proof-of-concept) would look like this:
// Host application.
// Creates 20 byte memory mapped file with name "Global\test_mmap_file" and
// containing 5 uint32s.
// Note: Requires SeCreateGlobalPrivilege to create global memory mapped
#include <windows.h>
#include <iostream>
#include <cassert>
file
int main()
{
HANDLE file_mapping_handle = NULL;
unsigned int* buffer = 0;
assert(sizeof(unsigned int) == 4); // Require compatability with np.uint32
const size_t buffer_sz = 5 * sizeof(unsigned int);
file_mapping_handle = CreateFileMapping(
INVALID_HANDLE_VALUE,
NULL,
PAGE_READWRITE,
0,
buffer_sz,
L"Global\\test_mmap_file");
if (!file_mapping_handle)
{
std::cout << "CreateFileMapping failed (Host).\n";
std::cout << "Error code: 0x" << std::hex << GetLastError() << std::endl;
std::cin.get();
return 1;
}
buffer = (unsigned int*)MapViewOfFile(
file_mapping_handle,
FILE_MAP_ALL_ACCESS,
0,
0,
buffer_sz);
if (!buffer)
{
CloseHandle(file_mapping_handle);
std::cout << " MapViewOfFile failed (Host).\n";
std::cout << "Error code: 0x" << std::hex << GetLastError() << std::endl;
std::cin.get();
return 1;
}
buffer[0] = 2;
buffer[1] = 3;
buffer[2] = 5;
buffer[3] = 7;
buffer[4] = 11;
std::cout << "Data sent, press enter to exit.\n";
std::cin.get();
UnmapViewOfFile(buffer);
CloseHandle(file_mapping_handle);
return 0;
}
I wanted some way to access this shared memory from Python and create a numpy array. I tried,
import numpy as np
L_mm = np.memmap('Global\\test_mmap_file')
but this fails as Global\test_mmap_file is not a filename. Following the hints given by abarnert I constructed the following client program which seems to work:
import numpy as np
import mmap
mm = mmap.mmap(0,20,'Global\\test_mmap_file')
L = np.frombuffer(mm,dtype = np.uint32)
print (L)
mm.close()
This requires admin privileges for both programs to run (or giving the user the right to SeCreateGlobalObjects). However I think this should easily be bypassed by not giving the shared memory a global name, and instead duplicating the handle and passing it to the Python program. It also doesn't control access to the shared memory properly, but that should be easy with a semaphore or some other such construct.
I am writing a Python Extension in C (on Linux (Ubuntu 14.04)) and ran into an issue with dynamic memory allocation. I searched through SO and found several posts on free() calls causing similar errors because free() tries to free memory that is not dynamically allocated. But I don't know if/how that is a problem in the code below:
#include <Python.h>
#include <stdio.h>
#include <stdlib.h>
static PyObject* tirepy_process_data(PyObject* self, PyObject *args)
{
FILE* rawfile = NULL;
char* rawfilename = (char *) malloc(128*sizeof(char));
if(rawfilename == NULL)
printf("malloc() failed.\n");
memset(rawfilename, 0, 128);
const int num_datapts = 0; /* Just Some interger variable*/
if (!PyArg_ParseTuple(args, "si", &rawfilename, &num_datapts)) {
return NULL;
}
/* Here I am going top open the file, read the contents and close it again */
printf("Raw file name is: %s \n", rawfilename);
free(rawfilename);
return Py_BuildValue("i", num_profiles);
}
The output is:
Raw file name is: \home\location_to_file\
*** Error in `/usr/bin/python': free(): invalid pointer: 0xb7514244 ***
According to the documentation:
These formats allow to access an object as a contiguous chunk of memory. You don’t have to provide raw storage for the returned unicode or bytes area. Also, you won’t have to release any memory yourself, except with the es, es#, et and et# formats.
(Emphasis is added by me)
So you do not need to first allocate memory with malloc(). You also do not need to free() the memory afterwards.
Your error occurs because you are trying to free memory that was provided/allocated by Python. So C (malloc/free) is unable to free it, as it is unknown to the C runtime.
Please consider the API docs for `PyArg_ParseTuple': https://docs.python.org/2/c-api/arg.html
You shall NOT pass a pointer to allocated memory, nor shall you free it afterwards.
Whenever I open() a file with Python, the last access time is not updated, that's very odd :
If I open with r/rb nothing changes if I stat the file
If I open with w/r+ or a the ctime and mtime update properly but not atime
It doesn't look like it is a filesystem problem (which is ext3 in this case) because if I touch or cat the file it does update properly.
I haven't been able to find a lot of information about it; is it supposed to behave this way or is there something wrong?
Please try running mount, and see, if noatime flag is used on the mounted fs. Also, if your kernel is fresh enough, it's the "relatime" that is set by-default.
The "open()" code is pretty self-explanatory and does not mess with ATIME flags:
/* >> fileutils.c from Python 3.2.3 */
FILE*
_Py_fopen(PyObject *path, const char *mode)
{
#ifdef MS_WINDOWS
wchar_t wmode[10];
int usize;
usize = MultiByteToWideChar(CP_ACP, 0, mode, -1, wmode, sizeof(wmode));
if (usize == 0)
return NULL;
return _wfopen(PyUnicode_AS_UNICODE(path), wmode);
#else
FILE *f;
PyObject *bytes = PyUnicode_EncodeFSDefault(path);
if (bytes == NULL)
return NULL;
/* >> Plain fopen(), nothing fancy here. */
f = fopen(PyBytes_AS_STRING(bytes), mode);
Py_DECREF(bytes);
return f;
#endif
}