I would like to serialize a dataframe to pipe it to an executable that at the moment for testing purposes just prints out whatever it is receiving from the command line.
import numpy as np
import pandas as pd
import subprocess
df = pd.DataFrame(np.random.normal(size = 1000).reshape(100,10))
exe = r"C:\Users\Snake91\source\repos\ConsoleApplication1\Debug\ConsoleApplication1.exe "
process = subprocess.Popen([exe],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr=subprocess.PIPE, shell = True)
stdout, stderr = process.communicate(np.array(df).tobytes())
Below the code of the executable
#include <iostream>
int main(int argc, char* argv[])
{
int size = sizeof(argv)/sizeof(argv[0]) + 1;
if (size > 1)
{
for (int idx = 1; idx < size; idx++)
{
std::cout << argv[idx];
}
}
}
When running the python script, however, I get an empty byte string instead of the serialized dataframe. What am I getting wrong?
As correctly pointed out in the comments, communicate does not pass the arguments to the command line, but through std::cin. The code of the executable has to be modified in the following way to make it work.
#include <string>
#include <iostream>
using namespace std;
int main(int argc, char* argv[])
{
std::string line;
while (getline(std::cin, line))
{
std::cout << line << std::endl;
}
}
Thanks for the help.
Related
I am trying to track the execution of python scripts with C++ Threads (If anyone knows a better approach, feel free to mention it)
This is the code I have so far.
#define PY_SSIZE_T_CLEAN
#include </usr/include/python3.8/Python.h>
#include <iostream>
#include <thread>
void launchScript(const char *filename){
Py_Initialize();
FILE *fd = fopen(filename, "r");
PyRun_SimpleFile(fd, filename);
PyErr_Print();
Py_Finalize();
}
int main(int argc, char const *argv[])
{
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
std::thread first (launchScript,"script.py");
std::cout << "Thread 1 is running with thread ID: " << first.get_id() << std::endl;
std::thread second (launchScript,"script2.py");
std::cout << "Thread 2 is running with thread ID: " << second.get_id() << std::endl;
first.join();
second.join();
Py_Finalize();
return 0;
}
Script.py just has a print statement that prints "Hello World"
Script2.py has a print statement that prints "Goodbye World"
I build the application with the following commands
g++ -pthread -I/usr/include/python3.8/ main.cpp -L/usr/lib/python3.8/config-3.8-x86_64 linux-gnu -lpython3.8 -o output
When I run ./output, I receive the following on my terminal
Thread 1 is running with thread ID: 140594340370176
Thread 2 is running with thread ID: 140594331977472
GoodBye World
./build.sh: line 2: 7864 Segmentation fault (core dumped) ./output
I am wondering why I am getting Segmentation Fault. I have tried to debug with PyErr_Print(); but that has not given me any clues.
Any feed back is appreciated.
After testing and debugging the program for about 20 minutes I found that the problem is caused because in your example you've created the second std::thread named second before calling join() on the first thread.
Thus, to solve this just make sure that you've used first.join() before creating the second thread as shown below:
int main(int argc, char const *argv[])
{
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
std::thread first (launchScript,"script.py");
std::cout << "Thread 1 is running with thread ID: " << first.get_id() << std::endl;
//--vvvvvvvvvvvvv-------->call join on first thread before creating the second std::thread
first.join();
std::thread second (launchScript,"script2.py");
std::cout << "Thread 2 is running with thread ID: " << second.get_id() << std::endl;
second.join();
Py_Finalize();
return 0;
}
I am trying to run a simple py file in dev c++.
/// **main file**
string script_name = "example.py"
char* script_name2 = new char[script_name.length() + 1];
strcpy(script_name2, script_name.c_str());
FILE* file_pointer;
Py_Initialize();
file_pointer = fopen(script_name2,"r");
if (file_pointer == NULL)
{
PyErr_Print();
cout << "Cannot read file -> " << script_name2 << " ... exiting" << "\n";
exit(0);
}
cout << file_pointer;
PyRun_SimpleFile(file_pointer, script_name2);
example.py
print("My Name is TUTANKHAMEN")
The file pointer is not null and prints an adress.
example.py is in the same folder containing main.cpp
Below statement runs fine and prints output on console.
Py_Initialize();
PyRun_SimpleString("import example\n");
Py_Finalize();
The script runs fine Visual Studio 2019.
But in Dev C++ :
file_pointer = _Py_fopen(script_name2, "r");
This does the trick.
This question already has answers here:
Dynamic communication between main and subprocess in Python
(2 answers)
Closed 3 years ago.
i want to write a Python script that creates a subprocess, reads from the stdout and writes to stdin. What is written to stdin should depend on whats been read from stdout.
I've tried pretty much everything i could find about subprocess.Popen, but nothing worked out.
Basically, i want to write a script that makes the following c-code print "success":
#include <stdio.h>
#include <stlib.h>
int main()
{
int var, inp;
for (int i = 0; i < 100; i++) {
var = rand();
printf("Please type %d:\n", var); //random var. is printed
scanf("%d", &inp); //inp == var?
if (inp != var) {
printf("you failed miserably\n");
return 0;
}
}
printf("success\n");
return 0;
}
I'm failing at reading from stdout while still keeping the subprocess alive. The task seems so simple, but I can't find a simple solution.
Python code that i would expect to work:
from subprocess import *
def getNum(s): # "Please type 1234567:\t" -> "1234567"
return "".join([t for t in s if t.isdigit()])
p = Popen("./io", stdin=PIPE, stdout=PIPE) #io is the binary obtained from above c code
for i in range(100):
out = p.stdout.readline() #script deadlocks here
print( out )
inp = getNum(out)+"\n"#convert out into desired inp
p.stdin.write(inp)
print (p.communicate()[0]) #kill p and get last output
This approach might be a little naive, but i also don't understand why it doesn't work.
The python program gets stuck waiting to receive something from stdout. This can be due to stdout buffering. There are probably a few ways to change this behavior, but one quick way to test this out is to change the C executable to force a fflush right after it prints out the randomly generated number. The C code would look like this:
#include <stdio.h>
#include <stdlib.h>
int main()
{
int var, inp;
for (int i = 0; i < 100; i++) {
var = rand();
printf("Please type %d:\n", var); //random var. is printed
// ADDED EXPLICIT stdout flush
fflush(stdout);
scanf("%d", &inp); //inp == var?
if (inp != var) {
printf("you failed miserably\n");
return 0;
}
}
printf("success\n");
return 0;
}
With that change, the original python script is able to drive the whole interaction.
I'm trying to establish communication between a Python program and a C program.
The Python part of this project manages the application logic and the GUI, while I have written a C program to interface with a sensor that has a manufacturer supplied C library.
Now I need an easy way of communicating between these two programs on a Windows machine. The C program will continously stream data to the Python program. The Python software should be able to send commands to the C software for changing settings etc.
What I found so far is:
the ZeroMQ library which seemed pretty promising, but I could not get it to run on windows and it seems to not be mantained anymore.
the Python subprocess module which is able to pipe the stdin and stdout of the called process, but I don't seem to get it working like I want it to..
The C code I have written now is just a dummy program that streams an output string and prints any commands that are given. The problem is, that the data from the C program does not continously stream to the output of the python program but just shows when I close the program. Also I'm not able to send a command back to the c program.
-Python Code:
import subprocess as subp
pro = subp.Popen("C:/Users/schoenhofer/Documents/Programming/Py2C/simpleIO/bin/Debug/simpleIO.exe",
stdin=subp.PIPE, stdout=subp.PIPE, bufsize=-1, universal_newlines=True)
while not pro.poll():
ln = pro.stdout.read()
if ln == '':
pro.kill()
break
else:
print(ln)
-C code:
#include <stdlib.h>
#include <stdint.h>
#include <memory.h>
#include <windows.h>
#include <conio.h>
#include <time.h>
void delay(unsigned int mseconds)
{
clock_t goal = mseconds + clock();
while (goal > clock());
}
int main()
{
TCHAR buf[20] = {0};
DWORD dwLength, dwRead;
DWORD got = 0;
uint16_t i = 0;
LPDWORD mode = 0;
HANDLE h_in = GetStdHandle(STD_INPUT_HANDLE);
if(h_in == INVALID_HANDLE_VALUE){
printf("Error getting input handle\n");
exit(EXIT_FAILURE);
}
dwLength = sizeof(buf);
SetConsoleMode(h_in, ENABLE_PROCESSED_INPUT|ENABLE_LINE_INPUT|ENABLE_EXTENDED_FLAGS);
while(1){
if(kbhit()){
if(ReadConsole(h_in, buf, dwLength, &got, NULL) == 0){
DWORD err = GetLastError();
printf("Error reading from console: %u", err);
exit(EXIT_FAILURE);
}
}
if(got > 0){
printf("Got: %s", buf);
memset(buf, 0, 20);
got = 0;
}
else{
printf("Got nothing\n");
}
delay(300);
}
return 0;
}
Any help would be greatly appreciated.
Thanks in advance, Thomas
I have the following python code, which is supposed to provide the intial input to a C++ program, then take its output and feed it back into it, until the program finishes execution:
comm.py
p = subprocess.Popen('test__1.exe', bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
p.stdin.write(x)
p.stdin.flush()
p.stdout.flush()
x = p.stdout.readline()
print(x)
if p.poll() != None:
break
I am currently testing it with two simple C++ programs:
test__1.cpp:
#include <iostream>
using namespace std;
int main()
{
for( int i = 0; i < 3; ++i )
{
int n;
cin >> n;
cout << n+1 << endl;
}
return 0;
}
test__2.cpp
#include <cstdio>
int main()
{
for( int i = 0; i < 3; ++i )
{
int n;
scanf("%d", &n);
printf("%d\n", n+1);
}
return 0;
}
When comm.py opens test__1.exe everything works fine, but when it opens test__2.exe it hangs on the first call to readline().
Note that this problem does not occur when I feed test__2.exe the whole input before execution (i.e. initially set x = '1\n2\n3\n')
What could be causing this issue?
(Also, comm.py should be able to handle any valid C++ program, so only using iostream would not be an acceptable solution.)
EDIT: I also need the solution to work on Windows.
It is caused by the fact that std::endl flushes the ostream and printf does not flush stdout,
as you can see by amending test__2.cpp as follows:
#include <cstdio>
int main()
{
for( int i = 0; i < 3; ++i )
{
int n;
scanf("%d", &n);
printf("%d\n", n+1);
fflush(stdout); //<-- add this
}
return 0;
}
You say that you want to module to work correctly with any C++ program, so
you can't rely upon it flushing the standard output (or standard error)
after every write.
That means you must cause the program's standard streams to be unbuffered
and do so externally to the program itself. You will need to do that
in comm.py.
In Linux (or other host providing GNU Core Utils), you could so by
executing the program via stdbuf, e.g.
import subprocess
cmd = ['/usr/bin/stdbuf', '-i0','-o0', '-e0', './test__2']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
p.stdin.write(x)
x = p.stdout.readline()
print(x)
if p.poll() != None:
break
which unbuffers all the standard streams. For Windows, you will need
to research how do the same thing. For the time being I don't know.