Develop Reference

Debug Python Programmatically using C# Process

I can call a python script in C# and redirect the output/error using the following code:
using System.Diagnostics;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
// Configure new process
Process cmd = new Process();
cmd.StartInfo.FileName = "python.exe";
cmd.StartInfo.Arguments = "<PYTHON FILE>";
cmd.StartInfo.RedirectStandardInput = true;
cmd.StartInfo.RedirectStandardOutput = true;
cmd.StartInfo.RedirectStandardError = true;
cmd.StartInfo.CreateNoWindow = true;
cmd.StartInfo.UseShellExecute = false;
cmd.StartInfo.WorkingDirectory = "<PATH TO WORKING DIRECTORY>";
string stdout, stderr;
// Start process and wait for the process to exit
cmd.Start();
cmd.StandardInput.WriteLine();
cmd.StandardInput.Flush();
cmd.StandardInput.Close();
cmd.WaitForExit();
stderr = cmd.StandardError.ReadToEnd();
stdout = cmd.StandardOutput.ReadToEnd();
// Check for error
if (!string.IsNullOrEmpty(stderr))
{
Console.WriteLine(stderr);
}
else
{
Console.WriteLine(stdout);
}
}
}
}
I have been trying to debug the python file using the following:
using System.Diagnostics;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
// Configure new process
Process cmd = new Process();
cmd.StartInfo.FileName = "python.exe";
cmd.StartInfo.Arguments = "-mpdb <PYTHON FILE>";
cmd.StartInfo.RedirectStandardInput = true;
cmd.StartInfo.RedirectStandardOutput = true;
cmd.StartInfo.RedirectStandardError = true;
cmd.StartInfo.CreateNoWindow = true;
cmd.StartInfo.UseShellExecute = false;
cmd.StartInfo.WorkingDirectory = "<PATH TO WORKING DIRECTORY>";
string stdin = "", stdout, stderr;
cmd.Start();
while (true)
{
// This works the first iteration, but throws a 'Cannot write to a closed TextWriter' error on the following iteration
// But I need to close the StandardInput in order to get the output from the process (stdout/stderr)
cmd.StandardInput.WriteLine(stdin);
cmd.StandardInput.Flush();
cmd.StandardInput.Close();
cmd.WaitForExit();
stderr = cmd.StandardError.ReadToEnd();
stdout = cmd.StandardOutput.ReadToEnd();
// Check if the CMD was valid
if (!string.IsNullOrEmpty(stderr))
{
Console.WriteLine(stderr);
}
else
{
Console.WriteLine(stdout);
}
stdin = Console.ReadLine();
if (string.IsNullOrEmpty(stdin)) break;
}
cmd.WaitForExit();
}
}
}
I am sort of in a catch-22 problem where I want stdin to remain open so I can feed more commands to the python/pdb process (i.e. 'step', 'next', 'continue'...), but in order to get stdout or stderr I need to close and wait. Calling cmd.Start(); multiple times does not work because that just restarts the process using the same StartInfo.FileName and StartInfo.Arguments.
Am I going about this the wrong way or is there something trivial that I have missed in the documentation? Thanks in advance.

How to run python script from C# code by using command

I have a script file(run_edr.py) in my local machine and when I run it by using "cmd" and the following command then the script works perfectly. The script takes fewer parameters, the first parameter is an input document folder path and the second parameter is the output folder path to store the output documents.
my python command,
python run_edr.py -input_path "C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\files\EDR" -output_path "C:\Users\aslamm5165\Downloads\test" -site_name "a" -site_address "b" -site_city "c" -site_county "d" -site_state "e" -site_type "1"
I have tried like below, but not working, where Did I go wrong?
ScriptRuntimeSetup setup = Python.CreateRuntimeSetup(null);
ScriptRuntime runtime = new ScriptRuntime(setup);
ScriptEngine engine = Python.GetEngine(runtime);
ScriptSource source = engine.CreateScriptSourceFromFile(#"C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\run_edr.py");
ScriptScope scope = engine.CreateScope();
List<String> argv = new List<String>();
//Do some stuff and fill argv
argv.Add("python"+#" C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\run_edr.py -input_path" + #"C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\files\EDR");
argv.Add("-output_path"+ #"C:\Users\aslamm5165\Downloads\test");
argv.Add("-site_name 'a' -site_address 'b' -site_city 'c' -site_county 'd' -site_state 'e' -site_type '1'");
engine.GetSysModule().SetVariable("argv", argv);
source.Execute(scope);
I have tried with the system process as well as shown below, no error in the code, but the script is not getting executed. So I don't know what is the correct way of doing this but I want to start my script from my .Net Core application.
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"cmd.exe";
start.Arguments = string.Format("python run_edr.py -input_path {0} -output_path {1} -site_name 'a' -site_address 'b' -site_city 'c' -site_county 'd' -site_state 'e' -site_type '1'", #"C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\files\EDR", #"C:\Users\aslamm5165\Downloads\test");
start.UseShellExecute = true;
start.RedirectStandardOutput = false;
start.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
Process.Start(start);

I solved a similar running Python scripts in .NET Core 3.1 problem by changing the executable file from cmd.exe or /bin/bash in Linux to a batch script (Windows) or shell script (Linux) file. Here's my approach:
1, for Windows OS, create a run.bat file which include the python.exe and the %* to pass all arguments to it:
C:\YOUR_PYTHON_PATH\python.exe %*
2, for LInux OS, create a run.sh file to execute python with arguments:
#!/bin/bash
/usr/bin/python3 "$#"
3, use Process and ProcessStartInfo (your second approach):
string fileName = null;
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
fileName = "path_to_bat/run.bat"
}
else
{
fileName = "path_to_bat/run.sh"
}
ProcessStartInfo start = new ProcessStartInfo
{
FileName = fileName,
Arguments = string.Format("\"{0}\" \"{1}\"", script, args),
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true
};
using Process process = Process.Start(start);
the .NET code is same to Windows except the FileName should be the shell script's name with path.

How to send EOF using scala process utility

I want to start a python program from inside a scala program that has to receive a possibly infinitely long string. Thus it is not possible to pass it as a cmd argument.
My solution is to transmit the data via the stdstreams. However, I am unable to find the scala version of the working bash code:
bash code:
#/bin/bash
var="SOME REALLY LONG STRING THAT IS SEND TO THE PYTHON PROGRAM"
echo "$var" | ./readUntilEOF.py
scala code:
import sys.process._
object main {
def main(args : Array[String]) : Unit = {
val cmd = "./readUntilEOF.py"
val string = "SOME REALLY LONG STRING THAT IS SEND TO THE PYTHON PROGRAM"
print("I am starting to send stuff...")
val resultString = (string #| cmd.!!).!!
print(resultString)
}
}
readUntilEOF.py:
#!/usr/bin/python3
import sys
if __name__ == "__main__":
read = sys.stdin.read()
print(read)
Output running the bash command:
#> ./scalaBashEquivalent.sh
SOME REALLY LONG STRING THAT IS SEND TO THE PYTHON PROGRAM
Output running the scala code:
#> scala scala.sc
I am starting to send stuff...
/* and then it never terminates */

#< can take InputStream so try
(cmd #< new ByteArrayInputStream(string.getBytes)).!!
scastie

It is indeed a bit more complex than expected. But the below code seems to work.
import java.io.PrintWriter
object main {
def main(args : Array[String]) : Unit = {
val cmd = "./readUntilEOF.py"
val string = "SOME REALLY LONG STRING THAT IS SEND TO THE PYTHON PROGRAM"
println("I am starting to send stuff...")
val processOutput : StringBuilder = new StringBuilder()
val process = Process(cmd).run(new ProcessIO(
in => {
val writer = new PrintWriter(in)
writer.write(string)
writer.close()
},
out => {
processOutput.addAll(scala.io.Source.fromInputStream(out))
out.close()
},
_.close()
))
assert(process.exitValue() == 0)
print(processOutput.toString)
}
}

How to start and stop a python program in C++?

I am trying start another program in C++(windows).
The C++ needs to execute starter.bat --debug --verbose, in the starter.bat, it runs %SF_HOME%\files\program-start.pyc %*. After the program runs, how could I stop the program-start.pyc in C++? How to send control+c to it ? I cannot find it by using taskList in windows command line.
SHELLEXECUTEINFO processInfo;
processInfo.cbSize = sizeof(SHELLEXECUTEINFO);
processInfo.fMask = SEE_MASK_NOCLOSEPROCESS;
processInfo.hwnd = NULL;
processInfo.lpVerb = NULL;
processInfo.lpFile = L"C:\\Users\\Li\\ddddp\\bin\\scripts\\dddpstart.pyc";
processInfo.lpParameters = L"--device-type --device-serial yyyyy --device-model TEST --debug --verbose";
processInfo.lpDirectory = NULL;
processInfo.nShow = SW_MAXIMIZE;
processInfo.hInstApp = NULL;
ShellExecuteEx(&processInfo);
TerminateProcess(processInfo.hProcess, 1);
Then a console pops up but it doesn't stop at the end of the code.

NSTask requires flush when reading from a process' stdout, Terminal does not

I have a simple Python script that asks for your name, then spits it back out:
def main():
print('Enter your name: ')
for line in sys.stdin:
print 'You entered: ' + line
Pretty simple stuff! When running this in the OS X Terminal, it works great:
$ python nameTest.py
Enter your name:
Craig^D
You entered: Craig
But, when attempting to run this process via an NSTask, the stdout only appears if additional flush() calls are added to the Python script.
This is how I have my NSTask and piping configured:
NSTask *_currentTask = [[NSTask alloc] init];
_currentTask.launchPath = #"/usr/bin/python";
_currentTask.arguments = [NSArray arrayWithObject:#"nameTest.py"];
NSPipe *pipe = [[NSPipe alloc] init];
_currentTask.standardOutput = pipe;
_currentTask.standardError = pipe;
dispatch_queue_t stdout_queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0);
__block dispatch_block_t checkBlock;
checkBlock = ^{
NSData *readData = [[pipe fileHandleForReading] availableData];
NSString *consoleOutput = [[NSString alloc] initWithData:readData encoding:NSUTF8StringEncoding];
dispatch_sync(dispatch_get_main_queue(), ^{
[self.consoleView appendString:consoleOutput];
});
if ([_currentTask isRunning]) {
[NSThread sleepForTimeInterval:0.1];
checkBlock();
} else {
dispatch_sync(dispatch_get_main_queue(), ^{
NSData *readData = [[pipe fileHandleForReading] readDataToEndOfFile];
NSString *consoleOutput = [[NSString alloc] initWithData:readData encoding:NSUTF8StringEncoding];
[self.consoleView appendString:consoleOutput];
});
}
};
dispatch_async(stdout_queue, checkBlock);
[_currentTask launch];
But when running the NSTask, this is how it appears (it is initially blank, but after entering my name and pressing CTRL+D, it finishes all at once):
Craig^DEnter your name:
You entered: Craig
So, my question is: How can I read the stdout from my NSTask without requiring the additional flush() statements in my Python script? Why does the Enter your name: prompt not appear immediately when run as an NSTask?

When Python sees that its standard output is a terminal, it arranges to automatically flush sys.stdout when the script reads from sys.stdin. When you run the script using NSTask, the script's standard output is a pipe, not a terminal.
UPDATE
There is a Python-specific solution to this. You can pass the -u flag to the Python interpreter (e.g. _currentTask.arguments = #[ #"-u", #"nameTest.py"];), which tells Python not to buffer standard input, standard output, or standard error at all. You can also set PYTHONUNBUFFERED=1 in the process's environment to achieve the same effect.
ORIGINAL
A more general solution that applies to any program uses what's called a “pseudo-terminal” (or, historically, a “pseudo-teletype”), which we shorten to just “pty”. (In fact, this is what the Terminal app itself does. It is a rare Mac that has a physical terminal or teletype connected to a serial port!)
Each pty is actually a pair of virtual devices: a slave device and a master device. The bytes you write to the master, you can read from the slave, and vice versa. So these devices are more like sockets (which are bidirectional) than like pipes (which are one-directional). In addition, a pty also let you set terminal I/O flags (or “termios”) that control whether the slave echoes its input, whether it passes on its input a line at a time or a character at a time, and more.
Anyway, you can open a master/slave pair easily with the openpty function. Here's a little category that you can use to make an NSTask object use the slave side for the task's standard input and output.
NSTask+PTY.h
#interface NSTask (PTY)
- (NSFileHandle *)masterSideOfPTYOrError:(NSError **)error;
#end
NSTask+PTY.m
#import "NSTask+PTY.h"
#import <util.h>
#implementation NSTask (PTY)
- (NSFileHandle *)masterSideOfPTYOrError:(NSError *__autoreleasing *)error {
int fdMaster, fdSlave;
int rc = openpty(&fdMaster, &fdSlave, NULL, NULL, NULL);
if (rc != 0) {
if (error) {
*error = [NSError errorWithDomain:NSPOSIXErrorDomain code:errno userInfo:nil];
}
return NULL;
}
fcntl(fdMaster, F_SETFD, FD_CLOEXEC);
fcntl(fdSlave, F_SETFD, FD_CLOEXEC);
NSFileHandle *masterHandle = [[NSFileHandle alloc] initWithFileDescriptor:fdMaster closeOnDealloc:YES];
NSFileHandle *slaveHandle = [[NSFileHandle alloc] initWithFileDescriptor:fdSlave closeOnDealloc:YES];
self.standardInput = slaveHandle;
self.standardOutput = slaveHandle;
return masterHandle;
}
#end
You can use it like this:
NSTask *_currentTask = [[NSTask alloc] init];
_currentTask.launchPath = #"/usr/bin/python";
_currentTask.arguments = #[[[NSBundle mainBundle] pathForResource:#"nameTest" ofType:#"py"]];
NSError *error;
NSFileHandle *masterHandle = [_currentTask masterSideOfPTYOrError:&error];
if (!masterHandle) {
NSLog(#"error: could not set up PTY for task: %#", error);
return;
}
Then you can read from the task and write to the task using masterHandle.