Develop Reference

How to resume a terminal with subprocess python

I am working with the subprocess module in Python. I am trying to run a series of terminals to automate a process.
To break it down:
I am suppose to have 3 terminals open to run a set of commands
like so:
Terminal 1: `cd src` -> `./run_script.sh`
Terminal 2: cd data -> `python prepare_data.py`
Terminal 3: `cd src` -> `./do_something.sh` #runs some docker container
Terminal 4: `cd src` -> `./do_another.sh`
Terminal 3: `./another_bash.sh`
To automate this the following:
class AutomateProcesses:
def run_terminal_1(self):
subprocess.call('./run_script.sh', shell=True, cwd='../src')
def run_terminal_2(self):
subprocess.call('python prepare_data.py', shell=True, cwd='../../data')
def run_terminal_3(self):
subprocess.call('./do_something.sh.sh', shell=True, cwd='../src')
def run_terminal_4(self):
subprocess.call('./do_another.sh', shell=True, cwd='../src')
How do I get back to terminal 3 to run the command?

It looks like you want to run several commands on a "terminal" (actually you don't see any terminal), it is just a sub-process that runs a shell.
I use the tool called pexpect (https://pexpect.readthedocs.io/en/latest/overview.html), it has the Windows-variant wexpect (https://pypi.org/project/wexpect/).
Below is the code sample, using the child variable, you can keep the "terminal" and send commands to it.
import pexpect
# log file to capture all the commands sent to the shell and their responses
output_file = open('log.txt','wb')
# create the bash shell sub-process
child = pexpect.spawn('/bin/bash', logfile=output_file)
child.stdout = output_file
child.expect(bytes('>', 'utf-8'))
# make sure you use the pair (sendline() and expect()) to wait until the command finishes
child.sendline(bytes('ls', 'utf-8'))
child.expect(bytes('>', 'utf-8'))
child.sendline(bytes('echo Hello World', 'utf-8'))
child.expect(bytes('>', 'utf-8'))
output_file.close()

When running python script background, the job always stops

I wrote a python script, "download_vod.py" that runs a child process using subprocess module.
download_vod.py
#!/usr/bin/python
import subprocess
url = xxx
filename = xxx.mp4
cmd = "ffmpeg -i " + url + " -c:v copy -c:a copy " + "\"" + filename + "\""
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
proc.wait()
When I run foreground on a bash shell as below, it works fine and terminates properly
./download_vod.py
But a problem occurs when I run the script background as below.
./download_vod.py&
The script always stops as below.
[1]+ Stopped download_vod.py
If I resume the job as below, it resumes and terminates properly.
bg
I assume it is caused by running subprocess because it never happens without subprocess.
Would you let me know what happens to the subprocess (child process) when I run the python script as background? And how would it be fixed?

Communication of a python parent process and a python subprocess

I have only recently started working with the subprocess-module, so i am sure, this is a rookie-question:
I am trying to start a python-subprocess from a python 3.5.2. parent script and retrieve information from it:
import subprocess
process = subprocess.Popen(
'C:\\IDLEX (Python GUI).exe',
shell = True,
stdout = subprocess.PIPE,
)
while True:
lines = process.stdout.readlines()
for line in lines:
print (line)
What command do i have to give in the child-process to generate an output in the parent process?
I already tried print('something') and sys.stdout.write('something else') (coupled with sys.stdout.flush()) but nothing seems to work.

The subprocess is running and its output is already directed to the parent process. There is no output generated because of 'C:\\IDLEX (Python GUI).exe' does not flush anything to stdout.
Your script is working:
process = subprocess.Popen(
'echo Hello World', # or change to any other executable or script to test
shell = True,
stdout = subprocess.PIPE,
)
Output:
Hello World
You may try to run C:\\IDLEX (Python GUI).exe directly in the cmd to check.

getting subprocess result in variable

I am executing python script using subprocess.call() in pytho script. The script which gets executed using subproecss is a server process which send result back to calling client.
result = subprocess.call('python -m module/coref_resolution/src/coref/corenlp &', shell = True)
Is there any way to receive result from corenlp.py into result variable?

import shlex
cmd = shlex.split('your command')
output = subprocess.Popen( cmd, stdout = subprocess.PIPE).communicate()[0]

Not able to give inputs to subprocess(process which runs adb shell command) after 100 iterations

I want to run a stress test for adb(android debug bridge) shell. ( adb shell in this respect just a command line tool provided by Android phones).
I create a sub-process from python and in this subprocess i execute 'adb shell' command. there are some commands which has to be given to this subprocess which I am providing via stdin proper of the sub process.
Everything seems to be fine but when I am running a stress test. after around 100 iterations the command which I give to stdin does not reach to subprocess. If I run commands in separate terminal it is running fine. but the problem is with this stdin.
Can anyone tell me what I am doing wrong. Below is the code sample
class ADB():
def __init__(self):
self.proc = subprocess.Popen('adb shell', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True,bufsize=0)
def provideAMcommand(self, testParam):
try:
cmd1 = "am startservice -n com.test.myapp/.ADBSupport -e \"" + "command" + "\" \"" + "test" + "\""
cmd2 = " -e \"" + "param" + "\"" + " " + testParam
print cmd1+cmd2
sys.stdout.flush()
self.proc.stdin.write(cmd1 + cmd2 + "\n")
except:
raise Exception("Phone is not Connected to Desktop or ADB is not available \n")

If it works for the first few commands but blocks later then you might forgot to read from self.proc.stdout that might lead to (as the docs warn) to OS pipe buffer filling up and blocking the child process.
To discard the output, redirect it to os.devnull:
import os
from subprocess import Popen, PIPE, STDOUT
DEVNULL = open(os.devnull, 'wb')
# ...
self.proc = Popen(['adb', 'shell'], stdin=PIPE, stdout=DEVNULL, stderr=STDOUT)
# ...
self.proc.stdin.write(cmd1 + cmd2 + "\n")
self.proc.stdin.flush()
There is pexpect module that might be a better tool for a dialog-based interaction (if you want both read/write intermitently).

IN provideAMcommand you are writing to and flushing the stdout of your main process. That will not send anything to the stdin of the child process you have created with Popen. The following code creates a new bash child process, a bit like the code in your __init__:
import subprocess as sp
cproc = sp.Popen("bash", stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.PIPE, shell=True)
Now, the easiest way to communicate with that child process is the following:
#Send command 'ls' to bash.
out, err = cproc.communicate("ls")
This will send the text "ls" and EOF to bash (equal to running a bash script with only the text "ls" in it). Bash will execute the ls command and then quit. Anything that bash or ls write to stdout and stderr will end up in the variables out and err respectively. I have not used the adb shell, but I guess it behaves like bash in this regard.
If you just want your child process to print to the terminal, don't specify the stdout and stderr arguments to Popen.
You can check the exit code of the child, and raise an exception if it is non-zero (indicating an error):
if (cproc.returncode != 0):
raise Exception("Child process returned non-zero exit code")