Develop Reference

How do I write to a Python subprocess?

I'm trying to write a Python script that starts a subprocess to run an Azure CLI command once the file is executed.
When I run locally, I run:
az pipelines create --name pipeline-from-cli --repository https://github.com/<org>/<project> --yml-path <path to pipeline>.yaml --folder-path _poc-area
I get prompted for an input which looks like:
Which service connection do you want to use to communicate with GitHub?
[1] Create new GitHub service connection
[2] <my connection name>
[3] <org name>
Please enter a choice [Default choice(1)]:
I can type in 2 and press enter then my pipeline is successfully created in Azure DevOps. I would like to run this command being dynamically entered when prompted.
So far I have tried:
import subprocess
cmd = 'az pipelines create --name pipeline-from-cli --repository https://github.com/<org>/<project> --yml-path <path to pipeline>.yaml --folder-path _poc-area
cmd = cmd.split()
subprocess.run(cmd, shell=True)
This will run in the exact same way as when I try to run it locally.
Try to follow answers from here I have also tried:
p = subprocess.run(cmd, input="1", capture_output=True, text=True, shell=True)
print(p)
Which gives me an error saying raise NoTTYException(error_msg)\nknack.prompting.NoTTYException.
Is there a way where I can execute this Python script, and it will run the Azure CLI command then enter 2 when prompted without any manually intervention?

You are trying to solve the wrong problem. az pipeline create takes a --service-connection parameter. You don't need to respond to the prompt, you can provide the service connection value on the command line and skip the prompt entirely.

IMHO, Daniel is right, you're not supposed to deal with stdin in your program.
Nevertheless, if you really need to, you should use pexpect package, which basically opens a process, waits for given output, and then sends input to the process' stdin.
Here's a basic example:
import pexpect
from pexpect.popen_spawn import PopenSpawn
cmd = 'az pipelines create --name pipeline-from-cli --repository https://github.com/<org>/<project> --yml-path <path to pipeline>.yaml --folder-path _poc-area'
child = pexpect.popen_spawn.PopenSpawn('cmd', timeout=1)
child.expect ('.*Please enter a choice.*')
child.sendline ('2')
# child.interact() # Give control of the child to the user.
Have a look at pexpect documentation for more details. MS Windows support is available since v4.0.
Another archaic solution would be to use subprocess the following way, emulating basically what expect would do:
import subprocess
from time import sleep
p = subprocess.Popen(azure_command, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
sleep(.5)
stdout = p.communicate(input=b'2\n')[0]
print(stdout.decode())
Still, best solution is to use non-interactive mode of most CLI programs.

Python: Keep processes started by subprocess.Popen alive after exiting

I am making a virtual assistant that can start several programs using subprocess.Popen("path/to/app.exe"). But when I exit the python program, all of processes are killed. I want the processes (the applications started with Popen) to be independent and remain alive after main process is killed.
I have tried adding start_new_session=True as argument in subprocess.Popen() as some posts have suggested, but it's still not working.
I don't think showing the code is necessary, but still, here you go.
app_path = r'C:\Users\myusername\AppData\Local\Discord\app-1.0.9001\discord.exe'
subprocess.Popen(app_path) # also tried adding start_new_session=True as argument

Since you're on Windows, you can call the start command, which exists for this very purpose: to run another program independently of the one that starts it.
The start command is provided by the command-line interpreter cmd.exe. It is not an executable: there is no start.exe. It is a "shell command" (in Linux terminology), which is why shell=True must be passed when creating the subprocess.
You won't be able to communicate with the subprocess started in this way, that is, not via the pipe mechanism provided by the subprocess module. So instead of Popen, you may just use the convenience function run:
from subprocess import run
app = 'notepad'
run(['start', app], shell=True)
The example starts the Notepad editor (instead of Discord in the question) in order to make it easier to reproduce.
In cases where the full path to the app contains spaces, we can either call start like so
app = r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe'
run(f'start "" "{app}"', shell=True)
using the Edge browser in this example, or pass the directory separately:
folder = r'C:\Program Files (x86)\Microsoft\Edge\Application'
app = 'msedge.exe'
run(['start', '/d', folder, app], shell=True)
This is needed because start treats a single argument as the window title if that argument is in quotes. And only if not does it treat it as the command. See "Can I use the start command with spaces in the path?" (on SuperUser) for more details.

Answered here: https://stackoverflow.com/a/34718600/4355695
subprocess.Popen(full_command, shell=True, close_fds=True)
(In my linux system I have to put shell=True if I'm passing a full command as string instead of a split-up array of arguments. In windows it may differ, idk)

Running a secondary script in a new terminal

When running a secondary python script:
Is it possible to run a subprocess.Popen, or subprocess.call or even execfile in a new terminal? (as in simply a different terminal than the current terminal where the script is run).
Alternatively, if before running my program (main), I open two terminals first, can I then point the secondary script to the second terminal? (so somehow getting the ID of open terminals, and then using a specific one among them, to perform the subprocess).
An example, two subprocesses to be run, first.py should be called first, only then the second is called, second.py. Because the two scripts first.py and second.py are interdependent (as in first.py goes to wait mode, until second.py is run, then first.py resumes, and I don't know how to make this communication work between them in terms of subprocesses.)
import subprocess
command = ["python", "first.py"]
command2 = ["python", "second.py"]
n = 5
for i in range(n):
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = p.stdout.readline().strip()
print output
if output == 'stop':
print 'success'
p.terminate()
p2.terminate()
break
Framework (Ubuntu, python 2.7)

I guess you want something like
subprocess.call(['xterm','-e','python',script])
Good old xterm has almost no frills; on a Freedesktop system, maybe run xdg-terminal instead. On Debian, try x-terminal-emulator.
However, making your program require X11 is in most cases a mistake. A better solution is to run the subprocesses with output to a log file (or a socket, or whatever) and then separately run tail -f on those files (in a different terminal, or from a different server over ssh, or with output to a logger which supports rsyslog, or or or ...) which keeps your program simple and modular, free from "convenience" dependencies.

If you're using tmux, you can specify which target you want the command to run in:
tmux send -t foo.0 ls ENTER
So, if you've created a tmux session foo.0, you should be able to do:
my_command = 'ls'
tmux_cmd = ['tmux', 'send', '-t', 'foo.0', my_command]
p = subprocess.Popen(tmux_cmd)

You can specify the tty of the terminal window you wish the command to be carried out in:
ls > /dev/ttys004
However, I would recommend going for the tmux approach for greater control (see my other answer).

Python: Spawn New Minimized Shell

I am attempting to to launch a python script from within another python script, but in a minimized console, then return control to the original shell.
I am able to open the required script in a new shell below, but it's not minimized:
#!/usr/bin/env python
import os
import sys
import subprocess
pyTivoPath="c:\pyTivo\pyTivo.py"
print "Testing: Open New Console"
subprocess.Popen([sys.executable, pyTivoPath], creationflags = subprocess.CREATE_NEW_CONSOLE)
print
raw_input("Press Enter to continue...")
Further, I will need to be able to later remotely KILL this shell from the original script, so I suspect I'll need to be explicit in naming the new process. Correct?
Looking for pointers, please. Thanks!
Note: python27 is mandatory for this application. Eventually will also need to work on Mac and Linux.

Do you need to have the other console open? If you now the commands to be sent, then I'd recommend using Popen.communicate(input="Shell commands") and it will automate the process for you.
So you could write something along the lines of:
# Commands to pass into subprocess (each command is separated by a newline)
commands = (
"command1\n" +
"command2\n"
)
# Your process
py_process = subprocess.Popen(*yourprocess_here*, stdin=PIPE, shell=True)
# Feed process the needed input
py_process.communicate(input=commands)
# Terminate when finished
py_process.terminate()
The code above will execute the process you specify and even send commands but it won't open a new console.

Displaying output of shell commands with shared environments

Is there any way to display the output of a shell command in Python, as the command runs?
I have the following code to send commands to a specific shell (in this case, /bin/tcsh):
import subprocess
import select
cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
poll = select.poll()
poll.register(cmd.stdout.fileno(),select.POLLIN)
# The list "commands" holds a list of shell commands
for command in commands:
cmd.stdin.write(command)
# Must include this to ensure data is passed to child process
cmd.stdin.flush()
ready = poll.poll()
if ready:
result = cmd.stdout.readline()
print result
Also, I got the code above from this thread, but I am not sure I understand how the polling mechanism works.
What exactly is registered above?
Why do I need the variable ready if I don't pass any timeout to poll.poll()?

Yes, it is entirely possible to display the output of a shell comamand as the command runs. There are two requirements:
1) The command must flush its output.
Many programs buffer their output differently according to whether the output is connected to a terminal, a pipe, or a file. If they are connected to a pipe, they might write their output in much bigger chunks much less often. For each program that you execute, consult its documentation. Some versions of /bin/cat', for example, have the -u switch.
2) You must read it piecemeal, and not all at once.
Your program must be structured to one piece at a time from the output stream. This means that you ought not do these, which each read the entire stream at one go:
cmd.stdout.read()
for i in cmd.stdout:
list(cmd.stdout.readline())
But instead, you could do one of these:
while not_dead_yet:
line = cmd.stdout.readline()
for line in iter(cmd.stdout.readline, b''):
pass
Now, for your three specific questions:
Is there any way to display the output of a shell command in Python, as the command runs?
Yes, but only if the command you are running outputs as it runs and doesn't save it up for the end.
What exactly is registered above?
The file descriptor which, when read, makes available the output of the subprocess.
Why do I need the variable ready if I don't pass any timeout to poll.poll()?
You don't. You also don't need the poll(). It is possible, if your commands list is fairly large, that might need to poll() both the stdin and stdout streams to avoid a deadlock. But if your commands list is fairly modest (less than 5Kbytes), then you will be OK just writing them at the beginning.
Here is one possible solution:
#! /usr/bin/python
import subprocess
import select
# Critical: all of this must fit inside ONE pipe() buffer
commands = ['echo Start\n', 'date\n', 'sleep 10\n', 'date\n', 'exit\n']
cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# The list "commands" holds a list of shell commands
for command in commands:
cmd.stdin.write(command)
# Must include this to ensure data is passed to child process
cmd.stdin.flush()
for line in iter(cmd.stdout.readline, b''):
print line