I'm trying to write a Python script that starts a subprocess to run an Azure CLI command once the file is executed.
When I run locally, I run:
az pipelines create --name pipeline-from-cli --repository https://github.com/<org>/<project> --yml-path <path to pipeline>.yaml --folder-path _poc-area
I get prompted for an input which looks like:
Which service connection do you want to use to communicate with GitHub?
[1] Create new GitHub service connection
[2] <my connection name>
[3] <org name>
Please enter a choice [Default choice(1)]:
I can type in 2 and press enter then my pipeline is successfully created in Azure DevOps. I would like to run this command being dynamically entered when prompted.
So far I have tried:
import subprocess
cmd = 'az pipelines create --name pipeline-from-cli --repository https://github.com/<org>/<project> --yml-path <path to pipeline>.yaml --folder-path _poc-area
cmd = cmd.split()
subprocess.run(cmd, shell=True)
This will run in the exact same way as when I try to run it locally.
Try to follow answers from here I have also tried:
p = subprocess.run(cmd, input="1", capture_output=True, text=True, shell=True)
print(p)
Which gives me an error saying raise NoTTYException(error_msg)\nknack.prompting.NoTTYException.
Is there a way where I can execute this Python script, and it will run the Azure CLI command then enter 2 when prompted without any manually intervention?
You are trying to solve the wrong problem. az pipeline create takes a --service-connection parameter. You don't need to respond to the prompt, you can provide the service connection value on the command line and skip the prompt entirely.
IMHO, Daniel is right, you're not supposed to deal with stdin in your program.
Nevertheless, if you really need to, you should use pexpect package, which basically opens a process, waits for given output, and then sends input to the process' stdin.
Here's a basic example:
import pexpect
from pexpect.popen_spawn import PopenSpawn
cmd = 'az pipelines create --name pipeline-from-cli --repository https://github.com/<org>/<project> --yml-path <path to pipeline>.yaml --folder-path _poc-area'
child = pexpect.popen_spawn.PopenSpawn('cmd', timeout=1)
child.expect ('.*Please enter a choice.*')
child.sendline ('2')
# child.interact() # Give control of the child to the user.
Have a look at pexpect documentation for more details. MS Windows support is available since v4.0.
Another archaic solution would be to use subprocess the following way, emulating basically what expect would do:
import subprocess
from time import sleep
p = subprocess.Popen(azure_command, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
sleep(.5)
stdout = p.communicate(input=b'2\n')[0]
print(stdout.decode())
Still, best solution is to use non-interactive mode of most CLI programs.
Related
I am working with group of EC2 instances that are deployed in a subnet.
I usually use the aws ssm command
aws ssm start-session --region us-east-2 --target i-01234567abcdef --profile profile-one
to connect to that ec2 instance and run commands such as
ssh mirrora
to go into a different ec2 instance in the subnet from the one I am connected to
Then I can run ls on mirrora to look at the files and such.
Now I am trying to automate the same using python
I am new to python (and first time working with subprocess). I have researched a bit and learned that I can use Popen to open a subprocess and I set the stdin and stdout arguments to subprocess.PIPE
Here's a sample of the code I am trying to execute
command = "aws ssm start-session --region us-east-2 --target i-0123456abcdef --profile profile-one"
ssh = subprocess.Popen(command.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0, shell= True, universal_newlines=True)
ssh.stdin.write("ssh mirrora")
ssh.stdin.write("ls")
# output = ssh.stdout.read()
# output , err = ssh.communicate()
# print("output:" + output)
# print("err:" + err)
# output = ssh.stdout.read()
# print("Output of ls is:", output)
print("HERE")
If I use either stdout.read or communicate method, the program seems to get stuck and nothing works. I can't even ctrl + c the terminal.
If I run it without any stdout or communicate, it prints "HERE" in the terminal (almost instantaneously) and nothing actually seems to happen on the server side (as I tried to run the shutdown command to turn off the mirrora, but it is still running after program exits without stdout or communicate).
What am I doing wrong?
Ideally I want to run the commands I want inside the subprocess that aws ssm opens and run certain commands on one of the ec2 instances and get the output for one or more commands using stdout.read after certain stdin.write
Any help or links will be appreciated.
Thanks
It seems to me that you should use run-command instead start-session.
This is an SSM feature used to execute static commands and scripts, where a session is similar to just an ssh connection.
I think it should work like this:
import boto3
script = f"""
ssh mirrora 'ls'
ssh mirrora 'ls; ps -aux'
"""
target = "i-01234567abcdef "
session = boto3.Session(profile_name="profile-one")
ssm_client = session.client("ssm")
response = ssm_client.send_command(
InstanceIds=[target],
DocumentName="AWS-RunShellScript", # https://eu-central-1.console.aws.amazon.com/systems-manager/documents/AWS-RunShellScript/description?region=eu-central-1
TimeoutSeconds=20,
Comment="Example of send command",
Parameters={
'commands': [
script,
],
'workingDirectory': "/home/user1",
},
)
Here you can find the documentation for send-command API. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.send_command
If you want to view the logs in the python script then you will need to extend it - probably easiest way is to set an s3 bucket in the send_command (it will redirect your stdout and stderr to provided s3 bucket), download saved in s3 files, and cat/print them.
I'm trying to run a sequence of shell commands in the same environment:
same exported variables, persistent history, etc.
And I want to work with each commands output before running the next command.
After looking over python subprocess.run and Pexpect.spawn neither seem to provide both features.
subprocess.run allows me to run one command and then examine the output, but not to keep the environment open for another command.
Pexpect.spawn("bash") allows me to run multiple commands in the same environment, but i can't get the output until EOF; when bash itself exits.
Ideally i would like an interface that can do both:
shell = bash.new()
shell.run("export VAR=2")
shell.run("whoami")
print(shell.exit_code, shell.stdout())
# 0, User
shell.run("echo $VAR")
print(shell.stdout())
# 2
shell.run("!!")
print(shell.stdout())
# 2
shell.run("cat file -")
shell.stdin("Foo Bar")
print(shell.stdout())
# Foo Bar
print(shell.stderr())
# cat: file: No such file or directory
shell.close()
Sounds like a case for Popen. You can specify bufsize to disable buffering, if it gets in the way.
Example from the linked page:
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
There's also proc.stdin for sending more commands, and proc.stderr.
I've managed to get the cmd being opened by python. However, using runas administrator comes with a password check before cmd.exe is executed.
I'm using this to open cmd...
import subprocess
subprocess.call(["runas", "/user:Administrator", "cmd.exe"])
I'm looking for a way to automatically enter the password into the runas.exe prompt which opens when i run the code. Say if i were to create var = "test" and add it after import subprocess how would i make it so that this variable is passed to and seen as an input to the runas.exe?
The solution would require only python modules which are in version 3.4 or higher.
Update
I have found some code which appears to input straight into runas.exe. However, the apparent input is \x00\r\n when in the code the input is supposed to be test I am fairly certain that if i can get the input to be test then the code will be successful.
The code is as follows :
import subprocess
args = ['runas', '/user:Administrator', 'cmd.exe']
proc = subprocess.Popen(args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.stdin.write(b'test\n')
proc.stdin.flush()
stdout, stderr = proc.communicate()
print (stdout)
print (stderr)
Although not an answer to your question, this can be a solution to your problem. Use psexec instead of runas. You can run it like this:
psexec -u user -p password cmd
(or run it from Python using subprocess.Popen or something else)
This piece of code actually works (tested on a Windows 2008 server). I've used it to call runas for a different user and pass his password. A new command prompt opened with new user context, without needing to enter password.
Note that you have to install pywin32 to have access to the win32 API.
The idea is:
to Popen the runas command, without any input redirection, redirecting output
read char by char until we encounter ":" (last char of the password prompt).
send key events to the console using win32 packages, with the final \r to end the password input.
(adapted from this code):
import win32console, win32con, time
import subprocess
username = "me"
domain = "my_domain"
password ="xxx"
free_console=True
try:
win32console.AllocConsole()
except win32console.error as exc:
if exc.winerror!=5:
raise
## only free console if one was created successfully
free_console=False
stdin=win32console.GetStdHandle(win32console.STD_INPUT_HANDLE)
p = subprocess.Popen(["runas",r"/user:{}\{}".format(domain,username),"cmd.exe"],stdout=subprocess.PIPE)
while True:
if p.stdout.read(1)==b":":
for c in "{}\r".format(password): # end by CR to send "RETURN"
## write some records to the input queue
x=win32console.PyINPUT_RECORDType(win32console.KEY_EVENT)
x.Char=unicode(c) # remove unicode for python 3
x.KeyDown=True
x.RepeatCount=1
x.VirtualKeyCode=0x0
x.ControlKeyState=win32con.SHIFT_PRESSED
stdin.WriteConsoleInput([x])
p.wait()
break
When running a secondary python script:
Is it possible to run a subprocess.Popen, or subprocess.call or even execfile in a new terminal? (as in simply a different terminal than the current terminal where the script is run).
Alternatively, if before running my program (main), I open two terminals first, can I then point the secondary script to the second terminal? (so somehow getting the ID of open terminals, and then using a specific one among them, to perform the subprocess).
An example, two subprocesses to be run, first.py should be called first, only then the second is called, second.py. Because the two scripts first.py and second.py are interdependent (as in first.py goes to wait mode, until second.py is run, then first.py resumes, and I don't know how to make this communication work between them in terms of subprocesses.)
import subprocess
command = ["python", "first.py"]
command2 = ["python", "second.py"]
n = 5
for i in range(n):
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = p.stdout.readline().strip()
print output
if output == 'stop':
print 'success'
p.terminate()
p2.terminate()
break
Framework (Ubuntu, python 2.7)
I guess you want something like
subprocess.call(['xterm','-e','python',script])
Good old xterm has almost no frills; on a Freedesktop system, maybe run xdg-terminal instead. On Debian, try x-terminal-emulator.
However, making your program require X11 is in most cases a mistake. A better solution is to run the subprocesses with output to a log file (or a socket, or whatever) and then separately run tail -f on those files (in a different terminal, or from a different server over ssh, or with output to a logger which supports rsyslog, or or or ...) which keeps your program simple and modular, free from "convenience" dependencies.
If you're using tmux, you can specify which target you want the command to run in:
tmux send -t foo.0 ls ENTER
So, if you've created a tmux session foo.0, you should be able to do:
my_command = 'ls'
tmux_cmd = ['tmux', 'send', '-t', 'foo.0', my_command]
p = subprocess.Popen(tmux_cmd)
You can specify the tty of the terminal window you wish the command to be carried out in:
ls > /dev/ttys004
However, I would recommend going for the tmux approach for greater control (see my other answer).
I'm trying to write a python script that start a process and do some operations atferward.
The commands that I want to automate by script are circled as red in the picture.
The problem is that after performing first command, qemu environment will be run and the other commands should be executed on the qemu environment. So I want to know how can I do these commands by an script in python? Because as I know I can do the first command but I do not know how can I do those commands when I am going to qemu environment.
Could you help me how can I do this process?
First thing that came to mind was pexpect, a quick search on google turned up this blog post automatically-testing-vms-using-pexpect-and-qemu which seems to be pretty much along the lines of what you are doing:
import pexpect
image = "fedora-20.img"
user = "root"
password = "changeme"
# Define the qemu cmd to run
# The important bit is to redirect the serial to stdio
cmd = "qemu-kvm"
cmd += " -m 1024 -serial stdio -net user -net nic"
cmd += " -snapshot -hda %s" % image
cmd += " -watchdog-action poweroff"
# Spawn the qemu process and log to stdout
child = pexpect.spawn(cmd)
child.logfile = sys.stdout
# Now wait for the login
child.expect('(?i)login:')
# And login with the credentials from above
child.sendline(user)
child.expect('(?i)password:')
child.sendline(password)
child.expect('# ')
# Now shutdown the machine and end the process
if child.isalive():
child.sendline('init 0')
child.close()
if child.isalive():
print('Child did not exit gracefully.')
else:
print('Child exited gracefully.')
You could do it with subprocess.Popen also, checking the stdout for the (qemu) lines and writing to stdin. Something roughly like this:
from subprocess import Popen,PIPE
# pass initial command as list of individual args
p = Popen(["./tracecap/temu","-monitor",.....],stdout=PIPE, stdin=PIPE)
# store all the next arguments to pass
args = iter([arg1,arg2,arg3])
# iterate over stdout so we can check where we are
for line in iter(p.stdout.readline,""):
# if (qemu) is at the prompt, enter a command
if line.startswith("(qemu)"):
arg = next(args,"")
# if we have used all args break
if not arg:
break
# else we write the arg with a newline
p.stdin.write(arg+"\n")
print(line)# just use to see the output
Where args contains all the next commands.
Don't forget that Python has batteries included. Take a look of the Suprocess module in the standard lib. There a lot of pitfalls managing processes, and the module take care of them all.
You probably want to start a qemu process and send the next commands writing to its standard input (stdin). Subprocess module will allow you to do it. See that qemu has command line options to connect to stdi: -chardev stdio ,id=id