I need to automate a few bash scripts which involves answering to read prompts with y/n.
I tried to pipe stdout/stderr/stdin to a python script. Writing to stdin works but reading the prompt text from stdout/stderr doesn't for some reason? (I can read everything else that bash or sub-processes output fine.)
>>> from subprocess import Popen, PIPE
>>> proc = Popen(['bash','-c','read -r -p "Update system? [y/N] " response'],stdout=PIPE,stdin=PIPE,stderr=PIPE)
>>> proc.stdout.read(10) # <-- hangs, same with stderr, any length
I was expecting I would be able to read displayed prompt "Update system? [y/N] " somehow so I can decide what answer to pass back.
This is what expect is good at:
https://likegeeks.com/expect-command/
Expect and bash
https://unix.stackexchange.com/questions/351446/bash-and-expect-in-the-same-script
Related
I've seen examples and questions about how to do these things individually. But in this question I'm trying to do them all jointly.
Basically my case is that I have a command that needs me to write to its STDIN, read from its STDOUT, and to answer its TTY prompts. All done with a single execution of the command. Not that it matters, but if you're curious, the command is scrypt enc - out.enc.
Restrictions: must be pure Python.
Question: how to do it?
I tried these:
import pty
import os
import subprocess
master, slave = pty.openpty()
p = subprocess.Popen(['sudo', 'ls', '-lh'], stdin=slave, stdout=master)
x= os.read(master)
print(x)
stdout, stderr = p.communicate(b'lol\r\n')
import pty
import os
import sys
import subprocess
def read(fd):
data = os.read(fd, 1024)
data_str = data.decode()
if data_str.find('[sudo] password for') == 0:
data_str = 'password plz: '
sys.stdout.write(data_str)
sys.stdout.flush()
def write(fd):
x = 'lol\r\n'
for b in x.encode():
os.write(fd, b)
pty.spawn(['sudo', 'ls', '-lh'], read, write)
The goal is to fully wrap the TTY prompts so that they are not visible to the user, and at the same time to feed some password to processes TTY input to make sudo happy.
Based on that goal, none of these attempts work for various reasons.
But it is even worse: suppose that they work, how can I feed the process something to its STDIN and its TTY-input? What confuses me is that the Popen example literally states that stdin is mapped to TTY (pty), so how can it know which is which? How will it know that some input is for STDIN and not TTY-in?
Disclaimer:
Discussing this topic in detail would require a lot of text so I will try to simplify things to keep it short. I will try to include as many "for further reading" links as possible.
To make it short, there is only one input stream, that is STDIN. In a normal terminal, STDIN is connected to a TTY. So what you "type on TTY" will be read by the shell. The shell decides what to do with it then. It there is a program running, it will send it to STDIN of that program.
If you run something with Popen in python, that will not have a tty. You can check that easily by doing this:
from subprocess import Popen, PIPE
p = Popen("tty", stdin=PIPE, stdout=PIPE, stderr=PIPE)
o, e = p.communicate()
print(o)
It will produce this output: b'not a tty\n'
But how does scrypt then try to use a TTY? Because that is what it does.
You have to look at the manpage and code, to find the answer.
If -P is not given, scrypt reads passphrases from its controlling terminal, or failing that, from stdin.
What it does is actually, it is just opening /dev/tty (look at the code). That exists, even if the process does not have a TTY. So it can open it and it will try to read the password from it.
How can you solve your problem now?
Well, that is easy in this case. Check the manpage for the -P parameter.
Here is a working example:
from subprocess import Popen, PIPE
p = Popen("scrypt enc -P - out.enc", stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
p.communicate("pwd\nteststring")
This will encrypt the string "teststring" with the password "pwd".
There are a lot of "hacks" around ttys etc. but you should avoid those as they can have unexpected results. For example, start a shell and run tty then run a second shell and run cat with the output of the tty command (e.g. cat /dev/pts/7). Then type something in the first shell and watch what happens.
If you don't want to try it out, some characters will end up in the first shell, some in the second.
Check this post and this article about what a TTY is and where it comes from.
I have the next code:
from subprocess import Popen, PIPE
p = Popen("C:/cygwin64/bin/bash.exe", stdin=PIPE, stdout=PIPE)
path = "C:/Users/Link/Desktop/folder/"
p.stdin.write(b"cd " + str.encode(path)))
p.stdin.close()
out = p.stdout.read()
print(out)
The output is b''
Is there any way to pass a variable to the bash command p.stdin.write(b"cd " + path)
I ask because the way it is written above don't work. Output is null, just like Cygwin started and nothing else.
EDIT
As long as I see the question is not so clear, I'll add this scenario:
I am on Windows and I am using Python 3.6.
I have a bash cmd that requieres Cygwin to be executed. This cmd may have a variable in his string, which will change after every execution. Immagine a for loop which executes a command.
For example (an ImageMagick command):
convert image.jpg -resize 1024x768 output_file.jpg
How can I execute this cmd from Python with output_file.jpg as variable ?
Bash doesn't run in interactive mode by default unless it detects that standard input and output are connected to a terminal. You PIPEd these in, therefore they're definitely not connected to a terminal.
Bash does not display any prompts in non-interactive mode, hence you see nothing. You can force it to be interactive with -i switch.
However, even then, it is not going to write to stdout but stderr; you can try piping stderr to stdout
from subprocess import Popen, PIPE, STDOUT
p = Popen(["C:/cygwin64/bin/bash.exe", "-i"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
and you will capture the prompts and such.
Or use your original approach with a command that does produce output - here pwd that prints the current working directory:
p.stdin.write(b"cd " + path.encode() + b"\n")
p.stdin.write(b"pwd")
It is tricky to talk to an interactive process like this though - read too little => deadlock. Write too much => deadlock. This is why Popen has the .communicate method for providing all of input at once and getting the stdout and stderr afterwards.
As it seems you are using the Cygwin python, than you should use proper
Posix paths and not Windows-like ones
Instead of
p = Popen("C:/cygwin64/bin/bash.exe", stdin=PIPE, stdout=PIPE)
use
p = Popen("/bin/bash.exe", stdin=PIPE, stdout=PIPE)
I'm new to python and would like to open a windows cmd prompt, start a process, leave the process running and then issue commands to the same running process.
The commands will change so i cant just include these commands in the cmdline variable below. Also, the process takes 10-15 seconds to start so i dont want to waste time waiting for the process to start and run commands each time. just want to start process once. and run quick commands as needed in the same process
I was hoping to use subprocess.Popen to make this work, though i am open to better methods. Note that my process to run is not cmd, but im just using this as example
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi' readback
cmd.stdin.write("echo hi again") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi again' readback
The results arent what i expect. Seems as though the stdin.write commands arent actually getting in and the readline freezes up with nothing to read.
I have tried the popen.communicate() instead of write/readline, but it kills the process. I have tried setting bufsize in the Popen line, but that didn't make too much difference
Your comments suggest that you are confusing command-line arguments with input via stdin. Namely, the fact that system-console.exe program accepts script=filename parameter does not imply that you can send it the same string as a command via stdin e.g., python executable accepts -c "print(1)" command-line arguments but it is a SyntaxError if you pass it as a command to Python shell.
Therefore, the first step is to use the correct syntax. Suppose the system-console.exe accepts a filename by itself:
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE
with Popen(r'C:\full\path\to\system-console.exe -cli -',
stdin=PIPE, bufsize=1, universal_newlines=True) as shell:
for _ in range(10):
print('capture.tcl', file=shell.stdin, flush=True)
time.sleep(5)
Note: if you've redirected more than one stream e.g., stdin, stdout then you should read/write both streams concurrently (e.g., using multiple threads) otherwise it is very easy to deadlock your program.
Related:
Q: Why not just use a pipe (popen())? -- mandatory reading for Unix environment but it might also be applicable for some programs on Windows
subprocess readline hangs waiting for EOF -- code example on how to pass multiple inputs, read multiple outputs using subprocess, pexpect modules.
The second and the following steps might have to deal with buffering issues on the side of the child process (out of your hands on Windows), whether system-console allows to redirect its stdin/stdout or whether it works with a console directly, and character encoding issues (how various commands in the pipeline encode text).
Here is some code that I tested and is working on Windows 10, Quartus Prime 15.1 and Python 3.5
import subprocess
class altera_system_console:
def __init__(self):
sc_path = r'C:\altera_lite\15.1\quartus\sopc_builder\bin\system-console.exe --cli --disable_readline'
self.console = subprocess.Popen(sc_path, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def read_output(self):
rtn = ""
loop = True
i = 0
match = '% '
while loop:
out = self.console.stdout.read1(1)
if bytes(match[i],'utf-8') == out:
i = i+1
if i==len(match):
loop=False
else:
rtn = rtn + out.decode('utf-8')
return rtn
def cmd(self,cmd_string):
self.console.stdin.write(bytes(cmd_string+'\n','utf-8'))
self.console.stdin.flush()
c = altera_system_console()
print(c.read_output())
c.cmd('set jtag_master [lindex [get_service_paths master] 0]')
print(c.read_output())
c.cmd('open_service master $jtag_master')
print(c.read_output())
c.cmd('master_write_8 $jtag_master 0x00 0xFF')
print(c.read_output())
You need to use iter if you want to see the output in real time:
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi\n")#would like this to be written to the cmd prompt
for line in iter(cmd.stdout.readline,""):
print line
cmd.stdin.write("echo hi again\n")#would like this to be written to the cmd prompt
Not sure exactly what you are trying to do but if you want to input certain data when you get certain output then I would recommend using pexpect
I just want to build a little python music client on my raspberry pi. I installed "mpg321" and it works great but now my problem. After sending the command
os.system("mpg321 -R testPlayer")
python waits for user input like play, pause or quit. If I write this in my terminal the player pause the music oder quits. Perfect but I want python to do that so I send the command
os.system("LOAD test.mp3")
where LOAD is the command for loading this mp3. But nothing happens. When I quit the player via terminal I get the error:
sh: 1: LOAD: not found
I think this means that
os.system("mpg321 -R testPlayer")
takes the whole process and after I quit it python tries to execute the comman LOAD. So how do I get these things work together?
My code:
import os
class PyMusic:
def __init__(self):
print "initial stuff later"
def playFile(self, fileName, directory = ""):
os.system("mpg321 -R testPlayer")
os.system("LOAD test.mp3")
if __name__ == "__main__":
pymusic = PyMusic()
pymusic.playFile("test.mp3")
Thanks for your help!
First, you should almost never be using os.system. See the subprocess module.
One major advantage of using subprocess is that you can choose whatever behavior you want—run it in the background, start it and wait for it to finish (and throw an exception if it returns non-zero), interact with its stdin and stdout explicitly, whatever makes sense.
Here, you're not trying to run another command "LOAD test.mp3", you're trying to pass that as input to the existing process. So:
p = subprocess.Popen(['mpg321', '-R', 'testPlayer'], stdin=PIPE)
Then you can do this:
p.stdin.write('LOAD test.mp3\n')
This is roughly equivalent to doing this from the shell:
echo -e 'LOAD test.mp3\n' | mpg321 -R testPlayer
However, you should probably read about communicate, because whenever it's possible to figure out how to make your code work with communicate, it's a lot simpler than trying to deal with generic I/O (especially if you've never coded with pipes, sockets, etc. before).
Or, if you're trying to interact with a command-line UI (e.g., you can't send the command until you get the right prompt), you may want to look at an "expect" library. There are a few of these to choose from, so you should search at PyPI to find the right one for you (although I can say that I've used pexpect successfully in the past, and the documentation is full of samples that get the ideas across a lot more quickly than most expect documentation does).
You are looking for a way to send data to stdin. Here is an example of this using Popen:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['mpg321', '-R testPlayer'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
mpg123_stdout = p.communicate(input='LOAD test.mp3\n')[0]
print(mpg123_stdout)
You establish pointers to stdin and stdout, then after you start your process, you communicate with stdin and read from stdout. Be sure to send new lines (carriage returns)
I tried something like this, but with no effect:
command = "cmd.exe"
proc = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
proc.stdin.write("dir c:\\")
how about simply:
import os
os.system('dir c:\\')
You probably want to try something like this:
command = "cmd.exe /C dir C:\\"
I don't think you can pipe into cmd.exe... If you are coming from a unix background, well, cmd.exe has some ugly warts!
EDIT: According to Sven Marnach, you can pipe to cmd.exe. I tried following in a python shell:
>>> import subprocess
>>> proc = subprocess.Popen('cmd.exe', stdin = subprocess.PIPE, stdout = subprocess.PIPE)
>>> stdout, stderr = proc.communicate('dir c:\\')
>>> stdout
'Microsoft Windows [Version 6.1.7600]\r\nCopyright (c) 2009 Microsoft Corporatio
n. All rights reserved.\r\n\r\nC:\\Python25>More? '
As you can see, you still have a bit of work to do (only the first line is returned), but you might be able to get this to work...
Try:
import os
os.popen("Your command here")
Using ' and " at the same time works great for me (Windows 10, python 3)
import os
os.system('"some cmd command here"')
for example to open my web browser I can use this:
os.system(r'"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"')
(Edit)
for an easier way to open your browser I can use this:
import webbrowser
webbrowser.open('website or leave it alone if you only want to open the
browser')
Try adding a call to proc.stdin.flush() after writing to the pipe and see if things start behaving more as you expect. Explicitly flushing the pipe means you don't need to worry about exactly how the buffering is set up.
Also, don't forget to include a "\n" at the end of your command or your child shell will sit there at the prompt waiting for completion of the command entry.
I wrote about using Popen to manipulate an external shell instance in more detail at: Running three commands in the same process with Python
As was the case in that question, this trick can be valuable if you need to maintain shell state across multiple out-of-process invocations on a Windows machine.
Taking some inspiration from Daren Thomas's answer (and edit), try this:
proc = subprocess.Popen('dir C:\\', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = proc.communicate()
out will now contain the text output.
They key nugget here is that the subprocess module already provides you shell integration with shell=True, so you don't need to call cmd.exe directly.
As a reminder, if you're in Python 3, this is going to be bytes, so you may want to do out.decode() to convert to a string.
Why do you want to call cmd.exe ? cmd.exe is a command line (shell). If you want to change directory, use os.chdir("C:\\"). Try not to call external commands if Python can provide it. In fact, most operating system commands are provide through the os module (and sys). I suggest you take a look at os module documentation to see the various methods available.
It's very simple. You need just two lines of code with just using the built-in function and also it takes the input and runs forever until you stop it. Also that 'cmd' in quotes, leave it and don't change it. Here is the code:
import os
os.system('cmd')
Now just run this code and see the whole windows command prompt in your python project!
Here's a way to just execute a command line command and get its output using the subprocess module:
import subprocess
# You can put the parts of your command in the list below or just use a string directly.
command_to_execute = ["echo", "Test"]
run = subprocess.run(command_to_execute, capture_output=True)
print(run.stdout) # the output "Test"
print(run.stderr) # the error part of the output
Just don't forget the capture_output=True argument and you're fine. Also, you will get the output as a binary string (b"something" in Python), but you can easily convert it using run.stdout.decode().
In Python, you can use CMD commands using these lines :
import os
os.system("YOUR_COMMAND_HERE")
Just replace YOUR_COMMAND_HERE with the command you like.
From Python you can do directly using below code
import subprocess
proc = subprocess.check_output('C:\Windows\System32\cmd.exe /k %windir%\System32\\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f' ,stderr=subprocess.STDOUT,shell=True)
print(str(proc))
in first parameter just executed User Account setting you may customize with yours.