Develop Reference

Running multiple Bash commands interactively from Python

I have just come across pexpect and have been figuring out how to use it to automate various practices I would otherwise have to fill in manually in a command shell.
Here's an example script:
import pexpect, sys
child = pexpect.spawn("bash", timeout=60)
child.logfile = sys.stdout
child.sendline("cd /workspace/my_notebooks/code_files")
child.expect('#')
child.sendline('ls')
child.expect('#')
child.sendline('git add .')
child.expect('#')
child.sendline('git commit')
child.expect('#')
child.sendline('git push origin main')
child.expect('Username .*:')
child.sendline(<my_github_username>)
child.expect('Password .*:')
child.sendline(<my_github_password>)
child.expect('#')
child.expect(pexpect.EOF)
(I know these particular tasks do not necessarily require pexpect, just trying to understand its best practices.)
Now, the above works. It cds to my local repo folder, lists the files there, stages my commits, and pushes to Github with authentication, all the while providing real-time output to the Python stdout. But I have two areas I'd like to improve:
Firstly, .expect('#') between every line I would run in Bash (that doesn't require interactivity) is a little tedious. (And I'm not sure whether / why it always seems to work, whatever was the output in stdout - although so far it does.) Ideally I could just clump them into one multiline string and dispense with all those expects. Isn't there a more natural way to automate parts of the script that could be e.g., a multiline string with Bash commands separated by ';' or '&&' or '||'?
Secondly, if you run a script like the above you'll see it times out after 60 seconds sharp, then yields a TimeoutError in Python. Although - assuming the job fits within 60 seconds - it gets done, I would prefer something which (1) doesn't take unnecessarily long, (2) doesn't risk cutting off a >60 second process midway, (3) doesn't end the whole thing giving me an error in Python. Can we instead have it come to a natural end, i.e., when the shell processes are finished, that's when it stops running in Python too? (If (2) and (3) can be addressed, I could probably just set an enormous timeout value - not sure if there is better practice though.)
What's the best way of rewriting the code above? I grouped these two issues in one question because my guess is there is a generally better way of using pexpect, which could solve both problems (and probably others I don't even know I have yet!), and in general I'd invite being shown the best way of doing this kind of task.

You don't need to wait for # between each command. You can just send all the commands and ignore the shell prompts. The shell buffers all the inputs.
You only need to wait for the username and password prompts, and then the final # after the last command.
You also need to send an exit command at the end, otherwise you won't get EOF.
import pexpect, sys
child = pexpect.spawn("bash", timeout=60)
child.logfile = sys.stdout
child.sendline("cd /workspace/my_notebooks/code_files")
child.sendline('ls')
child.sendline('git add .')
child.sendline('git commit')
child.sendline('git push origin main')
child.expect('Username .*:')
child.sendline(<my_github_username>)
child.expect('Password .*:')
child.sendline(<my_github_password>)
child.expect('#')
child.sendline('exit')
child.expect(pexpect.EOF)
If you're running into the 60 second timeout, you can use timeout=None to disable this. See pexpect timeout with large block of data from child
You could also combine multiple commands in a single line:
import pexpect, sys
child = pexpect.spawn("bash", timeout=60)
child.logfile = sys.stdout
child.sendline("cd /workspace/my_notebooks/code_files && ls && git add . && git commit && git push origin main')
child.expect('Username .*:')
child.sendline(<my_github_username>)
child.expect('Password .*:')
child.sendline(<my_github_password>)
child.expect('#')
child.sendline('exit')
child.expect(pexpect.EOF)
Using && between the commands ensures that it stops if any of them fails.
In general I wouldn't recommend using pexpect for this at all. Make a shell script that does everything you want, and run the script with a single subprocess.Popen() call.

Linux: cat to named pipe in a python script

I have a Java program that uses video from a framegrabber card. This program is launched through a python launcher.py.
The easiest way to read the video stream I found, is to make Java read on a named pipe, and this works perfectly. So my session is like:
$ mkfifo videopipe
$ cat /dev/video1>videopipe
and in a second terminal (since the cat command is blocking):
$ python launcher.py
I would like to automate this process. Unfortunately, the result is always the same: the Java application starts (confirmed through a print statement in the java program), but then the terminal stalls and nothing appears, exception or else.
Since this process works manually, I guess I am doing something wrong in the python program. To simplify things, I isolated the piping part:
from subprocess import call, Popen, PIPE, check_call
BASH_SWITCHTO_WINTV = ['v4l2-ctl', '-d /dev/video1', '-i 2', '--set-standard=4']
BASH_CREATE_FIFO_PIPE = ['mkfifo', 'videopipe']
BASH_PIPE_VIDEO = 'cat /dev/video1>videopipe'
def run():
try:
print('running bash commands...')
call(BASH_SWITCHTO_WINTV)
call(BASH_CREATE_FIFO_PIPE)
Popen(['cat', '/dev/video1'], stdout=open('videopipe', 'w'))
except:
raise RuntimeError('An error occured while piping the video')
if __name__ == '__main__':
run()
which when run, outputs:
running bash commands...
Failed to open /dev/video1: No such file or directory
A little help would be very much appreciated :-)

If you're using shell=True, just pass a string:
BASH_PIPE_VIDEO = 'cat /dev/video1 > videopipe'
Currently, cat is passed to the shell as your script, and /dev/video>videopipe is passed to that shell as a literal argument -- not parsed as part of the script text at all, and having no effect since the script (just calling cat) doesn't look at its arguments.
Alternately, to avoid needless shell use (and thus shell-related bugs such as shellshock, and potential for injection attacks if you were accepting any argument from a non-hardcoded source):
Popen(['cat', '/dev/video1'], stdout=open('videopipe, 'w'))
On a note unrelated to your "cat to named pipe" question -- be sure you get your spaces correct.
BASH_SWITCHTO_WINTV = ['v4l2-ctl', '-d /dev/video1', ...]
...uses the name <space>/dev/video1, with a leading space, as the input device; it's the same as running v4l2-ctl "-d /dev/video1" in shell, which would cause the same problem.
Be sure that you split your arguments correctly:
BASH_SWITCHTO_WINTV = ['v4l2-ctl', '-d', '/dev/video1', ...]

Sending commans on sub process in python

I just want to build a little python music client on my raspberry pi. I installed "mpg321" and it works great but now my problem. After sending the command
os.system("mpg321 -R testPlayer")
python waits for user input like play, pause or quit. If I write this in my terminal the player pause the music oder quits. Perfect but I want python to do that so I send the command
os.system("LOAD test.mp3")
where LOAD is the command for loading this mp3. But nothing happens. When I quit the player via terminal I get the error:
sh: 1: LOAD: not found
I think this means that
os.system("mpg321 -R testPlayer")
takes the whole process and after I quit it python tries to execute the comman LOAD. So how do I get these things work together?
My code:
import os
class PyMusic:
def __init__(self):
print "initial stuff later"
def playFile(self, fileName, directory = ""):
os.system("mpg321 -R testPlayer")
os.system("LOAD test.mp3")
if __name__ == "__main__":
pymusic = PyMusic()
pymusic.playFile("test.mp3")
Thanks for your help!

First, you should almost never be using os.system. See the subprocess module.
One major advantage of using subprocess is that you can choose whatever behavior you want—run it in the background, start it and wait for it to finish (and throw an exception if it returns non-zero), interact with its stdin and stdout explicitly, whatever makes sense.
Here, you're not trying to run another command "LOAD test.mp3", you're trying to pass that as input to the existing process. So:
p = subprocess.Popen(['mpg321', '-R', 'testPlayer'], stdin=PIPE)
Then you can do this:
p.stdin.write('LOAD test.mp3\n')
This is roughly equivalent to doing this from the shell:
echo -e 'LOAD test.mp3\n' | mpg321 -R testPlayer
However, you should probably read about communicate, because whenever it's possible to figure out how to make your code work with communicate, it's a lot simpler than trying to deal with generic I/O (especially if you've never coded with pipes, sockets, etc. before).
Or, if you're trying to interact with a command-line UI (e.g., you can't send the command until you get the right prompt), you may want to look at an "expect" library. There are a few of these to choose from, so you should search at PyPI to find the right one for you (although I can say that I've used pexpect successfully in the past, and the documentation is full of samples that get the ideas across a lot more quickly than most expect documentation does).

You are looking for a way to send data to stdin. Here is an example of this using Popen:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['mpg321', '-R testPlayer'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
mpg123_stdout = p.communicate(input='LOAD test.mp3\n')[0]
print(mpg123_stdout)
You establish pointers to stdin and stdout, then after you start your process, you communicate with stdin and read from stdout. Be sure to send new lines (carriage returns)

How to mimic passing a file using python variable

I am trying to figure out how to mimic passing a file to a program and instead of passing the file pass a python variable that has what information the program needs in it.
The syntax I need to mimic:
msfconsole < setofcommands.txt 2>/dev/null
What I would like to do:
answer = os.system('msfconsole < ' + myvariable + ' 2>/dev/null')
I also would be extremely happy if someone knew how to interact with something like msfconsole over a period of time so that when the msfconsole shell is open I could continually get python to type different commands without have to relaunch a new application for each time I wanted to type a series of commands
Thanks a lot in advance.

Instead, of os.system, you should use subprocess:
import subprocess
p=subprocess.Popen(['msfconsole'], stdin=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate(myvariable)
Note that communicate will block until termination. In general, interactive-like communication with another process just via stdin/stdout is a hard problem, since you must know how the other process delimits ends of messages.
In the case of metasploit, you should use the XMLRPC API instead of the interactive tool msfconsole.

Cannot Launch Interactive Program While Piping to Script in Python

I have a python script that needs to call the defined $EDITOR or $VISUAL. When the Python script is called alone, I am able to launch the $EDITOR without a hitch, but the moment I pipe something to the Python script, the $EDITOR is unable to launch. Right now, I am using nano which shows
Received SIGHUP or SIGTERM
every time. It appears to be the same issue described here.
sinister:Programming [1313]$ echo "import os;os.system('nano')" > "sample.py"
sinister:Programming [1314]$ python sample.py
# nano is successfully launched here.
sinister:Programming [1315]$ echo "It dies here." | python sample.py
Received SIGHUP or SIGTERM
Buffer written to nano.save.1
EDIT: Clarification; inside the program, I am not piping to the editor. The code is as follows:
editorprocess = subprocess.Popen([editor or "vi", temppath])
editorreturncode = os.waitpid(editorprocess.pid, 0)[1]

When you pipe something to a process, the pipe is connected to that process's standard input. This means your terminal input won't be connected to the editor. Most editors also check whether their standard input is a terminal (isatty), which a pipe isn't; and if it isn't a terminal, they'll refuse to start. In the case of nano, this appears to cause it to exit with the message you included:
% echo | nano
Received SIGHUP or SIGTERM
You'll need to provide the input to your Python script in another way, such as via a file, if you want to be able to pass its standard input to a terminal-based editor.
Now you've clarified your question, that you don't want the Python process's stdin attached to the editor, you can modify your code as follows:
editorprocess = subprocess.Popen([editor or "vi", temppath],
stdin=open('/dev/tty', 'r'))

The specific case of find -type f | vidir - is handled here:
foreach my $item (#ARGV) {
if ($item eq "-") {
push #dir, map { chomp; $_ } <STDIN>;
close STDIN;
open(STDIN, "/dev/tty") || die "reopen: $!\n";
}
You can re-create this behavior in Python, as well:
#!/usr/bin/python
import os
import sys
sys.stdin.close()
o = os.open("/dev/tty", os.O_RDONLY)
os.dup2(o, 0)
os.system('vim')
Of course, it closes the standard input file descriptor, so if you intend on reading from it again after starting the editor, you should probably duplicate its file descriptor before closing it.