Develop Reference

How to use Subprocess in Python

I would like to execute following shell command in python: grep 'string' file | tail -1 | cut -c 1-3
I tried:
import subprocess
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
subprocess.call(grep 'string' file | tail -1 | cut -c 1-3)
i = i + 1
Any help would be appreciated. Thanks.

First of all, whatever you pass into the subprocess.call should be a string. Names grep, file, tail and cut are not defined in your code and you need to turn the whole expression into a string. Since the search string for the grep command should be dynamic, you need to construct the final string before passing it as argument into the function.
import subprocess
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
command_string = 'grep {0} {1} | tail -1 | cut -c 1-3'.format(string, file)
subprocess.call(command_string)
i = i + 1
You probably want to pass in an additional argument to subprocess.call: shell=True. The argument will make sure the command is executed through the shell.
Your command is using cut. You might want to retrieve the output of the subprocess, so a better option would be to create a new process object and use subprocess.communicate with turned out output capturing:
import subprocess
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
command_string = 'grep {0} {1} | tail -1 | cut -c 1-3'.format(string, file)
p = subprocess.Popen(command_string, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutdata, stderrdata = p.communicate()
# stdoutdata now contains the output of the shell commands and you can use it
# in your program
i = i + 1
EDIT: Here is the information on how to store the data into a text file, as requested in the comment.
import subprocess
outputs = []
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
command_string = 'grep {0} {1} | tail -1 | cut -c 1-3'.format(string, file)
p = subprocess.Popen(command_string, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
stdoutdata, stderrdata = p.communicate()
# stdoutdata now contains the output of the shell commands and you can use it
# in your program, like writing the output to a file.
outputs.append(stdoutdata)
i = i + 1
with open('output.txt', 'w') as f:
f.write('\n'.join(outputs))

Your command should be provided as a string.
In addition, if you want to get the output of your command, you can use the following:
subprocess.run("grep 'string' file | tail -1 | cut -c 1-3", shell=True, capture_output=True, check=True)
where capture_output (works in Python3.7+) returns object with returncode, stdout and stderr and the check flag will raise exception if your command fails.

Subprocess expects the arguments as a string or array:
subprocess.call("grep '{}' {} | tail -1 | cut -c 1-3".format(string, file), shell=True)
shell=True is nececairy because you are using shell-specific commands like the pipe.
However, in this case it might be a lot easier to implement the entire program in pure python.
Note that if either string or file contain any special characters including spaces or quotation marks, the command will not work, and could in fact do a variety of unwanted things to your system. If you need it to work on more than these simple values, consider either a pure-python solution, setting shell=False and using the array syntax with manual piping, or some form of escaping.

Cant use grep in subprocess command

I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)

When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)

Python - subprocess cmd as list not expanding properly

Any ideas why this list that I'm passing as my command to subprocess() isn't expanding properly?
file_name = "some_make_file"
cmd = ['gmake', '-pn', '-f', file_name, '|', 'grep', '-A1', '"^# makefile"', '|', 'grep', '-v', '"^#\|^--"', '|', 'sort', '|', 'uniq']
proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
stdout, stderr = prod.communicate()
The output is a gmake error, and if I join the cmd to test it on the cmd line myself, I get this:
#join
" ".join(cmd)
#output
('gmake -pn -f some_make_file | grep -A1 "^ makefile" | grep -v '
'"^#\\|^--" | sort | uniq')
For the life if me, I cannot seem to figure out what's wrong with this command. Any ideas? Am I not escaping something properly? Adding a special char and not realizing it?

You are not running an intermediate shell so there is nothing to interprete the | as a pipe command. Even if you set shell=True, because you passed in a list, it will be escaped. The proper way to pipeline is given in the official python docs Replacing Shell Pipeline – tdelaney

Why does this valid shell command throw an error in python via subprocess?

The line awk -F'[][]' '/dB/ { print $2 }' <(amixer sget Master) in bash returns my system's current volume (e.g. "97%").
I tried to incorporate this in Python 3
#!/usr/bin/env python3
import subprocess
command = "awk -F'[][]' '/dB/ { print $2 }' <(amixer sget Master)"
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()
print(output)
However the output from the shell returns
/bin/sh: 1: Syntax error: "(" unexpected
b''
Why does this fail and how do I fix my code?

As already pointed out, the syntax you are using is a bash syntax (a.k.a. bashism). The default shell used in subprocess.Popen is /bin/sh & it does not support process substitution.
You can specify the shell to be used via executable argument.
Try this:
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, executable="/bin/bash").stdout.read()

Because you are using bashism in form of a process substitution, and your /bin/sh doesn't support that:
<(...)
Changing this to a pipe should solve your problem:
command = "amixer sget Master | awk -F'[][]' '/dB/ { print $2 }'"
Alternative you can start bash from within sh:
command = "bash -c 'amixer sget Master | awk -F'\\''[][]'\\'' '\\''/dB/ { print $2 }'\\'''"
But as you will soon realize, quoting and escaping will become a nightmare

bash commands from within python

I'm looking for the best way to use bash commands from within python. What ways are there? I know of os.system and subprocess.Popen.
I have tried these:
bootfile = os.system("ls -l /jffs2/a.bin | cut -d '/' -f 4")
print bootfile
This returns a.bin as expected but also it retuns 0 afterwards and so prints:
a.bin
0
with bootfile now being set to 0. The next time I print bootfile it just shows up as 0. Which is the exit value I guess, how do i stop this value interfering?
I have also tried:
bootfile = subprocess.Popen("ls -l /jffs2/a.bin | cut -d '/' -f 4")
print bootfile
but it seems to break the script, as in I get nothing returned at all, have I done that right?
Also which of these is better and why? Are there other ways and what is the preferred way?

Using os.readlink (proposed by #kojiro) and os.path.basename for getting only the namefile:
os.path.basename(os.readlink('/jffs2/a.bin'))

kojiro's comment about os.readlink is probably what you want.
I am explaining what you were trying to implement.
os.system would return you exit status of the command run.
subprocess.Popen will create a pipe, so that you can capture the output of the command run.
Below line will capture output of the command run:
bootfile = subprocess.Popen(["bash","-c","ls -l /jffs2/a.bin | cut -d '/' -f 4"], stdout=subprocess.PIPE).communicate()[0]
More details at http://docs.python.org/library/subprocess.html

The right answer, as #kojiro says, is:
os.readlink('/jffs2/a.bin')
But if you really wanted to do this the complicated way, then in Python 2.7:
cmd = "ls -l /jffs2/a.bin | cut -d '/' -f 4"
bootfile = subprocess.check_output(cmd, shell=True)
Or on older Pythons:
cmd = "ls -l /jffs2/a.bin | cut -d '/' -f 4"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
bootfile = p.communicate()[0]
if p.returncode != 0:
raise Exception('It failed')