Develop Reference

subprocess.check_output fails with CalledProcessError but error is empty string. Command works in terminal

I want to run the command ffprobe -i test.m4a -show_entries format=duration -v quiet -of csv="p=0". It works in the terminal and returns output code 0, but running it with subprocess, i.e.
subprocess.check_output(['ffprobe', '-i', 'test.m4a', '-show_entries', 'format=duration', '-v', 'quiet', '-of', 'csv="p=0"'])
raises a CalledProcessError - {Command} returned non-zero exit status 1.. I tried running this command in a try-except loop and printing the error details, but it just outputs as an empty byte string b''.

One way for debugging the issue is adding -report argument:
subprocess.check_output(['ffprobe', '-i', 'output.mp4', '-show_entries', 'format=duration', '-v', 'quiet', '-of', 'csv="p=0"', '-report'])
-report is used for creating a log file with name like ffprobe-20220811-232043.log.
The log files shows the following error:
[csv # 00000213297fe640] Failed to set option '"p' with value '0"' provided to writer context
The log files shows that the executed "shell command" is:
ffprobe -i output.mp4 -show_entries "format=duration" -v quiet -of "csv=\"p=0\"" -report
The solution is removing the quotes from "p=0":
subprocess.check_output(['ffprobe', '-i', 'output.mp4', '-show_entries', 'format=duration', '-v', 'quiet', '-of', 'csv=p=0'])

I recommend using the subprocess.run instead of check_output.
subprocess.run(command, stdout=output, encoding="utf-8")
Command = is the variable that houses the command you want , no need for separation using the commas.
stdout = Output = means that the output should be recorded in the file called (Output) which you can create beforehand.
encoding = just means to make sure it's encoded into texts from bytes

Not able to run command via wsl.exe

I'm trying to run this commands from a python script:
def raw(path_avd_py, path_avd, snp_name, out_file):
if OS == 'Windows':
cmd_raw = f"wsl.exe -e sh -c 'python3 {path_avd_py} -a {path_avd}
-s {snp_name} -o {out_file}'"
else:
cmd_raw = f'python3 {path_avd_py} -a {path_avd} -s {snp_name} -o {out_file}'
subprocess.Popen(cmd_raw, shell=True)
time.sleep(25)
return None
def idiffer(i_path, raw_1, raw_2, path, state):
if OS == 'Windows':
cmd_idiff = f"wsl.exe -e sh -c 'python3 {i_path} {raw_1} {raw_2}'"
[...]
file = os.path.join(path, f'{state}.idiff')
with open(file, 'w') as f:
subprocess.Popen(cmd_idiff, stdout=f, text=True)
If im executing cmd_raw with subprocess.run from a python-shell (Powershell), things are working. If im try running this via script, this exception occurs, using different shells:
-e sh: avdecrypt-master\avdecrypt.py: 1: Syntax error: Unterminated quoted string
-e bash: avdecrypt-master\avdecrypt.py: -c: line 0: unexpected EOF while looking for matching `''
avdecrypt-master\avdecrypt.py: -c: line 1: syntax error: unexpected end of file
I already tried os.system, os.run([list]) no change.
Thanks for the help!

For those who have a similar question, I found a solution, which is working for me:
Apparently calling scripts with some argv has to be in one single quotation mark and can be executed via run (in my case important, because the process has to be terminated). This leads to a form like:
cmd = ['wsl.exe', '-e', 'bash', '-c', '-a foo -b bar [...]']
subprocess.run(cmd, shell=True)
Lib shlex is helping here and formatting the strings like subprocess is needing it:
cmd_finished = shlex.split(cmd)
https://docs.python.org/3/library/shlex.html

Cant use grep in subprocess command

I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)

When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)

Parameters ignored in call()

I am trying to call ssh from a Python program but it seems to be ignoring the arguments.
This is the Python program:
#!/usr/bin/python
from subprocess import Popen, PIPE, call
vm_name = 'vmName with-space'
vm_host = 'user#192.168.21.230'
def ssh_prefix_list(host=None):
if host:
# return ["ssh", "-v", "-v", "-v", host]
return ["scripts/ssh_wrapper", "-v", "-v", "-v", host]
else:
return []
def start(vm_name, vm_host=None): # vm_host defaults to None
print "vm_host = ", vm_host
vbm_args = ssh_prefix_list(vm_host) + ["VBoxManage", "startvm", vm_name]
print vbm_args
return call(vbm_args, shell=True)
start(vm_name, vm_host)
The wrapper prints the number of arguments, their values, and calls ssh.
#!/bin/bash
echo Number of arguments: $#
echo ssh arguments: "$#"
ssh "$#"
This is the output.
$ scripts/vm_test.py
vm_host = stephen#192.168.21.230
['scripts/ssh_wrapper', '-v', '-v', '-v', 'stephen#192.168.21.230', 'VBoxManage', 'startvm', 'atp-systest Clone']
Number of arguments: 0
ssh arguments:
usage: ssh [-1246AaCfgKkMNnqsTtVvXxY] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-e escape_char] [-F configfile]
[-i identity_file] [-L [bind_address:]port:host:hostport]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-R [bind_address:]port:host:hostport] [-S ctl_path]
[-w local_tun[:remote_tun]] [user#]hostname [command]
This is on Python 2.5.

When you are using shell=True , you need to pass in a string, not a list of arguments. Try -
return call(' '.join(vbm_args), shell=True)
Also, you should consider constructing the string from start, rather than list.
When you pass a list to call() or Popen() with shell=True , only the first element in the list is actually called , and that is the reason you are seeing the wrapper called with 0 arguments.
You should also try first without using shell=True , since its a security hazard, as clearly stated in the documentation of subprocess -
Using shell=True can be a security hazard. See the warning under Frequently Used Arguments for details.

I think this might be it:
prefix_list = ssh_prefix_list(vm_host)
prefix_list.append(["VBoxManage startvm %s" % vm_name])
however I would strongly recommend using paramiko - it makes things much easier.

When you use call with shell=True, you need to pass a single string, not an array of strings. So:
call("scripts/ssh_wrapper -v -v -v "+host+" VBoxManage startvm "+vmname)

Passing escape characters to grep -e command via Python subprocess.Popen()

I want to call this command from Python using subprocess: grep -e '^commit [a-z0-9]{40}'
When I call this command directly in terminal it doesn't work unless I escape the braces with backslashes like: grep -e '^commit [a-z0-9]\{40\}'
When I try to pass this string, with the escape characters, to the command using Popen in python it doesn't work. Here is what I have tried:
grepCommand = ['grep', '-e', "'^commit [a-z0-9]\\{{{0}\\}}'".format("40")]
grepCommand = ['grep', '-e', "'^commit [a-z0-9]\\\\{{{0}\\\\}}'".format("40")]
grepCommand = ['grep', '-e', "^commit [a-z0-9]\\{{{0}\\}}".format("40")]
grepCommand = ['grep', '-e', "^commit [a-z0-9]\\\\{{{0}\\\\}}".format("40")]
How can I properly format this string in python, so that I can pass it to grep via Popen?

The list already is a parameter seperation, so extra quoting with ' is not necessary:
grepCommand = ['grep', '-e', r"^commit [a-z0-9]\{{{0}\}}".format("40")]