I am trying to assign to a variable the fingerprint of a pgp key in a bash subprocess of a python script.
Here's a snippet:
import subprocess
subprocess.run(
'''
export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"
echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"
''',
shell=True, check=True,
executable='/bin/bash')
The code runs but echo shows an empty variable:
KEY FINGERPRINT IS:
and if I try to use that variable for other commands I get the following error:
gpg: key "" not found: Not found
HOWEVER, if I run the same exact two lines of bash code in a bash script, everything works perfectly, and the variable is correctly assigned.
What is my python script missing?
Thank you all in advance.
The problem is the backslashes in your sed command. When you paste those into a Python string, python is escaping the backslashes. To fix this, simply add an r in front of your string to make it a raw string:
import subprocess
subprocess.run(
r'''
export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"
echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"
''',
shell=True, check=True,
executable='/bin/bash')
in order to run 2 commands in subprocess you need to run them one after each other or use ;
import subprocess
ret = subprocess.run('export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"; echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"', capture_output=True, shell=True)
print(ret.stdout.decode())
you can use popen:
commands = '''
export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"
echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands)
print out
Related
import subprocess
cmd = "source ~/.bash_profile || echo hello"
proc = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
(out, err) = proc.communicate()
code = proc.returncode
print(f"code:{code}, out:{out}, err:{err}")
Output:
code:1, out:b'', err:b'/bin/sh: /Users/xxx/.bash_profile: No such file or directory\n'
I have no bash_profile, but why doesn't out contain "hello"?
It could be difference between bash and sh
You can reproduce it without python. It also makes sense python relies on /bin/sh by default, so it exposes the behavior
#!/bin/sh
. abc || echo hello
# output: ./test.sh: line 3: .: abc: file not found
#!/bin/bash
. abc || echo hello
# output: ./test.sh: line 3: abc: No such file or directory
# hello
probably changing the default setting to /bin/bash by the executable argument can somehow correct it
cmd = "source ~/.bash_profile || echo hello"
proc = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
executable="/bin/bash")
This behavior does not seem to be documented in the bash man page, but the open standard states: If no readable file is found, a non-interactive shell shall abort; an interactive shell shall write a diagnostic message to standard error, but this condition shall not be considered a syntax error. (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_18_01) When running non-interactively from python, the shell is simply aborting and not running the code after the ||
I am trying to store the output of a cmd command as a variable in python.
To achieve this i am using os.system() but os.system() just runs the process,it doesn't capture the output.
import os
PlatformName = os.system("adb shell getprop | grep -e 'bt.name'")
DeviceName = os.system("adb shell getprop | grep -e '.product.brand'")
DeviceID = os.system("adb shell getprop | grep -e 'serialno'")
Version = os.system("adb shell getprop | grep -e 'version.release'")
print(PlatformName)
print(DeviceName)
print(DeviceID)
print(Version)
Then i tried to use the subprocess module.
import subprocess
import os
PlatformName = subprocess.check_output(["adb shell getprop | grep -e 'bt.name'"])
DeviceName = subprocess.check_output(["adb shell getprop | grep -e '.product.brand'"])
DeviceID = subprocess.check_output(["adb shell getprop | grep -e 'serialno'"])
Version = subprocess.check_output(["adb shell getprop | grep -e 'version.release'"])
print(PlatformName)
print(DeviceName)
print(DeviceID)
print(Version)
I am getting the following error
FileNotFoundError: [WinError 2] The system cannot find the file
specified
How can I store the output of the command as a variable?
The issues here:
passing arguments like this (string in a list, with spaces) is really not recommended
passing arguments like this need shell=True for it to have a slight chance to work, and shell=True is known for security issues (and other issues as well, like non-portability)
grep is not standard on windows, and the pattern is a regex which means you'd probably have to escape . ("bt\.name").
when not found grep returns 1 and would make check_output fail.
when found grep returns match(es), and a newline, that you'd have to strip
I'd rewrite this:
PlatformName = subprocess.check_output(["adb shell getprop | grep -e 'bt.name'"])
as:
output = subprocess.check_output(["adb","shell","getprop"])
platform_name = next((line for line in output.decode().splitlines() if "bt.name" in line),"")
The second line is a "native" version of grep (without regexes). It returns the first occurrence of "bt.line" in the output lines or empty string if not found.
You don't need grep here (the above is not strictly equivalent, as it yields the first occurrence, not all the occurrences but that should be okay on your case). And your clients may not have grep installed on Windows.
Hey I got the same problem as you. Sub-process can do what you want even with the shell=False. The trick is the communicate() method.
with subprocess.Popen(cmdCode,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd = workingDir,
bufsize=1,
universal_newlines = True) as proc:
#output is stored in proc.stdout
#errors are stored in proc.stderr
Now you just need a little function to scan the proc.stdout for the information you need: PlatformName, etc
I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)
When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)
The line awk -F'[][]' '/dB/ { print $2 }' <(amixer sget Master) in bash returns my system's current volume (e.g. "97%").
I tried to incorporate this in Python 3
#!/usr/bin/env python3
import subprocess
command = "awk -F'[][]' '/dB/ { print $2 }' <(amixer sget Master)"
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()
print(output)
However the output from the shell returns
/bin/sh: 1: Syntax error: "(" unexpected
b''
Why does this fail and how do I fix my code?
As already pointed out, the syntax you are using is a bash syntax (a.k.a. bashism). The default shell used in subprocess.Popen is /bin/sh & it does not support process substitution.
You can specify the shell to be used via executable argument.
Try this:
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, executable="/bin/bash").stdout.read()
Because you are using bashism in form of a process substitution, and your /bin/sh doesn't support that:
<(...)
Changing this to a pipe should solve your problem:
command = "amixer sget Master | awk -F'[][]' '/dB/ { print $2 }'"
Alternative you can start bash from within sh:
command = "bash -c 'amixer sget Master | awk -F'\\''[][]'\\'' '\\''/dB/ { print $2 }'\\'''"
But as you will soon realize, quoting and escaping will become a nightmare
How do I place output of bash command to Python variable?
I am writing a Python script, which I want to enter the output of
bash command:
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat'|wc -l, and place it to Python variable, let say R.
After that I want to do, Python if R != 0
then run some Linux command.
How do I achieve that?
There are various options, but the easiest is probably using subprocess.check_output() with shell=True although this can be security hazard if you don't fully control what command is passed in.
import subprocess
var = subprocess.check_output('rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat'|wc -l', shell = True)
var = int(var)
You need to use shell=True as otherwise the pipes would not be interpreted.
If you need more control you might want to look at plumbum where you can do:
from plumbum.cmd import rpm, grep, wc
chain = rpm["-qa", "--qf", r"%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n"] | grep["-v", "Red Hat"] | wc["-l"]
R = int(chain())
Although I would probably not invoke wc and get the whole output and count its length within python (easier to check that you got just the lines that you expected, piping through wc -l throws away all of the details)
I would recommend envoy primarily because the API is much more intuitive to use for 90% of the use cases.
r = envoy.run('ls ', data='data to pipe in', timeout=2)
print r.status_code # returns status code
print r.std_out # returns the output.
See the Envoy Github page for more details.
You can use stdin.
#!/usr/bin/python
import sys
s = sys.stdin.read()
print s
Then you will run a bash command like this
echo "Hello" | ./myscript.py
Output
Hello
You can replace shell pipeline using Popen:
from subprocess import PIPE,Popen
p1 = Popen(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'],stdout=PIPE)
p2 = Popen(["grep", "-v", 'Red Hat'],stdin=p1.stdout,stdout=PIPE)
p1.stdout.close()
p3 = Popen(["wc", "-l"],stdin=p2.stdout,stdout=PIPE)
p2.stdout.close()
out,err = p3.communicate()
If you just want to check if grep returned any matches then forget the wc - l and just check what grep returns:
p1 = Popen(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'],stdout=PIPE)
p2 = Popen(["grep", "-v", 'Red Hat'],stdin=p1.stdout,stdout=PIPE)
p1.stdout.close()
out,err = p2.communicate()
if out:
...
Or just use check_output to run the rpm command and check the string for "Red Hat":
out = check_output(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'])
if "Red Hat" not in out:
....
Which is the same as inverting the search with grep -v then checking if there are any matches with wc.