Develop Reference

Can't assign bash variable in python subprocess

I am trying to assign to a variable the fingerprint of a pgp key in a bash subprocess of a python script.
Here's a snippet:
import subprocess
subprocess.run(
'''
export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"
echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"
''',
shell=True, check=True,
executable='/bin/bash')
The code runs but echo shows an empty variable:
KEY FINGERPRINT IS:
and if I try to use that variable for other commands I get the following error:
gpg: key "" not found: Not found
HOWEVER, if I run the same exact two lines of bash code in a bash script, everything works perfectly, and the variable is correctly assigned.
What is my python script missing?
Thank you all in advance.

The problem is the backslashes in your sed command. When you paste those into a Python string, python is escaping the backslashes. To fix this, simply add an r in front of your string to make it a raw string:
import subprocess
subprocess.run(
r'''
export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"
echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"
''',
shell=True, check=True,
executable='/bin/bash')

in order to run 2 commands in subprocess you need to run them one after each other or use ;
import subprocess
ret = subprocess.run('export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"; echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"', capture_output=True, shell=True)
print(ret.stdout.decode())
you can use popen:
commands = '''
export KEYFINGERPRINT="$(gpg --with-colons --fingerprint --list-secret-keys | sed -n 's/^fpr:::::::::\([[:alnum:]]\+\):/\1/p')"
echo "KEY FINGERPRINT IS: ${KEYFINGERPRINT}"
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands)
print out

Cant use grep in subprocess command

I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)

When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)

Error using shlex and subprocess [duplicate]

This question already has answers here:
How do I use subprocess.Popen to connect multiple processes by pipes?
(9 answers)
Closed 7 years ago.
Hi I am trying to run this command in python's subprocess with shlex split, however, I haven't found anything helpful for this particular case :
ifconfig | grep "inet " | grep -v 127.0.0.1 | grep -v 192.* | awk '{print $2}'
I get an with ifconfig error because the split with the single and double quotes and even the white space before the $ sign are not correct.
Please Help.

You can use shell=True (shell will interpret |) and triple quote string literal (otherwise you need to escape ", ' inside the string literal):
import subprocess
cmd = r"""ifconfig | grep "inet " | grep -v 127\.0\.0\.1 | grep -v 192\. | awk '{print $2}'"""
subprocess.call(cmd, shell=True)
or you can do it in harder way (Replacing shell pipeline from subprocess module documentation):
from subprocess import Popen, PIPE, call
p1 = Popen(['ifconfig'], stdout=PIPE)
p2 = Popen(['grep', 'inet '], stdin=p1.stdout, stdout=PIPE)
p3 = Popen(['grep', '-v', r'127\.0\.0\.1'], stdin=p2.stdout, stdout=PIPE)
p4 = Popen(['grep', '-v', r'192\.'], stdin=p3.stdout, stdout=PIPE)
call(['awk', '{print $2}'], stdin=p4.stdout)

write bash command in python script [duplicate]

This question already has answers here:
Perform commands over ssh with Python
(17 answers)
Closed 8 years ago.
Bash script:
ACTIVE_MGMT_1=`ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
STANDBY_MGMT_1=`ssh -n ${MGMT_IP_2} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " S " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
MGMT_HOSTNAME_1=`ssh -n ${MGMT_IP_1} hostname 2>/dev/null`;
MGMT_HOSTNAME_2=`ssh -n ${MGMT_IP_2} hostname`2>/dev/null;
ssh -n ${MGMT_IP_1} ". .bash_profile; if [ ! -d "$MGMT_FOLDER" ]; then mkdir $MGMT_FOLDER; fi " 2>/dev/null 1>&2
ssh -n ${MGMT_IP_2} ". .bash_profile; if [ ! -d "$MGMT_FOLDER" ]; then mkdir $MGMT_FOLDER; fi " 2>/dev/null 1>&2
if [ -z "${ACTIVE_MGMT_1}" ] && [ -z "${STANDBY_MGMT_1}" ]; then
log "ERROR" "No active blade on Site: ${SITE_NAME}, first cluster. Skipping";
continue;
fi
log "INFO" "First Active blade: ${ACTIVE_MGMT_1} , standby: ${STANDBY_MGMT_1}";
log "INFO" "First Mng $MGMT_HOSTNAME_1 $MGMT_HOSTNAME_2";
i start to translate it to python:
#Check active MGMT on cluster 1
sp = subprocess.Popen(['ssh', '-n', '10.128.136.36', '. .bash_profile; xms sho proc TRAF.*', 'egrep', 'A'], stdout=subprocess.PIPE, stderr=None)
#ACTIVE_MGMT_1=`ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
sp = subprocess.Popen(['ssh', '-n', '10.128.136.36', '. .bash_profile; xms sho proc TRAF.*'], stdout=subprocess.PIPE, stderr=None)
#STANDBY_MGMT_1=`ssh -n ${MGMT_IP_2} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " S " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
any advise how to do the other lines?

You could consider using a python module like paramiko, and issue your command like their example:
import paramiko, base64
key = paramiko.RSAKey(data=base64.decodestring('AAA...'))
client = paramiko.SSHClient()
client.get_host_keys().add('ssh.example.com', 'ssh-rsa', key)
client.connect('ssh.example.com', username='strongbad', password='thecheat')
stdin, stdout, stderr = client.exec_command('. .bash_profile; xms sho proc TRAF.*')
for line in stdout:
# do your grep/awk stuff here!
print '... ' + line.strip('\n')
client.close()
I never used that module in the wild, so I'm not sure that you can do exec_command() with multiple commands, but I guess it could be a good idea to write a little script on the distant host that sets up the environment so that everything can work well, and just call that script.
Another reason why this is a good idea is that you open only one SSH connection to the server instead of opening/closing several sessions.

Python escaping sed and bash command with subprocess

Question:
How do I use sed with python successfully? I have to run this command on a remote server to get a list of comma delimited hosts. When ran from bash I get what I want which is something like host1, host2, host3
Here is what I have:
process = subprocess.Popen(["ssh $USER#mychefserver knife search node "chef_environment:*" | grep -i "node name" | egrep -i "stuff|ruff" | uniq -u | sort -n | cut -d ":" -f 2 | sed -e 's/^[ \t]*//' | tr '\n' ', '
"], shell=False, stdout=PIPE)
I know I'll have to escape the \n, \t, etc, but I'm having trouble with the rest. Whenever I try to run it from my Python script I get an error for invalid syntax even though I've tried a cornucopia of escapes.

You string quoting is broken as you use " inside a double quoted string. You have to escape the " like \". Further note, that most of the double quotes in command line can be replaced by single quotes '. The following code should work:
process = subprocess.Popen(["ssh $USER#mychefserver knife search node \"chef_environment:*\" | grep -i 'node name' | egrep -i 'stuff|ruff' | uniq -u | sort -n | cut -d':' -f 2 | sed -e 's/^[ \t]*//' | tr '\n' ', '"], shell=False, stdout=subprocess.PIPE)