This question already has answers here:
Perform commands over ssh with Python
(17 answers)
Closed 8 years ago.
Bash script:
ACTIVE_MGMT_1=`ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
STANDBY_MGMT_1=`ssh -n ${MGMT_IP_2} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " S " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
MGMT_HOSTNAME_1=`ssh -n ${MGMT_IP_1} hostname 2>/dev/null`;
MGMT_HOSTNAME_2=`ssh -n ${MGMT_IP_2} hostname`2>/dev/null;
ssh -n ${MGMT_IP_1} ". .bash_profile; if [ ! -d "$MGMT_FOLDER" ]; then mkdir $MGMT_FOLDER; fi " 2>/dev/null 1>&2
ssh -n ${MGMT_IP_2} ". .bash_profile; if [ ! -d "$MGMT_FOLDER" ]; then mkdir $MGMT_FOLDER; fi " 2>/dev/null 1>&2
if [ -z "${ACTIVE_MGMT_1}" ] && [ -z "${STANDBY_MGMT_1}" ]; then
log "ERROR" "No active blade on Site: ${SITE_NAME}, first cluster. Skipping";
continue;
fi
log "INFO" "First Active blade: ${ACTIVE_MGMT_1} , standby: ${STANDBY_MGMT_1}";
log "INFO" "First Mng $MGMT_HOSTNAME_1 $MGMT_HOSTNAME_2";
i start to translate it to python:
#Check active MGMT on cluster 1
sp = subprocess.Popen(['ssh', '-n', '10.128.136.36', '. .bash_profile; xms sho proc TRAF.*', 'egrep', 'A'], stdout=subprocess.PIPE, stderr=None)
#ACTIVE_MGMT_1=`ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
sp = subprocess.Popen(['ssh', '-n', '10.128.136.36', '. .bash_profile; xms sho proc TRAF.*'], stdout=subprocess.PIPE, stderr=None)
#STANDBY_MGMT_1=`ssh -n ${MGMT_IP_2} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " S " |awk '/TRAF/{print $1}' |cut -d "." -f2`;
any advise how to do the other lines?
You could consider using a python module like paramiko, and issue your command like their example:
import paramiko, base64
key = paramiko.RSAKey(data=base64.decodestring('AAA...'))
client = paramiko.SSHClient()
client.get_host_keys().add('ssh.example.com', 'ssh-rsa', key)
client.connect('ssh.example.com', username='strongbad', password='thecheat')
stdin, stdout, stderr = client.exec_command('. .bash_profile; xms sho proc TRAF.*')
for line in stdout:
# do your grep/awk stuff here!
print '... ' + line.strip('\n')
client.close()
I never used that module in the wild, so I'm not sure that you can do exec_command() with multiple commands, but I guess it could be a good idea to write a little script on the distant host that sets up the environment so that everything can work well, and just call that script.
Another reason why this is a good idea is that you open only one SSH connection to the server instead of opening/closing several sessions.
Related
I need to run a shell command inside subprocess.Popen in Python.
The command is:
$ virsh dumpxml server1 | grep 'source file' | awk -F\' '{print $2}'
The output is:
/vms/onion.qcow2
I'm having two challenges with the above command:
1) The command is inside a loop, and where you see 'server1', it is a variable that will have a server name.
2) Python is complaining about KeyError: 'print $2'
Here is what I have so far:
proc = subprocess.Popen(["virsh dumpxml {0} | grep 'source file' | awk -F\' '{print $2}'".format(vm)], stdout=subprocess.PIPE, shell=True)
stdout = proc.communicate()[0]
Thanks in advance.
While it's possible use libvirt directly from python, your problem is that { is the format string, and surrounds print $2 in your awk script as well, so you have to escape those braces like
proc = subprocess.Popen(["virsh dumpxml {0} | grep 'source file' | awk -F\\' '{{print $2}}'".format(vm)], stdout=subprocess.PIPE, shell=True)
stdout = proc.communicate()[0]
This question already has an answer here:
sed: unterminated 's' command`
(1 answer)
Closed 4 years ago.
Below command runs fine in bash.
/folk/vlm/commandline/./vlmTool find -l all | grep -e "^Target ID" -e "City" -e "Zone" | sed "s#.*City.*#&\n#g" > new.txt
But in python when I try to execute the same command I get:
['sed', 's#.*City.*#&\n#g']
**sed: -e expression #1, char 12: unterminated `s' command**
code:
#!/usr/local/bin/python -tt
import subprocess
import shlex
cmd = '/folk/vlm/commandline/./vlmTool find -l all | grep -e "^Target ID" -e "City" -e "Zone" | sed "s#.*City.*#&\n#g" > new.txt'
cmd2= "/folk/vlm/commandline/./vlmTool find -l all"
args1 = shlex.split(cmd2)
cmd3 = 'grep -e "^Target ID" -e "City" -e "Zone"'
args2 = shlex.split(cmd3)
cmd4 = 'sed "s#.*City.*#&\n#g"'
args3 = shlex.split(cmd4)
print args3
p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(args3, stdin=p2.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
p2.stdout.close()
output = p3.communicate()[0]
the \n sequence is understood in python as an escape sequence for a newline; so sed is thinking you're done with the s command and starting a new one. to get the backslash included in the string, either escape it, as in ...\\n... or use a raw string: r's#.*City.*#&\n#g'
I am working on Python along with bash shell script. I need to execute shell script from the Python script. I am successfully able to do that.. I need to have a shell script in a single line in a JSON String..
Below is an example which is working fine for a simple shell script which I have made it in a JSON document...
import os
import json
import subprocess
jsonData = '{"pp": [0,3,5,7,9], "sp": [1,2,4,6,8]}'
jj = json.loads(jsonData)
os.putenv( 'jj3', ' '.join( str(v) for v in jj['pp'] ) )
os.putenv( 'jj4', ' '.join( str(v) for v in jj['sp'] ) )
jsonStr = '{"script":"#!/bin/bash \\n echo Hello World \\n "}'
j = json.loads(jsonStr)
shell_script = j['script']
print "start"
proc = subprocess.Popen(shell_script, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if proc.returncode != 0:
print "Shell script gave some error"
print stdout
else:
print "end"
print stdout
Now I have a below shell script which I need to represent it in a single line in a JSON document as I have done for above Hello World use case..
#!/bin/bash
set -e
readonly PRIMARY=/tech01/primary
readonly SECONDARY=/tech02/secondary
readonly LOCATION=(machineA machineB)
readonly MAPPED_LOCATION=/bat/data/snapshot
HOSTNAME=$hostname
dir1=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[0]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[1]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
echo $dir1
echo $dir2
length1=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[0]} "ls '$dir1' | wc -l")
length2=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[1]} "ls '$dir2' | wc -l")
echo $length1
echo $length2
if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
rm -rf $PRIMARY/*
rm -rf $SECONDARY/*
for el in $primary_partition
do
scp david#${LOCATION[0]}:$dir1/weekly_8880_"$el"_5.data $PRIMARY/. || scp david#${LOCATION[1]}:$dir2/weekly_8880_"$el"_5.data $PRIMARY/.
done
fi
I have made the above shell script in a single line JSON document like this but this doesn't work and I always get an error
jsonStr = '{"script":"#!/bin/bash \n set -e \n readonly PRIMARY=/tech01/primary \n readonly SECONDARY=/tech02/secondary \n readonly LOCATION=(machineA machineB) \n readonly MAPPED_LOCATION=/bat/data/snapshot \n HOSTNAME=$hostname \n dir1=$(ssh -o \"StrictHostKeyChecking no\" david#${LOCATION[0]} ls -dt1 \"$MAPPED_LOCATION\"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1) \n dir2=$(ssh -o \"StrictHostKeyChecking no\" david#${LOCATION[1]} ls -dt1 \"$MAPPED_LOCATION\"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1) \n echo $dir1 \n echo $dir2 \n length1=$(ssh -o \"StrictHostKeyChecking no\" david#${LOCATION[0]} \"ls \'$dir1\' | wc -l\") \n length2=$(ssh -o \"StrictHostKeyChecking no\" david#${LOCATION[1]} \"ls \'$dir2\' | wc -l\") \n echo $length1 \n echo $length2 \n if [ \"$dir1\" = \"$dir2\" ] && [ \"$length1\" -gt 0 ] && [ \"$length2\" -gt 0 ] \n then \n rm -rf $PRIMARY/* \n rm -rf $SECONDARY/* \n for el in $primary_partition \n do \n scp david#${LOCATION[0]}:$dir1/t1_weekly_1680_\"$el\"_200003_5.data $PRIMARY/. || scp david#${LOCATION[1]}:$dir2/t1_weekly_1680_\"$el\"_200003_5.data $PRIMARY/. \n done \n fi"}'
This is the error I am getting -
ValueError: Invalid control character
Can anyone help me what wroing I am doing? And what's the right way to make above shell script in a Single line JSON document?
UPDATE:-
There is another twist into this which I thought its worth mentioning..
I need to store this single line shell script data in a Zookeeper node. So for Hello World shell script example case, I am storing by using the following code. I am using Curator library to write into Zookeeper.
client.create().creatingParentsIfNeeded().forPath("/be/wf/error1/v1/step1", "{\"description\":\"Hello World 1.\", \"script\":\"#!/bin/bash \\n set -e \\n echo Hello World 1 \\n\"}".getBytes());
And then I have my Python program to read the Zookeeper node data and then execute the shell script by extracting from the script tag..
Now I tried representing the same script as mentioned in the answer, in the above format, and my Eclipse started giving compilation errors on the string. So how do I represent the shell script as given in the answer in the above..
I guess I really need a JSON so that I can store it properly in Zookeeper node and then my Python program can read the same JSON data from the node and execute the shell script from the script tag.
First, consider whether you really need JSON. In your example code, you create a JSON string and then immediately decode it into a Python dict. Would it be simpler to just use a dict directly?
The problem with your current string is that you're not escaping your quotation marks properly. To avoid confusing multi-level escaping, use a triple-quoted string to represent the shell script and json.dumps to convert a dict into a JSON string:
import json
jsonstr = json.dumps({"script": """\
#!/bin/bash
set -e
readonly PRIMARY=/tech01/primary
readonly SECONDARY=/tech02/secondary
readonly LOCATION=(machineA machineB)
readonly MAPPED_LOCATION=/bat/data/snapshot
HOSTNAME=$hostname
dir1=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[0]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[1]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
echo $dir1
echo $dir2
length1=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[0]} "ls '$dir1' | wc -l")
length2=$(ssh -o "StrictHostKeyChecking no" david#${LOCATION[1]} "ls '$dir2' | wc -l")
echo $length1
echo $length2
if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
rm -rf $PRIMARY/*
rm -rf $SECONDARY/*
for el in $primary_partition
do
scp david#${LOCATION[0]}:$dir1/weekly_8880_"$el"_5.data $PRIMARY/. || scp david#${LOCATION[1]}:$dir2/weekly_8880_"$el"_5.data $PRIMARY/.
done
fi"""})
Alternatively, you could put the shell script in its own file and open the file to read the string from it.
I'm trying to grep a list of file from the "*.nasl" of "Openvas" which contains a certain port's number.
I can make it directly in the terminal with the command :
egrep --only-match '111' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl |cut -d ":" -f1
This command return all the name of the nasl file which contains 111.
like :
/home/gwvm/Openvas/var/lib/openvas/plugins/SolarWinds_TFTP.nasl:111
/home/gwvm/Openvas/var/lib/openvas/plugins/trojan_horses.nasl:111
and after the cut :
/home/gwvm/Openvas/var/lib/openvas/plugins/SolarWinds_TFTP.nasl
/home/gwvm/Openvas/var/lib/openvas/plugins/trojan_horses.nasl
But when I'm in python(3.1.3) the output give me an error :
egrep:/home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl: No such file or directory
i was thinking about a problem because of the "*.nasl" but when I'm trying with an existing file, same result.
Here is the part of code :
command = ("egrep --only-match '"+ str(port[0]) +"' "+ openvas_directory["locate"]["nasl"] + '*.nasl' + ' |cut -d ":" -f1 ')
process=sp.Popen(command,shell=True, stdout= sp.PIPE)
or
exec(command)
I was thinking too of a bad construction but wen I'm printing the command it gives me what i want :
egrep --only-match '111' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl |cut -d ":" -f1
If there are any idea!
from subprocess import PIPE, Popen
x = Popen('egrep --only-match \'111\' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl', stdout=PIPE, stderr=PIPE, shell=True)
y = Popen('cut -d ":" -f1', stdin=x.stdout, stdout=PIPE, stderr=PIPE, shell=True)
for row in y.stdout.readline():
print row
Or just use check_output()
And this is btw how you | in Popen ;)
Guidelines:
When using Popen, if you supply a command as a string, use shell=True.
If you however supply Popen with a list ['ls, '-l'] then use shell=False, that's just how it works.
If you're piping data, execute two different Popen's and use the output from the first command as stdin for the second command, this is equivilant to doing | in Linux.
I'm looking for the best way to use bash commands from within python. What ways are there? I know of os.system and subprocess.Popen.
I have tried these:
bootfile = os.system("ls -l /jffs2/a.bin | cut -d '/' -f 4")
print bootfile
This returns a.bin as expected but also it retuns 0 afterwards and so prints:
a.bin
0
with bootfile now being set to 0. The next time I print bootfile it just shows up as 0. Which is the exit value I guess, how do i stop this value interfering?
I have also tried:
bootfile = subprocess.Popen("ls -l /jffs2/a.bin | cut -d '/' -f 4")
print bootfile
but it seems to break the script, as in I get nothing returned at all, have I done that right?
Also which of these is better and why? Are there other ways and what is the preferred way?
Using os.readlink (proposed by #kojiro) and os.path.basename for getting only the namefile:
os.path.basename(os.readlink('/jffs2/a.bin'))
kojiro's comment about os.readlink is probably what you want.
I am explaining what you were trying to implement.
os.system would return you exit status of the command run.
subprocess.Popen will create a pipe, so that you can capture the output of the command run.
Below line will capture output of the command run:
bootfile = subprocess.Popen(["bash","-c","ls -l /jffs2/a.bin | cut -d '/' -f 4"], stdout=subprocess.PIPE).communicate()[0]
More details at http://docs.python.org/library/subprocess.html
The right answer, as #kojiro says, is:
os.readlink('/jffs2/a.bin')
But if you really wanted to do this the complicated way, then in Python 2.7:
cmd = "ls -l /jffs2/a.bin | cut -d '/' -f 4"
bootfile = subprocess.check_output(cmd, shell=True)
Or on older Pythons:
cmd = "ls -l /jffs2/a.bin | cut -d '/' -f 4"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
bootfile = p.communicate()[0]
if p.returncode != 0:
raise Exception('It failed')