I was trying perform replace using sed in VMkernel. I used the following command,
sed s/myname/sample name/g txt.txt
I got an error saying sed: unmatched '/'.
I replaced space with \. It worked.
When I tried the same using python,
def executeCommand(cmd):
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
print (output.decode("utf-8"))
executeCommand('sed s/myname/sample\ name/g txt.txt')
I am getting the error sed: unmatched '/' again. I used \s instead of space I am getting the name replaced with samplesname.
How can I replace a string with space?
The simplest thing would be to not be smart about splitting the command:
executeCommand(['sed', 's/myname/sample name/g', 'txt.txt'])
Otherwise you are opening a can of worms, effectively playing a shell parser role.
Alternatively you may run the command in a shell and let the shell parse and run the command:
import subprocess
def executeCommand(cmd):
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
# Or:
# This will run the command in /bin/bash (instead of /bin/sh)
process = subprocess.Popen(['/bin/bash', '-c', cmd], stdout=subprocess.PIPE)
output, error = process.communicate()
print (output.decode("utf-8"))
executeCommand("sed 's/myname/sample name/g' txt.txt")
Related
I'm trying to setup a system to run some commands on VM's in google cloud, in my case we want to run a tcpdump at a certain time using the 'at' command. Right now I'm just trying to execute any commands succesfully, when I have to pass arguments along with the command and getting confusing behaviour, which appears to be that the command, and the arguments are executed as a single long command instead of seperate arguements.
I first tried in bash, and thinking my issue was one of quoting, I moved to using python to hopefully make things easier to understand, but I appear to be hitting the same issue and figure I must be doing something wrong.
I have the following functions defined in python, and call them
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
def runCapture(project, instance, zone, time, duration):
## Run capture against server
print ("Running capture against Project: " + project + ", Instance: " + instance + ", Zone: " + zone, "at: " + time, "for " + str(duration) + " minutes")
## First connect, schedule capture
## Connect again, schedule upload of capture at capture time + duration time + some overrun.
## gcloud compute ssh --project=${PROJECT} ${INSTANCE} --zone="${ZONE}" --command="...do stuff..." --tunnel-through-iap
## CMD=\${1:-"/usr/sbin/tcpdump -nn -i ens4 -G \$(( ${DURATION}*60 )) -W 1 -w ./\$(uname -n)-%Y-%m-%d_%H.%M.%S.pcap"}
total_time=str(duration*60)
command="/bin/bash -c 'echo \"hello world\"'"
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
print(path, end="")
The resulting errors are as follows:
bash: /bin/bash -c 'echo hello: No such file or directory
Traceback (most recent call last):
File "./ingressCapture.py", line 79, in <module>
results = runCapture(project, instance, zone, time, duration)
File "./ingressCapture.py", line 33, in runCapture
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
File "./ingressCapture.py", line 17, in execute
raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcloud', 'compute', 'ssh', 'tbtst-test3-app-egress-nztw', '--project=devops-tb-sandbox-250222', '--zone=europe-west1-b', '--tunnel-through-iap', '--command="/bin/bash -c \'echo "hello world"\'"']' returned non-zero exit status 127.
It appears to me, that instead of invoking the bash shell and running the echo command, it is instead invoking a command that includes the bash shell and then all the arguments too. I have a bash shell when I login normally via SSH, and can run the commands manually (and they work). Why are the arguments for the command from --command="....." getting called like this and how do I prevent it?
I'm pretty sure your problem is that you have too many quotes.
When you write --command="bash -c 'echo \"Hello World\"'" on the command line, the shell internally marks all the stuff inside the quotes as being in a quoted state and then removes the quotes. The actual argument that ends up going to the program is --command=bash -c 'echo "Hello World"' as a single string in argv (or your language's equivalent).
Try putting import sys ; print(sys.argv[1]) inside a small python script and calling it with ./test.py --command="bash -c 'echo \"Hello World\"'" to see for yourself.
However, in your arglist to subprocess, you're forming this string: --command="/bin/bash -c 'echo "hello world"'", presumably because you thought you needed to match what you'd normally type on the command line. You can see this in the stacktrace (minus the escaped single quotes, since that's syntax highlighting from python). Since python does not perform quote removal, those quotes are going through to the other side of your ssh connection where the login shell is attempting to reparse it as a shell command. The first "word" on the other end of the connection is /bin/bash -c 'echo hello because of those extra quotes so the shell attempts to find a command with that name on the path, and it clearly doesn't exist.
What you need to put into your arglist for subprocess is simply "--command="+command.
I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)
When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)
My python script needs to invoke a program, detect if it failed (eg, result != 0) and send the output of the program to both stdout like normal plus a log file.
My default shell is bash. I'm using Python 2.7.9
To send output to both stdout and a file I'd normally use tee:
result = subprocess.call('some_program --an-option | tee -a ' + logfile , shell=True)
However, the pipe in bash will return true even if the first command fails, so this approach fails to detect if the command fails.
If I try to use set -o pipefail in the command (so that the result will indicate if the first command fails) like this:
result = subprocess.call('set -o pipefail && some_program --an_option | tee -a ' + logfile , shell=True)
I get the error /bin/sh: 1: set: Illegal option -o pipefail
Is there a way in python to invoke a command, send the output to both the normal stdout console and a logfile, and still detect if the command failed?
Note: we have to continue sending some_program's output to stdout since stdout is being sent to a websocket.
I get the error /bin/sh: 1: set: Illegal option -o pipefail
Pass executable='/bin/bash' otherwise /bin/sh is used.
You could implement tee in pure Python:
#!/usr/bin/env python2
import sys
from subprocess import Popen, PIPE
chunk_size = 1 << 13
p = Popen(["some_program", "--an-option"], stdout=PIPE, bufsize=1)
with p.stdout, open('logfile', 'ab') as logfile:
for chunk in iter(lambda: p.stdout.read(chunk_size), b''):
sys.stdout.write(chunk)
logfile.write(chunk)
if p.wait() != 0:
raise Error
My preference would to to send stdout to a pipe, and then read the pipe in the Python code. The Python code can write to stdout, a file, etc as required. It would also enable you to set shell=False as setting it to True is a potential security issue, as mentioned in the documentation.
However, the pipe in bash will return true even if the first command
fails, so this approach fails to detect if the command fails.
That is not true.
But I think you mean: the 'some_program --an-option | tee -a ' + logfile exit status code always is 0 even though fails in any command part.
Well, using multiple commands (when using && or ||) or connecting multiple commands together via pipes causes unreliable exit status code when returned.
Regardless, in the command: some_program --an-option | tee -a ' + logfile logfile is not written if some_program fails. So you don't need to worry regarding exit code.
Anyway the best way to do pipe along with subprocess is creating Popen objects ans handling stdout and stdin:
import subprocess as sp
STATUS_OK = 0
logfile = '/tmp/test.log'
commands = {
'main' : 'ls /home',
'pipe_to': 'tee -a ' + logfile
}
process = sp.Popen(commands['main'], shell=True, stdout=sp.PIPE)
# explicitly force waits till command terminate, set and return exit status code
process.wait()
if process.returncode == STATUS_OK:
stdoutdata = process.communicate()[0]
# pipe last command output to "tee" command
sp.Popen(commands['pipe_to'], stdin=sp.PIPE, shell=1).communicate(stdoutdata)
else:
# do something when command fails 'ls /hom' (in this case) fails
pass
That is it!
I the last Popen we invoke Popen.communicate() to send the last output from ls command to tee command STDIN.
In the Python doc there's a tiny tutorial called Replacing shell pipeline, maybe you want take a look.
I'm using Python 2.7.3.
I have a function that runs tesseract as a command line. Everything is working fine and now I would like to add a new parameter to the command -l rus (signifying russian language). Eventhough this works on my commandline, it doesn't seem to work from Python.
Command line:
$ /usr/local/bin/tesseract /Users/anthony/Downloads/rus.png outfile -l rus && more outfile.txt
Tesseract Open Source OCR Engine v3.02.02 with Leptonica
Полу-Милорд, полу-купец,
Полу-мудрец, полу-невежда,
Полу-подлец, но есть надежда,
Что будет полным наконец.
Python function
def ocr(self,path):
path = "/Users/anthony/Downloads/rus.png"
process = subprocess.Popen(['/usr/local/bin/tesseract', path,'outfile','-l rus'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, err = process.communicate()
print err
print out
with open('outfile.txt', 'r') as handle:
contents = handle.read()
os.remove(temp.name + '.txt')
os.remove(temp.name)
return contents, out
the above returns "HOIIY nony HOIIY nony Hony no ecTb HHJICXQRI 6y11e" which suggests that the -l rus flag is being ignored.
Question
How can I execute the following command as a python subprocess?
/usr/local/bin/tesseract /Users/anthony/Downloads/rus.png outfile -l rus
You need to split the '-l rus' argument to two separate ones to make sure it's parsed correctly by the program:
process = subprocess.Popen(
['/usr/local/bin/tesseract', path, 'outfile', '-l', 'rus'],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
It might be handy to use str.split() or shlex.split() for this:
cmd = '/usr/local/bin/tesseract /Users/anthony/Downloads/rus.png outfile -l rus'
process = subprocess.Popen(
cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
process = subprocess.Popen('/usr/local/bin/tesseract '+path+' outfile -l rus', stdout=subprocess.PIPE, stderr=subprocess.STDOUT,shell=True)
You can run it with shell=True.
I have the following script:
import subprocess
arguments = ["d:\\simulator","2332.txt","2332.log", "-c"]
output=subprocess.Popen(arguments, stdout=subprocess.PIPE).communicate()[0]
print(output)
which gives me b'' as output.
I also tried this script:
import subprocess
arguments = ["d:\\simulator","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments,stdout=subprocess.PIPE)
process.wait()
print(process.stdout.read())
print("ERROR:" + str(process.stderr))
which gives me the output: b'', ERROR:None
However when I run this at the cmd prompt I get a 5 lines of text.
d:\simulator atp2332.txt atp2332.log -c
I have added to simulator a message box which pops up when it launches. This is presented for all three cases. So I know that I sucessfully launch the simulator. However the python scripts are not caturing the stdout.
What am I doing wrong?
Barry.
If possible (not endless stream of data) you should use communicate() as noted on the page.
Try this:
import subprocess
arguments = ["d:\\simulator","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
sout, serr = process.communicate()
print(sout)
print(serr)
The following code gives me text output on stdout.
Perhaps you could try it, and then substitute your command for help
import subprocess
arguments = ["help","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.wait()
print 'Return code', process.returncode
print('stdout:', process.stdout.read())
print("stderr:" + process.stderr.read())