I'm using this command:
docker ps | awk '$2 ~ /^selenium/ { print $1 }'
which works fine in the shell, but when running it with sh -c it doesn't work and I'm getting this error:
awk: cmd. line:1: ~ /^selenium/ { print }
awk: cmd. line:1: ^ syntax error
The full command that I want to is part from a Python script:
os.popen("nsenter -t 1 -m -u -n -i sh -c \"docker ps | awk -F/ '\''$2 ~ /^selenium/ { print $1 }'\''\"")
It's probably some escaping problem but I couldn't solve any.
You've got several levels of quoting there that are causing problems. If you start by using Python triple-quotes on the outside (''' or """), you can reduce the amount of escaping you need to perform.
That gets us:
os.popen('''nsenter -t 1 -m -u -n -i sh -c "docker ps | awk -F/ '\$2 ~ /^selenium/ { print \$1 }'"''')
We still need to escape the $ because otherwise they would be escaped by the outer shell (the one that os.popen calls to run the command).
I'm a little supicious of the -F/ in your awk command, but I assume you've tested this out and confirmed it turns the output you want.
By using the subprocess module, which doesn't call /bin/sh by default, you can further reduce the escaping (at the expense of having to tokenize the command line yourself):
import subprocess
subprocess.check_output([
'nsenter', '-t', '1', '-m', '-u', '-n', '-i',
'sh', '-c', "docker ps | awk -F/ '$2 ~ /^selenium/ { print $1 }'"
])
Do you have trying: ... awk -F/ '\\$2 ~ /^s...
(double backslash before $2)
Related
Anyone can explain me why this
python -V | awk '{print $2}'
returns this
Python 2.7.5
instead of
2.7.5
What to do to return only the version number without "Python " ?
If you run
python -V >/dev/null
you will notice that you still get output! Apparently, python -V prints its output to stderr, not to stdout.
In a bourne-like shell, this should work:
python -V 2>&1 | awk '{print $2}'
How about using pure python command itself(I need to format it with dots in between though)
python -c 'import sys; print sys.version_info[0],sys.version_info[1],sys.version_info[2]'
OR as per Chris's comment use:
python -c 'import sys; print(".".join(map(str, sys.version_info[:3])))'
This question already has answers here:
Process substitution not allowed by Python's subprocess with shell=True?
(3 answers)
Closed 2 years ago.
I have this bash command I want to run from Python2.7:
time ( s=172.20.16 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )
I tried running it like this:
cmd = 'time ( s=172.20.16 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )'
#1. subprocess.call(cmd.split())
#2. subprocess.call(cmd, shell=True)
#3. os.system(cmd)
But all returned /bin/sh: 1: Syntax error: word unexpected (expecting ")"), while running it from bash worked prefectly. I also tried adding a /bin/bash to the head of the command, but that didn't work.
When using os.system('bash "{}"'.format(cmd)) it didn't crash with the previous error, but the loop unfolded incorecctly (it printed 1..254 instead of using them as the IP suffix)
I managed to make it work by saving the command in a bash script and then calling the script from python, but I would rather do that directly. What is the problem here?
shell=True uses /bin/sh. /bin/sh is not bash.
Leaving all the problems with the shell script in place, but invoking it with bash, would look like the following:
cmd = 'time ( s=172.20.16 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )'
subprocess.call(['bash', '-c', cmd])
Rewriting it to actually be a better shell script might instead look like:
cmd = r'''
time {
s=172.20.16
for ((i=1; i<=254; i++)); do
{ ping -n -c 1 -w 1 "$s.$i" >/dev/null 2>&1 && \
printf "%-16s %s\n" "$s.$i" "responded"
} &
done
wait
echo
}
'''
subprocess.call(['bash', '-c', cmd])
Note that we're using { ...; }, not ( ... ), for grouping (thus avoiding more subshell creations than necessary); and that we're always quoting substitutions.
you are splitting by space to construct the array of commands/parameters for the suprocess call method; but notice that there are parameters that include spaces, so it should count as a single parameter, not two (ie this one: "%-16s %s\n")
Try using subprocess as per this link Running Bash commands in Python
import subprocess
subprocess.call("{}".format(cmd).split())
I want to run a set of docker commands from python.
I tried creating a script like below and run the script from python using paramiko ssh_client to connect to the machine where the docker is running:
#!/bin/bash
# Get container ID
container_id="$(docker ps | grep hello | awk '{print $1}')"
docker exec -it $container_id sh -c "cd /var/opt/bin/ && echo $1 &&
echo $PWD && ./test.sh -q $1"
But docker exec ... never gets executed.
So I tried to run the below python script below, directly in the machine where the docker is running:
import subprocess
docker_run = "docker exec 7f34a9c1b78f /bin/bash -c \"cd
/var/opt/bin/ && ls -a\"".split()
subprocess.call(docker_run, shell=True)
I get a message: "Usage: docker COMMAND..."
But I get the expected results if I run the command
docker exec 7f34a9c1b78f /bin/bash -c "cd /var/opt/bin/ && ls -a"
directly in the machine
How to run multiple docker commands from the python script? Thanks!
You have a mistake in your call to subprocess.call. subprocess.call expects a command with a series of parameters. You've given it a list of parameter pieces.
This code:
docker_run = "docker exec 7f34a9c1b78f /bin/bash -c \"cd
/var/opt/bin/ && ls -a\"".split()
subprocess.call(docker_run, shell=True)
Runs this:
subprocess.call([
'docker', 'exec', '7f34a9c1b78f', '/bin/bash', '-c',
'"cd', '/var/opt/bin/', '&&', 'ls', '-a"'
], shell=True)
Instead, I believe you want:
subprocess.call([
'docker', 'exec', '7f34a9c1b78f', '/bin/bash', '-c',
'"cd /var/opt/bin/ && ls -a"' # Notice how this is only one argument.
], shell=True)
You might need to tweak that second call. I suspect you don't need the quotes ('cd /var/opt/bin/ && ls -a' might work instead of '"cd /var/opt/bin/ && ls -a"'), but I haven't tested it.
Following are a few methods worked:
Remove double quotes:
subprocess.call([
'docker', 'exec', '7f34a9c1b78f', '/bin/bash', '-c',
'cd /opt/teradata/tdqgm/bin/ && ./support-archive.sh -q 6b171e7a-7071-4975-a3ac-000000000241'
])
If you are not sure of how the command should be split up to pass it as an argument of subprocess method, shlex module:
https://docs.python.org/2.7/library/shlex.html#shlex.split
I need to use double quotes when passing a command to subprocess and can't use shell=true. The following is simplified version of the code for clarity:
def run_command(cmd):
if isinstance(cmd, str):
cmd = cmd.split()
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
output, error = proc.communicate()
if proc.returncode != 0:
raise subprocess.CalledProcessError(proc.returncode, cmd, error)
cmd = 'sh -c " sudo ls "'
run_command(cmd)
I tried the followings but still fails:
cmd = 'sh -c " sudo ls "' #sudo: Syntax error: Unterminated quoted string
cmd = 'sh -c "sudo ls"' #ls": Syntax error: Unterminated quoted string
cmd = 'sh -c sudo ls' #usage: sudo -h | -K | -k | -V
The use of a string input with .split() is going to get in your way here as split() will not respect your quotes.
Instead try Popen(['sh', '-c', 'sudo ls'])
Note that the double quotes are not present. This is because the entire string sudo ls (including space) is placed into one argv entry this way.
For example, in /tmp I have files ending in .txt, .doc, and .jpg that I'd like to delete in one step using shred and subprocess.
The following does the job:
subprocess.call('bash -c "shred -n 5 -uz /tmp/{*.txt,*.pdf,*.doc}"', shell=True)
How would I do this command without using shell=True. I've tried the following:
subprocess.call(['bash', '-c', '"shred -n 10 -uz /tmp/{*.txt,*.pdf,*.doc}"'])
subprocess.call(['bash', '-c', 'shred', '-n 10', '-uz', '/tmp/{*.txt,*.pdf,*.doc}'])
Any suggestions?
I believe that other guy is spot on (haven't tried it myself though). However if you ever find yourself having similar issues again shlex.split(s) might be helpful. It takes the string 's' and splits it "using shell-like syntax".
In [3]: shlex.split(s)
Out[3]: ['bash', '-c', 'shred -n 5 -uz /tmp/{*.txt,*.pdf,*.doc}']
subprocess.call(['bash', '-c', 'shred -n 10 -uz /tmp/{*.txt,*.pdf,*.doc}'])
You can tell how a command is expanded and split up with:
$ printf "Argument: %s\n" bash -c "shred -n 5 -uz /tmp/{*.txt,*.pdf,*.doc}"
Argument: bash
Argument: -c
Argument: shred -n 5 -uz /tmp/{*.txt,*.pdf,*.doc}
In the more general case (but overkill here), if you're ever in doubt of what's executed by something with which parameters, you can use strace:
$ cat script
import subprocess
subprocess.call('bash -c "shred -n 5 -uz /tmp/{*.txt,*.pdf,*.doc}"', shell=True)
$ strace -s 1000 -fe execve python script
...
execve("/bin/bash", ["bash", "-c", "shred -n 5 -uz /tmp/{*.txt,*.pdf,*.doc}"], [/* 49 vars */]) = 0
...
$
If the command is coming from a trusted source e.g., it is hardcoded then there is nothing wrong in using shell=True:
#!/usr/bin/env python
from subprocess import check_call
check_call("shred -n 10 -uz /tmp/{*.txt,*.pdf,*.doc}",
shell=True, executable='/bin/bash')
/bin/bash is used to support {} inside the command.
This command doesn't run /bin/sh