Develop Reference

awk command doesn't work under /bin/sh -c

I'm using this command:
docker ps | awk '$2 ~ /^selenium/ { print $1 }'
which works fine in the shell, but when running it with sh -c it doesn't work and I'm getting this error:
awk: cmd. line:1: ~ /^selenium/ { print }
awk: cmd. line:1: ^ syntax error
The full command that I want to is part from a Python script:
os.popen("nsenter -t 1 -m -u -n -i sh -c \"docker ps | awk -F/ '\''$2 ~ /^selenium/ { print $1 }'\''\"")
It's probably some escaping problem but I couldn't solve any.

You've got several levels of quoting there that are causing problems. If you start by using Python triple-quotes on the outside (''' or """), you can reduce the amount of escaping you need to perform.
That gets us:
os.popen('''nsenter -t 1 -m -u -n -i sh -c "docker ps | awk -F/ '\$2 ~ /^selenium/ { print \$1 }'"''')
We still need to escape the $ because otherwise they would be escaped by the outer shell (the one that os.popen calls to run the command).
I'm a little supicious of the -F/ in your awk command, but I assume you've tested this out and confirmed it turns the output you want.
By using the subprocess module, which doesn't call /bin/sh by default, you can further reduce the escaping (at the expense of having to tokenize the command line yourself):
import subprocess
subprocess.check_output([
'nsenter', '-t', '1', '-m', '-u', '-n', '-i',
'sh', '-c', "docker ps | awk -F/ '$2 ~ /^selenium/ { print $1 }'"
])

Do you have trying: ... awk -F/ '\\$2 ~ /^s...
(double backslash before $2)

Convert a linux command to python command

Would you please say me how can I convert this code:
path to app/adb devices | awk 'NR>1{print $1}' | xargs -n1 -I% adb -s % install app.apk
to python command?

You can always utilize the os.system() method:
import os
command = "path to app/adb devices | awk 'NR>1{print $1}' | xargs -n1 -I% adb -s % install app.apk"
os.system(command)

KeyError: 'print $2' when I use subprocess.call("ps -ef | grep wget | grep {0} | awk '{print $2}'

In my macOS, I using a wget downloading the www.test.com index page.
then I have below python code:
#-*- coding:utf-8 -*-
import subprocess
url = "https://www.test.com"
subprocess.call("ps -ef | grep wget | grep {0} | awk '{print $2}'".format(url), shell=True)
when I run it, I get issue:
subprocess.call("ps -ef | grep wget | grep {0} | awk '{print $2}'".format(url), shell=True)
KeyError: 'print $2'
and I switch the subprocess.call() to os.system(), still get this issue.

Your error is not caused by subprocess, rather the string format.
"ps -ef | grep wget | grep {0} | awk '{print $2}'".format(url)
you can use %s to forming string.
"ps -ef | grep wget | grep %s | awk '{print $2}'"%(url)

You need to use subprocess.run()
From Docs
Code needing to capture stdout or stderr should use run() instead
Note : Do not use stdout=PIPE or stderr=PIPE with this function. The child process will block if it generates enough output to a pipe to fill up the OS pipe buffer as the pipes are not being read from.

broken pipe when executing bash in python

everyone, I encouter the error of Broken pipe when trying to executing bash in python.
Here is my bash file, run.sh
INPUT=`python -c "print 'uid='+'A'*0x4"`
TEST=$INPUT
LEN=$(echo -n "$INPUT" | wc -c)
cp $(which qemu-mipsel-static) ./qemu
echo "$INPUT" | chroot . ./qemu -E CONTENT_LENGTH=$LEN -E CONTENT_TYPE="application/x-www-form-urlencoded" -E REQUEST_METHOD="POST" -E HTTP_COOKIE=$TEST -E REQUEST_URI="/authentication.cgi" -E REMOTE_ADDR="192.168.1.1" htdocs/web/authentication.cgi 2>/dev/null
echo 'run ok'
rm -f ./qemu
Here is how I tried to call execute the bash in python:
bash_file_path =run.sh
op = commands.getstatusoutput("bash %s" % (bash_file_path) )
print op[1]
However, I encouter the error in the line 5 of the run.sh:
run.sh: line 5: echo: write error: Broken pipe
I also tried the subprocess, however, got the same errors:
p = subprocess.Popen(["bash", bash_file_path], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
print p.stdout.readlines()
print p.stderr.readlines()
the results:
[]
[run.sh: line 5: echo: write error: Broken pipe]

python subprocess.Popen vs os.popen

I'm trying to get the output of the following shell command in my python script,
hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}
I can successfully get the output through os.popen as follows:
import os
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = os.popen(cmd,"r")
while 1:
line = p.readline()
if not line: break
print line
But os.popen() is deprecated since python 2.6 so I wanted to replace the above snippet with the subprocess.Popen() function.
But the code snippet for subprocess.Popen() below gives a different result than the code snippet above.
import subprocess as sub
import shlex
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
args = shlex.split(cmd)
p = sub.Popen(args,stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output
The above command just gives output for 'hadoop fs -ls /projectpath/' part of the command.
I have tried consulting several references (http://docs.python.org/2/library/subprocess.html#popen-objects, Python, os.system for command-line call (linux) not returning what it should?) for subpocess.Popen() but cannot get it to execute the command in the string cmd. Can anyone point out what I'm doing wrong?

try this:
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = sub.Popen(cmd,stdout=sub.PIPE,stderr=sub.PIPE, shell=True)