Develop Reference

awk command doesn't work under /bin/sh -c

I'm using this command:
docker ps | awk '$2 ~ /^selenium/ { print $1 }'
which works fine in the shell, but when running it with sh -c it doesn't work and I'm getting this error:
awk: cmd. line:1: ~ /^selenium/ { print }
awk: cmd. line:1: ^ syntax error
The full command that I want to is part from a Python script:
os.popen("nsenter -t 1 -m -u -n -i sh -c \"docker ps | awk -F/ '\''$2 ~ /^selenium/ { print $1 }'\''\"")
It's probably some escaping problem but I couldn't solve any.

You've got several levels of quoting there that are causing problems. If you start by using Python triple-quotes on the outside (''' or """), you can reduce the amount of escaping you need to perform.
That gets us:
os.popen('''nsenter -t 1 -m -u -n -i sh -c "docker ps | awk -F/ '\$2 ~ /^selenium/ { print \$1 }'"''')
We still need to escape the $ because otherwise they would be escaped by the outer shell (the one that os.popen calls to run the command).
I'm a little supicious of the -F/ in your awk command, but I assume you've tested this out and confirmed it turns the output you want.
By using the subprocess module, which doesn't call /bin/sh by default, you can further reduce the escaping (at the expense of having to tokenize the command line yourself):
import subprocess
subprocess.check_output([
'nsenter', '-t', '1', '-m', '-u', '-n', '-i',
'sh', '-c', "docker ps | awk -F/ '$2 ~ /^selenium/ { print $1 }'"
])

Do you have trying: ... awk -F/ '\\$2 ~ /^s...
(double backslash before $2)

Why does 'python --version' not print string [duplicate]

This question already has answers here:
Detect python version in shell script
(18 answers)
Closed 4 years ago.
I was trying to write a bash script to test the python version. However, I found python --version behave weirdly for python 2, as I can't process its output using any tool. I have tested the same script on Mac OS (10.13.5) and AWS Linux (GUN/Linux #1 SMP Fri Feb 16 00:18:48 UTC 2018). So I think the problem is related to python 2.
The script and corresponding output are:
$ echo $(python --version) | awk '{print $2}'
> Python 2.7.10
But the output should be 2.7.10.
$ echo $(python --version) > weird.txt
> Python 2.7.10
$ cat weird.txt
>
So the output cannot be written into a file as well.
The same script to test for python3 has a totally different result
$ echo $(python3 --version) | awk '{print $2}'
> 3.6.5
But python3's output can be written into a file.
$ echo $(python3 --version) > weird.txt
$ cat weird.txt
> Python 3.6.5
I have found the reason for this difference is that python --version does not output a normal string or whatsoever. Maybe it calls another command to output the result for it??? Thus the result cannot be caught by current process?? (just pure guess here)
Can anyone help me to figure out why there is the difference here? There are probably of million ways to test for python version. But I'm just super curious about what is happening here.
Thanks for all the replies. Just found a useful answer explaining that why python -V outputs to stderr:
Why does python print version info to stderr?

Python outputs the version to standard error (stderr) up to version 3.3 according to issue 18338 and as noted here, so redirect accordingly:
$ echo $(python --version 2>&1) | awk '{print $2}'
2.7.14
The command substitution is unnecessary and this could be written as:
$ python --version 2>&1 | awk '{print $2}'

How about using Python command itself using platform library of it(which is a very common in use).
python -c 'import platform; print(platform.python_version())'
When I run it I get Python's exact version.

Why they are different if I run the command in linux directly and run it by importing os module in python?

I am doing two "similar" things:
(1) in python
import os
os.system('cat ...input | awk -f ...awk' -v seed=$RANDOM)
(2) in linux terminal
cat ...input | awk -f ...awk' -v seed=$RANDOM
Actually, my awk file will return a randomized input file, but if I run way(1) many times, the result always be same(only one result). But If I run way(2), then every time I can get a randomized file. What's wrong with it?
If I want to run this command in python, how should I do then?
Thank you so much for you answer.
EDIT:
Adding the actual code:
(1) in python
import os
os.system("cat data/MD-00001-00000100.input | awk -f utils/add_random_real_weights.awk -v seed=$RANDOM")
(2) in linux:
cat data/MD-00001-00000100.input | awk -f utils/add_random_real_weights.awk -v seed=$RANDOM

python -V >> logfile.txt What is needed to make this work?

This command does not work as expected:
$ python -V >> logfile.txt
The python version is displayed on screen, but does not end up in the logfile, what is needed to get it into the logfile?

$ python -V 2>> logfile.txt
python -V writes to stderr instead of stdout that's why you must append your append operator with 2.

the -V output is going to stderr, so you need to redirect it like this:
python -V 2>> logfile.txt

Python one-liner to print every file in the current directory

How can I make the following one liner print every file through Python?
python -c "import sys;print '>>',sys.argv[1:]" | dir *.*
Specifically would like to know how to pipe into a python -c.
DOS or Cygwin responses accepted.

python -c "import os; print os.listdir('.')"
If you want to apply some formatting like you have in your question,
python -c "import os; print '\n'.join(['>>%s' % x for x in os.listdir('.')])"
If you want to use a pipe, use xargs:
ls | xargs python -c "import sys; print '>>', sys.argv[1:]"
or backticks:
python -c "import sys; print '>>', sys.argv[1:]" `ls`

You can read data piped into a Python script by reading sys.stdin. For example:
ls -al | python -c "import sys; print sys.stdin.readlines()"
It is not entirely clear what you want to do (maybe I am stupid). The confusion comes from your example which is piping data out of a python script.

If you want to print all files:
find . -type f
If you want to print only the current directory's files
find . -type f -maxdepth 1
If you want to include the ">>" before each line
find . -type f -maxdepth 1 | xargs -L 1 echo ">>"
If you don't want the space between ">>" and $path from echo
find . -type f -maxdepth 1 | xargs -L 1 printf ">>%s\n"
This is all using cygwin, of course.

ls | python -c "import sys; print sys.stdin.read()"
just read stdin as normal for pipes

would like to know how to pipe though
You had the pipe the wrong way round, if you wanted to feed the output of ‘dir’ into Python, ‘dir’ would have to be on the left. eg.:
dir "*.*" | python -c "import sys;[sys.stdout.write('>>%s\n' % line) for line in sys.stdin]"
(The hack with the list comprehension is because you aren't allowed a block-introducing ‘for’ statement on one line after a semicolon.)
Clearly the Python-native solution (‘os.listdir’) is much better in practice.

Specifically would like to know how to pipe into a python -c
see cobbal's answer
piping through a program is transparent from the program's point of view, all the program knows is that it's getting input from the standard input stream
Generally speaking, a shell command of the form
A | B
redirects the output of A to be the input of B
so if A spits "asdf" to standard output, then B gets "asdf" into its standard input
the standard input stream in python is sys.stdin