Develop Reference

awk command doesn't work under /bin/sh -c

I'm using this command:
docker ps | awk '$2 ~ /^selenium/ { print $1 }'
which works fine in the shell, but when running it with sh -c it doesn't work and I'm getting this error:
awk: cmd. line:1: ~ /^selenium/ { print }
awk: cmd. line:1: ^ syntax error
The full command that I want to is part from a Python script:
os.popen("nsenter -t 1 -m -u -n -i sh -c \"docker ps | awk -F/ '\''$2 ~ /^selenium/ { print $1 }'\''\"")
It's probably some escaping problem but I couldn't solve any.

You've got several levels of quoting there that are causing problems. If you start by using Python triple-quotes on the outside (''' or """), you can reduce the amount of escaping you need to perform.
That gets us:
os.popen('''nsenter -t 1 -m -u -n -i sh -c "docker ps | awk -F/ '\$2 ~ /^selenium/ { print \$1 }'"''')
We still need to escape the $ because otherwise they would be escaped by the outer shell (the one that os.popen calls to run the command).
I'm a little supicious of the -F/ in your awk command, but I assume you've tested this out and confirmed it turns the output you want.
By using the subprocess module, which doesn't call /bin/sh by default, you can further reduce the escaping (at the expense of having to tokenize the command line yourself):
import subprocess
subprocess.check_output([
'nsenter', '-t', '1', '-m', '-u', '-n', '-i',
'sh', '-c', "docker ps | awk -F/ '$2 ~ /^selenium/ { print $1 }'"
])

Do you have trying: ... awk -F/ '\\$2 ~ /^s...
(double backslash before $2)

subprocess command execution

What is the best way to execute the below command in Python in a single line?
echo $(readlink /sys/dev/block/$(mountpoint -d /))
Tried using individual os.system(cmd) by separating - "mountpoint -d /" first and taking the output and appending to "readlink /sys/dev/block/${0}".format(out.strip()) and doing an echo works. Tried using subprocess and subprocess.Popen and subprocess.check_output but it raises raise CalledProcessError
cmd = "echo $(readlink /sys/dev/block/$(mountpoint -d /))"

You have to call the subcommand separately. And you can use python methods to read the link:
import subprocess
import os
path = "/"
device = subprocess.run(["mountpoint", "-d", path], stdout=subprocess.PIPE, encoding="utf8").stdout.strip()
link = os.readlink("/sys/dev/block/" + device)
print(link)

You probably want to use something like the following:
cmd = "bash -c 'echo $(readlink /sys/dev/block/$(mountpoint -d /))'"
echo doesn't substitute $() blocks, that's what your shell does, so you have to call the shell. os.system(cmd) should work then.

python bash command: how to escape single quote?

I'm trying to execute a shell command through python. The command is like the following one:
su -c "lftp -c 'open -u user,password ftp://127.0.0.1; get ivan\'s\ filename.pdf' " someuser
So, when I try to do it in python:
command = "su -c \"lftp -c 'open -u user,password ftp://127.0.0.1; get ivan\'s\ filename.pdf' \" someuser"
os.system(command)
Or:
command = subprocess.Popen(["su", "-c", "lftp -c 'open -u user,password ftp://127.0.0.1; get ivan\'s\ filename.pdf'", "someuser"])
I get the following error:
bash: -c: line 0: unexpected EOF while looking for matching `''
bash: -c: line 1: syntax error: unexpected end of file
Referred to: ivan\'s single quote.
I know there are a lot of single/double quotes in that but how can I escape this?
Thanks in advance!
EDIT: THIS WORKED FOR ME:
subprocess.call(["su","-c",r"""lftp -c "open -u user,password ftp://127.0.0.1; get ivan\'s\ filename.pdf" """, "someuser"])
Thank you all very much!

If you printed your test string you would notice that it results in the following:
su -c "lftp -c 'open -u user,password ftp://127.0.0.1; get ivan's\ filename.pdf' " someuser
The problem is that you need to escape the slash that you use to escape the single quote in order to keep Python from eating it.
command = "su -c \"lftp -c 'open -u user,password ftp://127.0.0.1; get ivan\\'s\\ filename.pdf' \" someuser"
will get the backslash across, you will then get an error from lftp instead...
This works:
command = "su -c \"lftp -c \\\"open -u user,password ftp://127.0.0.1; get ivan\\'s\\ filename.pdf\\\" \" someuser"
(It uses (escaped) double quotes instead, to ensure that the shell started by su still interprets the escape sequences)
(os.system(a) effectively does subprocess.call(["sh","-c",a]), which means that sh sees su -c "lftp -c 'open -u user,password ftp://127.0.0.1; get ivan's\ filename.pdf' " someuser (for the original one). It does escape sequence processing on this and sees an unclosed single quote (it is initially closed by ivan'), resulting in your error). Once that is fixed, sh calls su, which in turn starts up another instance of sh doing more escape processing, resulting in the error from lftp (since sh doesn't handle escape sequences in the single quotes)
subprocess.call() or curl are better ways to implement this - curl will need much less escaping, you can use curl "ftp://user:password#127.0.0.1/ivan's filename.pdf" on the command line, some more escaping is needed for going viasu -cand for python.sudoinstead ofsu` also results in less escaping being needed....
If you want to use subprocess.call() (which removes one layer of shell), you can use
subprocess.call(["su","-c","lftp -c \\\"open -u user,password ftp://127.0.0.1; get ivan\\'s\\ filename.pdf\\\"", "someuser"])
(The problem is that python deals with one level of escaping, and the sh -c invoked from su with the next layer... This results in quite an ugly command...) (different quotes might slightly reduce that...)
Using r"" can get rid of the python level escape processing: (needing only the shell level escapes) (Using triple quotes to allow quotes in the string)
subprocess.call(["su","-c",r"""lftp -c \"open -u user,password ftp://127.0.0.1; get ivan\'s\ filename.pdf\"""", "someuser"])
Adding a space allows for stripping the shell escapes, since lftp doesn't seem to need the filename escaped for the spaces and single quote.
subprocess.call(["su","-c",r"""lftp -c "open -u user,password ftp://127.0.0.1; get ivan's filename.pdf" """, "someuser"])
This results in the eventual lftp ARGV being
["lftp","-c","open -u user,password ftp://127.0.0.1; get ivan's filename.pdf"]
For curl instead (it still ends up bad due to the su being involved):
subprocess.call(["su","-c",r"""curl "ftp://user:password#127.0.0.1/ivan's filename.pdf" """, "someuser"])

Using subprocess.call() is the best and more secure way to perform this task.
Here's an example from the documentation page:
subprocess.call(["ls", "-l"]) # As you can see we have here the command and a parameter
About the error I think it is something related to the spaces and the ' charachter.
Try using string literals (pay attention to the r before the string, also be sure that the command is 100% matching the one you use in BASH):
r"My ' complex & string"
So, in your case:
command = subprocess.Popen(["su", "-c", r"lftp -c 'open -u user,password ftp://127.0.0.1; get ivan's filename.pdf'", "someuser"])

Execute a shell command with python variables

I had script on bash where I generated username, password, ssh-key for user.
Part for creating of ssh-key:
su $user -c "ssh-keygen -f /home/$user/.ssh/id_rsa -t rsa -b 4096 -N ''"
How can I do the same in Python with os.system? I tried this:
os.system('su %s -c "ssh-keygen -f /home/%s/.ssh/id_rsa -t rsa -b 4096 -N ''"', user)
TypeError: system() takes at most 1 argument (2 given)
Also I tried:
os.system('su user -c "ssh-keygen -f /home/user/.ssh/id_rsa -t rsa -b 4096 -N ''"')
Of course, it doesn't work either.

Format your instructions with the os package; for instance:
import os
user = 'joe'
ssh_dir = "/home/{}/.ssh/id_rsa".format(user)
os.system("ssh-keygen -f {} -t rsa -b 4096 -N ''".format(ssh_dir))

os.system is very close to a bash command line because it uses an underlying shell (like its cousins subprocess.call... using shell=True)
In your case, there's little interest using subprocess since your command runs a command, so you cannot really use argument protection by subprocess fully.
Pass the exact command, but the only change would be to protect the simple quotes, else python sees that as string end+string start (your string is protected by simple quotes already) and they're eliminated.
Check this simpler example:
>>> 'hello '' world'
'hello world'
>>> 'hello \'\' world'
"hello '' world"
that's a kind of worst-case when you cannot use either double or simple quotes to protect the string because you're using the other flavour within. In that case, escape the quotes using \:
os.system('su $user -c "ssh-keygen -f /home/$user/.ssh/id_rsa -t rsa -b 4096 -N \'\'"')

Use the subprocess module:
import subprocess
username = 'user'
result, err = subprocess.Popen(
'su %s -c "ssh-keygen -f /home/%s/.ssh/id_rsa -t rsa -b 4096 -N ''"' % (username, username),
stdout=subprocess.PIPE,
shell=True
).communicate()
if err:
print('Something went wrong')
else:
print(result)
Edit: this is the 'fast' way to do that, you should't use shell=True if you can't control the input since it allows code execution as said here

form a subprocess.call() statement from system(cmd) command in python

I have a ssh command which I was using in a system() statement but I want to replace it with a subprocess.call() statement. My ssh command is:
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
system(cmd)
I want to replace above with a subprocess.call() statement as it is giving me some performance issues and I read that subprocess.call() is a much better option to use. I formulated this query but it is not executing:
result = subprocess.call(["ssh","-i", "pem-file.pem","user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"])
What is the mistake I am doing and what is the correct syntax?

The function shlex.split() is useful for parsing command line arguments into the proper format. This should resolve your syntax error:
import shlex
import subprocess
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
result = subprocess.call(shlex.split(cmd))
If that doesn't fix your error, then you can pass subprocess.call the shell=True argument:
import subprocess
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
result = subprocess.call(cmd, shell=True)
Using the shell argument will cause your command to be executed through a shell, rather than having the interpreter parse it. However, don't use the shell option if cmd can ever come from an untrusted source. Take at look at the warning in the Python docs.
One more note:
subprocess.system() is newer and more flexible than os.system(), but don't worry too much about "upgrading" to the new function. The advantages of subprocess.call() are in the more flexible options for communicating with your subprocess. If all you're doing is executing a single command and getting the return code, os.system() is probably fine. If you're finding that your command is being flaky and unreliable, switching to subprocess.call() probably isn't going to help much.

Assuming there are no shell meta-characters in hostname (likely), your command could look like this: each command-line argument is a separate list item:
#!/usr/bin/env python
import subprocess
cmd = ["ssh", "-i", "pem-file.pem", "user#" + hostname,
"cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 "
"-f text-file_" + ts + ".txt"]
subprocess.check_call(cmd)
Unlike os.system(); it doesn't run the (local) shell.
You could get the argument list using shlex.split(your_original_system_command) (mentioned by #skrrgwasme) but shlex.split() can be fooled and therefore it is mostly useful as a hint on how the result should look like.