I'm creating a python program that calls a number of other programs and scripts (on Unix(SUNos) + Linux). I'm using subprocess everywhere except for 1 script.
The script for which I don't use subprocess is a perl-script which has been made into an executable. Somehow it does not let me use subprocess on, but it works with the (deprecated) commands package.
I would like to understand why it does not work with subprocess (in other words: what am I doing wrong ;-) )
(What the actual perl command is not important, but it returns the full name and email of a user as result)
What I tried:
PERL_CMD = [ '<executable perl-script>', '-rt', '"users"', '-eq', '"name"' '"<user_name>", '-fs', '":"', '-fld', '"fullname"', '"email"' ]
full_name, email = subprocess.check_output( PERL_CMD ).split(':')
But this does not work.
Where the commands variant does work:
PERL_CMD = '<executable perl-script> -rt "users" -eq "name" "<user_name>" -fs ":" -fld "full_name" "email"'
full_name, email = commands.getoutput( PERL_CMD ).split(':')
Has anybody an idea why I can't get subprocess to work?
It is annoying me that I can get it to work for everything except this (eventhough I have an acceptable (but deprecated) workaround).
You're using syntactic quotes in the commands.getoutput() case, and literal quotes in the subprocess.check_output() case. Without shell=True (which you shouldn't use), there's no shell to parse quotes as syntax, so there's no such thing as a syntactic quote, other than the quotes that are syntax to Python itself.
So, just take out the "s that you injected into your arguments:
# this contains quotes that are syntactic to Python only, and no literal quotes
perl_cmd = [
'<executable perl-script>',
'-rt', 'users',
'-eq', 'name', '<user_name>',
'-fs', ':',
'-fld', 'fullname', 'email' ]
To explain a bit more detail --
When you pass "name" to a shell as part of a command, the quotes are consumed by the shell itself during its parsing process, not passed to the command as an argument. Thus, when you run sh -c 'echo "hello"', this passes the exact same argument to echo as sh -c 'echo hello'; the echo command can't even tell the difference between the two invocations!
When you pass '"hello"' as an argument to subprocess.Popen(), by contrast, the outer quotes are consumed by Python, and the inner quotes are passed as literal to the inner command. That makes it equivalent to sh -c 'echo "\"hello\""' (which likewise passes literal quotes through to echo), not sh -c 'echo "hello"' (which does not).
Related
I know similar questions have been asked before, but they all seem to have been resolved by reworking how arguments are passed (i.e. using a list, etc).
However, I have a problem here in that I don't have that option. There is a particular command line program (I am using a Bash shell) to which I must pass a quoted string. It cannot be unquoted, it cannot have a replicated argument, it just has to be either single or double quoted.
command -flag 'foo foo1'
I cannot use command -flag foo foo1, nor can I use command -flag foo -flag foo1. I believe this is an oversight in how the command was programmed to receive input, but I have no control over it.
I am passing arguments as follows:
self.commands = [
self.path,
'-flag1', quoted_argument,
'-flag2', 'test',
...etc...
]
process = subprocess.Popen(self.commands, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
results = process.communicate(input)
Where quoted_argument is something like 'foo foo1 foo2'.
I have tried escaping the single quote ("\'foo foo1 foo2\'"), but I get no output.
I know this is considered bad practice because it is ambiguous to interpret, but I don't have another option. Any ideas?
The shell breaks command strings into lists. The quotes tell the shell to put multiple words into a single list item. Since you are building the list yourself, you add the words as a single item without the quotes.
These two Popen commands are equivalent
Popen("command -flag 'foo foo1'", shell=True)
Popen(["command", "-flag", "foo foo1"])
EDIT
This answer deals with escaping characters in the shell. If you don't use the shell, you don't add any quotes or escapes, just put in the string itself. There are other issues with skipping the shell, like piping commands, running background jobs, using shell variables and etc. These all can be done in python instead of the shell.
A mental model of process and shells that I found very helpful:
This mental model has helped me a lot through the years.
Processes in your operating system receive an array of strings representing the arguments. In Python, this array can be accessed from sys.argv. In C, this is the argv array passed to the main function. And so on.
When you open a terminal, you are running a shell inside that terminal, for example bash or zsh. What happens if you run a command like this one?
$ /usr/bin/touch one two
What happens is that the shell interprets the command that you wrote and splits it by whitespace to create the array ["/usr/bin/touch", "one", "two"]. It then launches a new process using that list of arguments, in this case creating two files named one and two.
What if you wanted one file named one two with a space? You can't pass the shell a list of arguments as you might want to do, you can only pass it a string. Shells like Bash and Zsh use single quotes to workaround this:
$ /usr/bin/touch 'one two'
The shell will create a new process with the arguments ["/usr/bin/touch", "one two"], which in this case create a file named one two.
Shells have special features like piping. With a shell, you can do something like this:
$ /usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z
THIS IS AN EXAMPLE
In this case, the shell interprets the | character differently. In creates a process with the arguments ["/usr/bin/echo", "This is an example"] and another process with the arguments ["/usr/bin/tr", "a-z", "A-Z"], and will pipe the output of the former to the input of the latter.
How this applies to subprocess in Python
Now, in Python, you can use subprocess with shell=False (which is the default, or with shell=True. If you use the default behaviour shell=False, then subprocess expects you to pass it a list of arguments. You cannot use special shell features like shell piping. On the plus side, you don't have to worry about escaping special characters for the shell:
import subprocess
# create a file named "one two"
subprocess.call(["/usr/bin/touch", "one two"])
If you do want to use shell features, you can do something like:
subprocess.call(
"/usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z",
shell=True,
)
If you are using variables with no particular guarantees, remember to escape the command:
import shlex
import subprocess
subprocess.call(
"/usr/bin/echo " + shlex.quote(variable) + " | /usr/bin/tr a-z A-Z",
shell=True,
)
(Note that shlex.quote is only designed for UNIX shells, and not for DOS on Windows.)
I have a bash script which is running a bunch of python script all with arguments. In order to have a clean code, I wanted to use variables along the scripts
#!/bin/bash
START=0
SCRIPT_PATH="/opt/scripts/"
IP="192.168.1.111"
if [ "$START" = "0" ]; then
printf "%s: Starting\n" "$DATE_TIME"
PORT=1234
TEST_FILE="$SCRIPT_PATH/Test Scripts/test.TXT"
SCRIPT="$SCRIPT_PATH/script1.py"
ARGS="-P $SCRIPT_PATH/script2.py -n 15 -p $PORT -i $IP"
python "$SCRIPT" ${ARGS} -f "${TEST_FILE}" > ./out.log 2>&1 &
fi
This code is actually working but few things I don't understand :
Why, if I add quotes around ${ARGS}, the arguments are not parsed correctly by python ? What would be the best way to write this ?
What is the best method to add -f "${TEST_FILE}" to the ARGS variable without python blocking on the whitespace and throwing the error: "$SCRIPT_PATH/Test " not found
When you wrap quotes around an argument list, the argument vector receives a single argument with everything that is wrapped in quotes and so, the argument parser fails to do its job properly and you have your issue.
Regarding your second question, it is not easy to embed the quotes into the array, because the quotes will be parsed before being stored in the array, and then when you perform the array expansion to run the command, they will be missing and fail. I have tried this several times with no success.
An alternative approach would mean that you modify a little your script to use a custom internal field separator (IFS) to manually tell what should be considered an argument and what not:
#!/bin/bash
START=0
SCRIPT_PATH="/opt/scripts/"
IP="192.168.1.111"
if [ "$START" = "0" ]; then
printf "%s: Starting\n" "$DATE_TIME"
PORT=1234
TEST_FILE="$SCRIPT_PATH/Test Scripts/test.TXT"
SCRIPT="$SCRIPT_PATH/script1.py"
OLD_IFS=$IFS
IFS=';'
ARGS="$SCRIPT;-P;$SCRIPT_PATH/script2.py;-n;15;-p;$PORT;-i;$IP;-f;$TEST_FILE"
python ${ARGS} > ./out.log 2>&1 &
IFS=$OLD_IFS
fi
As you can see, I replace the spaces in ARGS with semicolons. This way, TEST_FILE variable contents will be considered as a single argument for bash and will be properly populated in argument vector. I'm also moving the script to the argument vector for simplicity, otherwise, Python will not get the proper script path and fail, due to this modification we did to IFS.
I was thinking something like this (with some cruft edited out to make it a standalone example):
#!/bin/bash
SCRIPT_PATH="/opt/scripts/"
IP="192.168.1.111"
PORT=1234
TEST_FILE="$SCRIPT_PATH/Test Scripts/test.TXT"
SCRIPT="$SCRIPT_PATH/script1.py"
set -a ARGS
ARGS=(-P "$SCRIPT_PATH/script2.py" -n 15 -p "$PORT" -i "$IP")
ARGS+=(-f "${TEST_FILE}")
python3 -c "import sys; print(*enumerate(sys.argv), sep='\n')" "${ARGS[#]}"
I am trying to run an ssh command within a python script using os.system to add a 0 at the end of a fully matched string in a remote server using ssh and sed.
I have a file called nodelist in a remote server that's a list that looks like this.
test-node-1
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21
I want to use sed to make the following modification, I want to search test-node-1, and when a full match is found I want to add a 0 at the end, the file must end up looking like this.
test-node-1 0
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21
However, when I run the first command,
hostname = 'test-node-1'
function = 'nodelist'
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/{hostname}/s/$/ 0/' ~/{function}.txt\"")
The result becomes like this,
test-node-1 0
test-node-2
...
test-node-11 0
test-node-12 0
test-node-13 0
...
test-node-21
I tried adding a \b to the command like this,
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/\b{hostname}\b/s/$/ 0/' ~/{function}.txt\"")
The command doesn't work at all.
I have to manually type in the node name instead of using a variable like so,
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/\btest-node-1\b/s/$/ 0/' ~/{function}.txt\"")
to make my command work.
What's wrong with my command, why can't I do what I want it to do?
This code has serious security problems; fixing them requires reengineering it from scratch. Let's do that here:
#!/usr/bin/env python3
import os.path
import shlex # note, quote is only here in Python 3.x; in 2.x it was in the pipes module
import subprocess
import sys
# can set these from a loop if you choose, of course
username = "whoever"
serverlocation = "whereever"
hostname = 'test-node-1'
function = 'somename'
desired_cmd = ['sed', '-i',
f'/\\b{hostname}\\b/s/$/ 0/',
f'{function}.txt']
desired_cmd_str = ' '.join(shlex.quote(word) for word in desired_cmd)
print(f"Remote command: {desired_cmd_str}", file=sys.stderr)
# could just pass the below direct to subprocess.run, but let's log what we're doing:
ssh_cmd = ['ssh', '-i', os.path.expanduser('~/.ssh/my-ssh-key'),
f"{username}#{serverlocation}", desired_cmd_str]
ssh_cmd_str = ' '.join(shlex.quote(word) for word in ssh_cmd)
print(f"Local command: {ssh_cmd_str}", file=sys.stderr) # log equivalent shell command
subprocess.run(ssh_cmd) # but locally, run without a shell
If you run this (except for the subprocess.run at the end, which would require a real SSH key, hostname, etc), output looks like:
Remote command: sed -i '/\btest-node-1\b/s/$/ 0/' somename.txt
Local command: ssh -i /home/yourname/.ssh/my-ssh-key whoever#whereever 'sed -i '"'"'/\btest-node-1\b/s/$/ 0/'"'"' somename.txt'
That's correct/desired output; the funny '"'"' idiom is how one safely injects a literal single quote inside a single-quoted string in a POSIX-compliant shell.
What's different? Lots:
We're generating the commands we want to run as arrays, and letting Python do the work of converting those arrays to strings where necessary. This avoids shell injection attacks, a very common class of security vulnerability.
Because we're generating lists ourselves, we can change how we quote each one: We can use f-strings when it's appropriate to do so, raw strings when it's appropriate, etc.
We aren't passing ~ to the remote server: It's redundant and unnecessary because ~ is the default place for a SSH session to start; and the security precautions we're using (to prevent values from being parsed as code by a shell) prevent it from having any effect (as the replacement of ~ with the active value of HOME is not done by sed itself, but by the shell that invokes it; because we aren't invoking any local shell at all, we also needed to use os.path.expanduser to cause the ~ in ~/.ssh/my-ssh-key to be honored).
Because we aren't using a raw string, we need to double the backslashes in \b to ensure that they're treated as literal rather than syntactic by Python.
Critically, we're never passing data in a context where it could be parsed as code by any shell, either local or remote.
I know similar questions have been asked before, but they all seem to have been resolved by reworking how arguments are passed (i.e. using a list, etc).
However, I have a problem here in that I don't have that option. There is a particular command line program (I am using a Bash shell) to which I must pass a quoted string. It cannot be unquoted, it cannot have a replicated argument, it just has to be either single or double quoted.
command -flag 'foo foo1'
I cannot use command -flag foo foo1, nor can I use command -flag foo -flag foo1. I believe this is an oversight in how the command was programmed to receive input, but I have no control over it.
I am passing arguments as follows:
self.commands = [
self.path,
'-flag1', quoted_argument,
'-flag2', 'test',
...etc...
]
process = subprocess.Popen(self.commands, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
results = process.communicate(input)
Where quoted_argument is something like 'foo foo1 foo2'.
I have tried escaping the single quote ("\'foo foo1 foo2\'"), but I get no output.
I know this is considered bad practice because it is ambiguous to interpret, but I don't have another option. Any ideas?
The shell breaks command strings into lists. The quotes tell the shell to put multiple words into a single list item. Since you are building the list yourself, you add the words as a single item without the quotes.
These two Popen commands are equivalent
Popen("command -flag 'foo foo1'", shell=True)
Popen(["command", "-flag", "foo foo1"])
EDIT
This answer deals with escaping characters in the shell. If you don't use the shell, you don't add any quotes or escapes, just put in the string itself. There are other issues with skipping the shell, like piping commands, running background jobs, using shell variables and etc. These all can be done in python instead of the shell.
A mental model of process and shells that I found very helpful:
This mental model has helped me a lot through the years.
Processes in your operating system receive an array of strings representing the arguments. In Python, this array can be accessed from sys.argv. In C, this is the argv array passed to the main function. And so on.
When you open a terminal, you are running a shell inside that terminal, for example bash or zsh. What happens if you run a command like this one?
$ /usr/bin/touch one two
What happens is that the shell interprets the command that you wrote and splits it by whitespace to create the array ["/usr/bin/touch", "one", "two"]. It then launches a new process using that list of arguments, in this case creating two files named one and two.
What if you wanted one file named one two with a space? You can't pass the shell a list of arguments as you might want to do, you can only pass it a string. Shells like Bash and Zsh use single quotes to workaround this:
$ /usr/bin/touch 'one two'
The shell will create a new process with the arguments ["/usr/bin/touch", "one two"], which in this case create a file named one two.
Shells have special features like piping. With a shell, you can do something like this:
$ /usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z
THIS IS AN EXAMPLE
In this case, the shell interprets the | character differently. In creates a process with the arguments ["/usr/bin/echo", "This is an example"] and another process with the arguments ["/usr/bin/tr", "a-z", "A-Z"], and will pipe the output of the former to the input of the latter.
How this applies to subprocess in Python
Now, in Python, you can use subprocess with shell=False (which is the default, or with shell=True. If you use the default behaviour shell=False, then subprocess expects you to pass it a list of arguments. You cannot use special shell features like shell piping. On the plus side, you don't have to worry about escaping special characters for the shell:
import subprocess
# create a file named "one two"
subprocess.call(["/usr/bin/touch", "one two"])
If you do want to use shell features, you can do something like:
subprocess.call(
"/usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z",
shell=True,
)
If you are using variables with no particular guarantees, remember to escape the command:
import shlex
import subprocess
subprocess.call(
"/usr/bin/echo " + shlex.quote(variable) + " | /usr/bin/tr a-z A-Z",
shell=True,
)
(Note that shlex.quote is only designed for UNIX shells, and not for DOS on Windows.)
I'm trying to write a function that will issue commands via ssh with Popen and return the output.
def remote(cmd):
escaped = escape(cmd)
return subprocess.Popen(escaped, ...).communicate()[0]
My trouble is how to implement the escape function. Is there a Python module (2.6) that has helpers for this? Google shows there's pipes.quote and re.escape but they seem like they only work for locally run commands. With commands passed to ssh, it seems the escaping needs to be more stringent:
For example on a local machine, this is ok:
echo $(hostname)
When passing to ssh it has to be:
ssh server "echo \$(hostname)"
Also, double quotes can be interpreted in different ways depending on the context. For literal quotes, you need the following:
ssh a4ipe511 "echo \\\"hello\\\""
To do variable exansion, double quotes are also used:
ssh a4ipe511 "echo \"\$(hostname)\""
As you can see, the rules for escaping a command that goes into SSH can get pretty complicated. Since this function will be called by anyone, I'm afraid some complex commands will cause the function to return incorrect output.
Is this something that can be solved with a built-in Python module or do I need to implement the escaping myself?
First:
pipes.quote and re.escape have nothing to do with locally run commands. They simply transform a string; what you do with it after that is your business. So either -- in particular pipes.quote -- is suitable for what you want.
Second:
If you want to run the command echo $(hostname) on a remote host using ssh, you don't need to worry about shell escaping, because subprocess.Popen does not pass your commands into a shell by default. So, for example, this works just fine:
>>> import subprocess
>>> subprocess.call([ 'ssh', 'localhost', 'echo $(hostname)'])
myhost.example.com
0
Double quotes also work as you would expect:
>>> subprocess.call([ 'ssh', 'localhost', 'echo "one two"; echo three'])
one two
three
0
It's not clear to me that you actually have a problem.