I'm trying to write a function that will issue commands via ssh with Popen and return the output.
def remote(cmd):
escaped = escape(cmd)
return subprocess.Popen(escaped, ...).communicate()[0]
My trouble is how to implement the escape function. Is there a Python module (2.6) that has helpers for this? Google shows there's pipes.quote and re.escape but they seem like they only work for locally run commands. With commands passed to ssh, it seems the escaping needs to be more stringent:
For example on a local machine, this is ok:
echo $(hostname)
When passing to ssh it has to be:
ssh server "echo \$(hostname)"
Also, double quotes can be interpreted in different ways depending on the context. For literal quotes, you need the following:
ssh a4ipe511 "echo \\\"hello\\\""
To do variable exansion, double quotes are also used:
ssh a4ipe511 "echo \"\$(hostname)\""
As you can see, the rules for escaping a command that goes into SSH can get pretty complicated. Since this function will be called by anyone, I'm afraid some complex commands will cause the function to return incorrect output.
Is this something that can be solved with a built-in Python module or do I need to implement the escaping myself?
First:
pipes.quote and re.escape have nothing to do with locally run commands. They simply transform a string; what you do with it after that is your business. So either -- in particular pipes.quote -- is suitable for what you want.
Second:
If you want to run the command echo $(hostname) on a remote host using ssh, you don't need to worry about shell escaping, because subprocess.Popen does not pass your commands into a shell by default. So, for example, this works just fine:
>>> import subprocess
>>> subprocess.call([ 'ssh', 'localhost', 'echo $(hostname)'])
myhost.example.com
0
Double quotes also work as you would expect:
>>> subprocess.call([ 'ssh', 'localhost', 'echo "one two"; echo three'])
one two
three
0
It's not clear to me that you actually have a problem.
Related
I have written two python scripts A.py and B.py So B.py gets called in A.py like this:
config_object = {}
with open(config_filename) as data:
config_object = json.load(data, object_pairs_hook=OrderedDict)
command = './scripts/B.py --config-file={} --token-a={} --token-b={}'.format(promote_config_filename, config_object['username'], config_object['password'])
os.system(command)
In here config_object['password'] contains & in it. Say it is something like this S01S0lQb1T3&BRn2^Qt3
Now when this value get passed to B.py it gets password as S01S0lQb1T3 So after & whatever it is getting ignored.
How to solve this?
os.system runs a shell. You can escape arbitrary strings for the shell with shlex.quote() ... but a much superior solution is to use subprocess instead, like the os.system documentation also recommends.
subprocess.run(
['./scripts/B.py',
'--config-file={}'.format(promote_config_filename),
'--token-a={}'.format(config_object['username']),
'--token-b={}'.format(config_object['password'])])
Because there is no shell=True, the strings are now passed to the subprocess verbatim.
Perhaps see also Actual meaning of shell=True in subprocess
#tripleee has good suggestions. In terms of why this is happening, if you are running Linux/Unix at least, the & would start a background process. You can search "linux job control" for more info on that. The shortest (but not best) solution is to wrap your special characters in single or double quotes in the final command.
See this bash for a simple example:
$ echo foo&bar
[1] 20054
foo
Command 'bar' not found, but can be installed with:
sudo apt install bar
[1]+ Done echo foo
$ echo "foo&bar"
foo&bar
I need to execute the following in bash, but called from a Python script:
<command> --searchBody="{\"query":{\"range":{\"#timestamp\":{\"gte\": \"2020-10-16T00:00:00\",\"lte\": \"2020-10-17T00:00:00\"}}}}"
The double quotes must be escaped except for the ones that bound the --searchBody section
I have the following code in Python
execution = cmd+ ' --searchBody="{\\"query\":{\\"range\":{\\"#timestamp\\":{\\"gte\\": \\"'+startQuery+'\\",\\"lte\\": \\"'+endQuery+'\\"}}}}"'
print(execution)
os.system(execution)
cmd is the rest of the command already pre-populated, startQuery and endQuery are some date strings
The print statement prints the command exactly as it needs to, but when the os.system is run all the backslashes are removed from what is sent to the CLI.
I have tried all manor of escaping with multiple quotes but cannot get it to work. Any ideas?
Thanks
It's better to use subprocess module.
Try something like:
arg = '--searchBody="{\\"query\":{\\"range\":{\\"#timestamp\\":{\\"gte\\": \\"'+startQuery+'\\",\\"lte\\": \\"'+endQuery+'\\"}}}}"'
cmd = 'foobar'
subprocess.call([cmd, arg])
You should carefully read info about shell parameter and shlex module:
https://docs.python.org/3/library/subprocess.html#security-considerations
Using os.system() is like "I don't care about security, and all this crap, just run it"
I am trying to run an ssh command within a python script using os.system to add a 0 at the end of a fully matched string in a remote server using ssh and sed.
I have a file called nodelist in a remote server that's a list that looks like this.
test-node-1
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21
I want to use sed to make the following modification, I want to search test-node-1, and when a full match is found I want to add a 0 at the end, the file must end up looking like this.
test-node-1 0
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21
However, when I run the first command,
hostname = 'test-node-1'
function = 'nodelist'
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/{hostname}/s/$/ 0/' ~/{function}.txt\"")
The result becomes like this,
test-node-1 0
test-node-2
...
test-node-11 0
test-node-12 0
test-node-13 0
...
test-node-21
I tried adding a \b to the command like this,
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/\b{hostname}\b/s/$/ 0/' ~/{function}.txt\"")
The command doesn't work at all.
I have to manually type in the node name instead of using a variable like so,
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/\btest-node-1\b/s/$/ 0/' ~/{function}.txt\"")
to make my command work.
What's wrong with my command, why can't I do what I want it to do?
This code has serious security problems; fixing them requires reengineering it from scratch. Let's do that here:
#!/usr/bin/env python3
import os.path
import shlex # note, quote is only here in Python 3.x; in 2.x it was in the pipes module
import subprocess
import sys
# can set these from a loop if you choose, of course
username = "whoever"
serverlocation = "whereever"
hostname = 'test-node-1'
function = 'somename'
desired_cmd = ['sed', '-i',
f'/\\b{hostname}\\b/s/$/ 0/',
f'{function}.txt']
desired_cmd_str = ' '.join(shlex.quote(word) for word in desired_cmd)
print(f"Remote command: {desired_cmd_str}", file=sys.stderr)
# could just pass the below direct to subprocess.run, but let's log what we're doing:
ssh_cmd = ['ssh', '-i', os.path.expanduser('~/.ssh/my-ssh-key'),
f"{username}#{serverlocation}", desired_cmd_str]
ssh_cmd_str = ' '.join(shlex.quote(word) for word in ssh_cmd)
print(f"Local command: {ssh_cmd_str}", file=sys.stderr) # log equivalent shell command
subprocess.run(ssh_cmd) # but locally, run without a shell
If you run this (except for the subprocess.run at the end, which would require a real SSH key, hostname, etc), output looks like:
Remote command: sed -i '/\btest-node-1\b/s/$/ 0/' somename.txt
Local command: ssh -i /home/yourname/.ssh/my-ssh-key whoever#whereever 'sed -i '"'"'/\btest-node-1\b/s/$/ 0/'"'"' somename.txt'
That's correct/desired output; the funny '"'"' idiom is how one safely injects a literal single quote inside a single-quoted string in a POSIX-compliant shell.
What's different? Lots:
We're generating the commands we want to run as arrays, and letting Python do the work of converting those arrays to strings where necessary. This avoids shell injection attacks, a very common class of security vulnerability.
Because we're generating lists ourselves, we can change how we quote each one: We can use f-strings when it's appropriate to do so, raw strings when it's appropriate, etc.
We aren't passing ~ to the remote server: It's redundant and unnecessary because ~ is the default place for a SSH session to start; and the security precautions we're using (to prevent values from being parsed as code by a shell) prevent it from having any effect (as the replacement of ~ with the active value of HOME is not done by sed itself, but by the shell that invokes it; because we aren't invoking any local shell at all, we also needed to use os.path.expanduser to cause the ~ in ~/.ssh/my-ssh-key to be honored).
Because we aren't using a raw string, we need to double the backslashes in \b to ensure that they're treated as literal rather than syntactic by Python.
Critically, we're never passing data in a context where it could be parsed as code by any shell, either local or remote.
I'm creating a python program that calls a number of other programs and scripts (on Unix(SUNos) + Linux). I'm using subprocess everywhere except for 1 script.
The script for which I don't use subprocess is a perl-script which has been made into an executable. Somehow it does not let me use subprocess on, but it works with the (deprecated) commands package.
I would like to understand why it does not work with subprocess (in other words: what am I doing wrong ;-) )
(What the actual perl command is not important, but it returns the full name and email of a user as result)
What I tried:
PERL_CMD = [ '<executable perl-script>', '-rt', '"users"', '-eq', '"name"' '"<user_name>", '-fs', '":"', '-fld', '"fullname"', '"email"' ]
full_name, email = subprocess.check_output( PERL_CMD ).split(':')
But this does not work.
Where the commands variant does work:
PERL_CMD = '<executable perl-script> -rt "users" -eq "name" "<user_name>" -fs ":" -fld "full_name" "email"'
full_name, email = commands.getoutput( PERL_CMD ).split(':')
Has anybody an idea why I can't get subprocess to work?
It is annoying me that I can get it to work for everything except this (eventhough I have an acceptable (but deprecated) workaround).
You're using syntactic quotes in the commands.getoutput() case, and literal quotes in the subprocess.check_output() case. Without shell=True (which you shouldn't use), there's no shell to parse quotes as syntax, so there's no such thing as a syntactic quote, other than the quotes that are syntax to Python itself.
So, just take out the "s that you injected into your arguments:
# this contains quotes that are syntactic to Python only, and no literal quotes
perl_cmd = [
'<executable perl-script>',
'-rt', 'users',
'-eq', 'name', '<user_name>',
'-fs', ':',
'-fld', 'fullname', 'email' ]
To explain a bit more detail --
When you pass "name" to a shell as part of a command, the quotes are consumed by the shell itself during its parsing process, not passed to the command as an argument. Thus, when you run sh -c 'echo "hello"', this passes the exact same argument to echo as sh -c 'echo hello'; the echo command can't even tell the difference between the two invocations!
When you pass '"hello"' as an argument to subprocess.Popen(), by contrast, the outer quotes are consumed by Python, and the inner quotes are passed as literal to the inner command. That makes it equivalent to sh -c 'echo "\"hello\""' (which likewise passes literal quotes through to echo), not sh -c 'echo "hello"' (which does not).
I'm using Python code to run a Hadoop program on a Linux (Cloudera) machine using SSH.
I'm having some trouble with compiling java files to class files. When I'm executing the command:
javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/* from the Linux terminal all the files get compiled successfully.
When I'm executing the same command through my Python SSH client I'm receiving an 'invalid flag' error:
spur.results.RunProcessError: return code: 2
output: b''
stderr output: b'javac: invalid flag: remote_hadoop/javasrc\nUsage: javac \nuse -help for a list of possible options\n'
The python code:
list_of_commands = ["javac", "-cp", r"/usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*", input_folder + r"/*"]
print ' '.join(list_of_commands)
self.shell.run(list_of_commands)
The command is getting rendered correctly, since what is getting printed is javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/*.
UPDATE: It's pretty weird. I can compile one file at a time over ssh, but not all of them. Seems like something happens to the "*" over ssh.
You're passing a list of arguments, not a list of commands. It's not even an accurate list of arguments.
If your underlying tool expects a list of arguments, then pass:
['sh', '-c', 'javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/*']
If it expects a list of commands:
['javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/*']
If it expects something else -- read the documentation and determine what that something is!
Note that SSH doesn't provide a way to pass a literal argv array when running an arbitrary command; rather, it expects -- at the protocol level -- a string ready for parsing by the remote shell. If your self.shell.run code is doing shell quoting before joining the argument list given, then it would be passing the last argument as the literal string remote_hadoop/javasrc/* -- not expanding it into a list of filenames as a shell would.
Using the sh -c form forces the remote shell to perform expansion on its end, assuming that contents are being given to it in a form which doesn't have remote expansion performed already.
The problem is the way that spur builds the command list into a command string. It takes every command token and encloses it in single quotes (["ls", "*.txt"]) becomes 'ls' '*.txt'). There is no shell expansion of * inside quotes, so the command doesn't work.
You can see the problem in spur's ssh.py on line 323:
def escape_sh(value):
return "'" + value.replace("'", "'\\''") + "'"
I don't use spur, but it looks like it just doesn't allow you to do such things. The problem with "simplifiers" like spur is that if they simplify in a way you don't want, you can't use them.