I am trying to call ssh from a Python program but it seems to be ignoring the arguments.
This is the Python program:
#!/usr/bin/python
from subprocess import Popen, PIPE, call
vm_name = 'vmName with-space'
vm_host = 'user#192.168.21.230'
def ssh_prefix_list(host=None):
if host:
# return ["ssh", "-v", "-v", "-v", host]
return ["scripts/ssh_wrapper", "-v", "-v", "-v", host]
else:
return []
def start(vm_name, vm_host=None): # vm_host defaults to None
print "vm_host = ", vm_host
vbm_args = ssh_prefix_list(vm_host) + ["VBoxManage", "startvm", vm_name]
print vbm_args
return call(vbm_args, shell=True)
start(vm_name, vm_host)
The wrapper prints the number of arguments, their values, and calls ssh.
#!/bin/bash
echo Number of arguments: $#
echo ssh arguments: "$#"
ssh "$#"
This is the output.
$ scripts/vm_test.py
vm_host = stephen#192.168.21.230
['scripts/ssh_wrapper', '-v', '-v', '-v', 'stephen#192.168.21.230', 'VBoxManage', 'startvm', 'atp-systest Clone']
Number of arguments: 0
ssh arguments:
usage: ssh [-1246AaCfgKkMNnqsTtVvXxY] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-e escape_char] [-F configfile]
[-i identity_file] [-L [bind_address:]port:host:hostport]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-R [bind_address:]port:host:hostport] [-S ctl_path]
[-w local_tun[:remote_tun]] [user#]hostname [command]
This is on Python 2.5.
When you are using shell=True , you need to pass in a string, not a list of arguments. Try -
return call(' '.join(vbm_args), shell=True)
Also, you should consider constructing the string from start, rather than list.
When you pass a list to call() or Popen() with shell=True , only the first element in the list is actually called , and that is the reason you are seeing the wrapper called with 0 arguments.
You should also try first without using shell=True , since its a security hazard, as clearly stated in the documentation of subprocess -
Using shell=True can be a security hazard. See the warning under Frequently Used Arguments for details.
I think this might be it:
prefix_list = ssh_prefix_list(vm_host)
prefix_list.append(["VBoxManage startvm %s" % vm_name])
however I would strongly recommend using paramiko - it makes things much easier.
When you use call with shell=True, you need to pass a single string, not an array of strings. So:
call("scripts/ssh_wrapper -v -v -v "+host+" VBoxManage startvm "+vmname)
Related
I am using subparsers to implement the following function:
import argparse
parser = argparse.ArgumentParser(prog='dj')
parser.add_argument('--log','-l',type=str, help='log file')
parser.add_argument('--parser','-p',type=bool, nargs='?', const=True, default=False, help='wheher parsing a dj file or not')
subparsers = parser.add_subparsers(help='execute specified command')
###############################################################################
parser_rt = subparsers.add_parser('run_test', help='kick off a case')
parser_rt.add_argument('--test', '-t', type=str, help='test case name')
parser_rt.add_argument('--config','-c', type=str, help='config name')
parser_rt.set_defaults(func=run_test)
everything is fine when kicking off the command:
dj -p -l dj.log run_test -t testa -c configa
or
dj -l dj.log run_test -t testa -c configa
However, issue was popping up when running the command:
dj -l dj.log -p run_test -t testa -c configa
Here is error:
usage: dj [-h] [--log log] [--parser [PARSER]]
{run_test}...
dj: error: invalid choice: 'testa' (choose from 'run_test')
It seems that the issue is related to the position of -p.
By defining -p as nargs='?', const=True, default=False it still consumes the next argument, just not doing anything with it...
When it has run_test after it, it is consumed as an argument and not as a subparser and then -t is not defined either...
To overcome this, simply use the idiomatic way for defining "flag" arguments which is either store_true or store_false as the action. So to fix your parser, change to:
parser.add_argument('--parser','-p', action='store_true', help='wheher parsing a dj file or not')
When you ran -p -l there was no issue because argparse knows to differentiate arguments using the -. So in this case -p had no value and -l started a new argument. run_test was taken as a value before argparse checked it for a subparser.
In your code, -p might take an argument. Is that intended or do you rather want just a flag?
parser.add_argument('--parser','-p', action="store_true", help='whether parsing a dj file or not')
When you run dj -l dj.log -p run_test -t testa -c configa the parser thinks that run_test is an argument to -p.
If I have the arguments '-a', '-b', '-c', '-d', with the add_mutually_exclusive_group() function my program will have to use just one of them. Is there a way to combine that, so that the program will accept only either '-a 999 -b 999' or '-c 999 -d 999'?
Edit: adding a simple program for more clarity:
>>> parser = argparse.ArgumentParser()
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('-a')
>>> group.add_argument('-b')
>>> group.add_argument('-c')
>>> group.add_argument('-d')
Then only ./app.py -a | ./app.py -b | ./app.py -c | ./app.py -d can be called. Is it possible to have argparse group the exclusion groups, so that only ./app.py -a .. -b .. | ./app.py -c .. -d .. be called?
EDIT: Never mind. Because argparse makes the horrible choice of having to create an option when invoking group.add_argument. That wouldn't be my design choice. If you're desperate for this feature, you can try doing it with ConflictsOptionParser:
# exclusivegroups.py
import conflictsparse
parser = conflictsparse.ConflictsOptionParser()
a_opt = parser.add_option('-a')
b_opt = parser.add_option('-b')
c_opt = parser.add_option('-c')
d_opt = parser.add_option('-d')
import itertools
compatible_opts1 = (a_opt, b_opt)
compatible_opts2 = (c_opt, d_opt)
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
parser.register_conflict(exclusive_grp)
opts, args = parser.parse_args()
print "opts: ", opts
print "args: ", args
Thus when we invoke it, we can see we get the desired effect.
$ python exclusivegroups.py -a 1 -b 2
opts: {'a': '1', 'c': None, 'b': '2', 'd': None}
args: []
$ python exclusivegroups.py -c 3 -d 2
opts: {'a': None, 'c': '3', 'b': None, 'd': '2'}
args: []
$ python exclusivegroups.py -a 1 -b 2 -c 3
Usage: exclusivegroups.py [options]
exclusivegroups.py: error: -b, -c are incompatible options.
The warning message doesn't inform you that both '-a' and '-b' are incompatible with '-c', however a more appropriate error message could be crafted. Older, wrong answer below.
OLDER EDIT: [This edit is wrong, although wouldn't it be just a perfect world if argparse worked this way?] My previous answer actually was incorrect, you should be able to do this with argparse by specifying one group per mutually exclusive options. We can even use itertools to generalize the process. And make it so we don't have to type out all the combinations explicitly:
import itertools
compatible_opts1 = ('-a', '-b')
compatible_opts2 = ('-c', '-d')
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
group = parser.add_mutually_exclusive_group()
group.add_argument(exclusive_grp[0])
group.add_argument(exclusive_grp[1])
Just stumbled on this problem myself. From my reading of the argparse docs, there doesn't seem to be a simple way to achieve that within argparse. I considered using parse_known_args, but that soon amounts to writing a special-purpose version of argparse ;-)
Perhaps a bug report is in order. In the meanwhile, if you're willing to make your user do a tiny bit extra typing, you can fake it with subgroups (like how git and svn's arguments work), e.g.
subparsers = parser.add_subparsers()
p_ab = subparsers.add_parser('ab')
p_ab.add_argument(...)
p_cd = subparsers.add_parser('cd')
p_cd.add_argument(...)
Not ideal, but at least it gives you the good from argparse without too much ugly hackery. I ended up doing away with the switches and just using the subparser operations with required subarguments.
The argparse enhancement request referenced in #hpaulj's comment is still open after more than nine years, so I figured other people might benefit from the workaround I just discovered. Based on this comment in the enhancement request, I found I was able to add an option to two different mutually-exclusive groups using this syntax:
#!/usr/bin/env python
import argparse
import os
import sys
def parse_args():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument("-d", "--device", help="Path to UART device", default="./ttyS0")
mutex_group1 = parser.add_mutually_exclusive_group()
mutex_group2 = parser.add_mutually_exclusive_group()
mutex_group1.add_argument(
"-o",
"--output-file",
help="Name of output CSV file",
default="sensor_data_sent.csv",
)
input_file_action = mutex_group1.add_argument(
"-i", "--input-file", type=argparse.FileType("r"), help="Name of input CSV file"
)
# See: https://bugs.python.org/issue10984#msg219660
mutex_group2._group_actions.append(input_file_action)
mutex_group2.add_argument(
"-t",
"--time",
type=int,
help="How long to run, in seconds (-1 = loop forever)",
default=-1,
)
# Add missing ']' to usage message
usage = parser.format_usage()
usage = usage.replace('usage: ', '')
usage = usage.replace(']\n', ']]\n')
parser.usage = usage
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
print("Args parsed successfully...")
sys.exit(0)
This works well enough for my purposes:
$ ./fake_sensor.py -i input.csv -o output.csv
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -o/--output-file: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv -t 30
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -t/--time: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv -t 30
Args parsed successfully...
Accessing private members of argparse is, of course, rather brittle, so I probably wouldn't use this approach in production code. Also, an astute reader may notice that the usage message is misleading, since it implies that -o and -i can be used together when they cannot(!) However, I'm using this script for testing only, so I'm not overly concerned. (Fixing the usage message 'for real' would, I think, require much more time than I can spare for this task, but please comment if you know a clever hack for this.)
Subparsers?
Similar to unhammer's answer, but with more user control. Note: I have not actually tested this method, but it should work in theory and with the capabilities of python.
You can create two parsers, one for each of the two groups, and use conditionals to do the mutually exclusive part. Essentially using argparse for only part of the argument parsing. Using this method, you can go beyond the limitations of unhammer's answer as well.
# Python 3
import argparse
try:
parser = argparse.ArgumentParser()
parser.add_argument('-a')
parser.add_argument('-b')
args = parser.parse_args
except argparse.ArgumentError:
parser = argparse.ArgumentParser()
parser.add_argument('-c')
parser.add_argument('-d')
args = parser.parse_args
How do I define an arbitrary string as an optional argument using argparse?
Example:
[user#host]$ ./script.py FOOBAR -a -b
Running "script.py"...
You set the option "-a"
You set the option "-b"
You passed the string "FOOBAR"
Ideally, I'd like the position of the arguments to not matter. i.e:
./script.py -a FOOBAR -b == ./script.py -a -b FOOBAR == ./script.py FOOBAR -a -b
In BASH, I can accomplish this while using getopts. After handling all desired switches in a case loop, I'd have a line that reads shift $((OPTIND-1)), and from there I can access all remaining arguments using the standard $1, $2, $3, etc...
Does something like that exisit for argparse?
To get exactly what you're looking for, the trick is to use parse_known_args() instead of parse_args():
#!/bin/env python
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action="store_true")
parser.add_argument('-b', action="store_true")
opts = parser.parse_known_args()
# Print info about flags
if opts[0].a: print('You set the option "-a"')
if opts[0].b: print('You set the option "-b"')
# Collect remainder (opts[1] is a list (possibly empty) of all remaining args)
if opts[1]: print('You passed the strings %s' % opts[1])
EDIT:
The above code displays the following help information:
./clargs.py -h
usage: clargs_old.py [-h] [-a] [-b]
optional arguments:
-h, --help show this help message and exit
-a
-b
If you want to inform the user about the optional arbitrary argument, the only solution I can think of is to subclass ArgumentParser and write it in yourself.
For example:
#!/bin/env python
import os
import argparse
class MyParser(argparse.ArgumentParser):
def format_help(self):
help = super(MyParser, self).format_help()
helplines = help.splitlines()
helplines[0] += ' [FOO]'
helplines.append(' FOO some description of FOO')
helplines.append('') # Just a trick to force a linesep at the end
return os.linesep.join(helplines)
parser = MyParser()
parser.add_argument('-a', action="store_true")
parser.add_argument('-b', action="store_true")
opts = parser.parse_known_args()
# Print info about flags
if opts[0].a: print('You set the option "-a"')
if opts[0].b: print('You set the option "-b"')
# Collect remainder
if opts[1]: print('You passed the strings %s' % opts[1])
Which displays the following help information:
./clargs.py -h
usage: clargs.py [-h] [-a] [-b] [FOO]
optional arguments:
-h, --help show this help message and exit
-a
-b
FOO some description of FOO
Note the addition of the [FOO] in the "usage" line and the FOO in the help under "optional arguments".
I've read a bunch of examples but none of them work for this specific task.
Python code:
x = Popen(commands, stdout=PIPE, stderr=PIPE, shell=True)
print commands
stdout = x.stdout.read()
stderr = x.stderr.read()
print stdout, stderr
return stdout
Output:
[user#host]$ python helpers.py
['ssh', '-t', 'user#host', ' ', "'service --status-all'"]
usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-e escape_char] [-F configfile]
[-I pkcs11] [-i identity_file]
[-L [bind_address:]port:host:hostport]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-R [bind_address:]port:host:hostport] [-S ctl_path]
[-W host:port] [-w local_tun[:remote_tun]]
[user#]hostname [command]
Why am i getting this error?
Using os.popen(...) it works, it executes at least but i can't retrieve the output of the remote command via the SSH tunnel.
I think your commands list is wrong:
commands = ['ssh', '-t', 'user#host', "service --status-all"]
x = Popen(commands, stdout=PIPE, stderr=PIPE)
Additionally, I don't think you should pass shell=True if you're going to pass a list to Popen.
e.g. either do this:
Popen('ls -l',shell=True)
or this:
Popen(['ls','-l'])
but not this:
Popen(['ls','-l'],shell=True)
Finally, there exists a convenience function for splitting a string into a list the same way your shell would:
import shlex
shlex.split("program -w ith -a 'quoted argument'")
will return:
['program', '-w', 'ith', '-a', 'quoted argument']
If I have the arguments '-a', '-b', '-c', '-d', with the add_mutually_exclusive_group() function my program will have to use just one of them. Is there a way to combine that, so that the program will accept only either '-a 999 -b 999' or '-c 999 -d 999'?
Edit: adding a simple program for more clarity:
>>> parser = argparse.ArgumentParser()
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('-a')
>>> group.add_argument('-b')
>>> group.add_argument('-c')
>>> group.add_argument('-d')
Then only ./app.py -a | ./app.py -b | ./app.py -c | ./app.py -d can be called. Is it possible to have argparse group the exclusion groups, so that only ./app.py -a .. -b .. | ./app.py -c .. -d .. be called?
EDIT: Never mind. Because argparse makes the horrible choice of having to create an option when invoking group.add_argument. That wouldn't be my design choice. If you're desperate for this feature, you can try doing it with ConflictsOptionParser:
# exclusivegroups.py
import conflictsparse
parser = conflictsparse.ConflictsOptionParser()
a_opt = parser.add_option('-a')
b_opt = parser.add_option('-b')
c_opt = parser.add_option('-c')
d_opt = parser.add_option('-d')
import itertools
compatible_opts1 = (a_opt, b_opt)
compatible_opts2 = (c_opt, d_opt)
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
parser.register_conflict(exclusive_grp)
opts, args = parser.parse_args()
print "opts: ", opts
print "args: ", args
Thus when we invoke it, we can see we get the desired effect.
$ python exclusivegroups.py -a 1 -b 2
opts: {'a': '1', 'c': None, 'b': '2', 'd': None}
args: []
$ python exclusivegroups.py -c 3 -d 2
opts: {'a': None, 'c': '3', 'b': None, 'd': '2'}
args: []
$ python exclusivegroups.py -a 1 -b 2 -c 3
Usage: exclusivegroups.py [options]
exclusivegroups.py: error: -b, -c are incompatible options.
The warning message doesn't inform you that both '-a' and '-b' are incompatible with '-c', however a more appropriate error message could be crafted. Older, wrong answer below.
OLDER EDIT: [This edit is wrong, although wouldn't it be just a perfect world if argparse worked this way?] My previous answer actually was incorrect, you should be able to do this with argparse by specifying one group per mutually exclusive options. We can even use itertools to generalize the process. And make it so we don't have to type out all the combinations explicitly:
import itertools
compatible_opts1 = ('-a', '-b')
compatible_opts2 = ('-c', '-d')
exclusives = itertools.product(compatible_opts1, compatible_opts2)
for exclusive_grp in exclusives:
group = parser.add_mutually_exclusive_group()
group.add_argument(exclusive_grp[0])
group.add_argument(exclusive_grp[1])
Just stumbled on this problem myself. From my reading of the argparse docs, there doesn't seem to be a simple way to achieve that within argparse. I considered using parse_known_args, but that soon amounts to writing a special-purpose version of argparse ;-)
Perhaps a bug report is in order. In the meanwhile, if you're willing to make your user do a tiny bit extra typing, you can fake it with subgroups (like how git and svn's arguments work), e.g.
subparsers = parser.add_subparsers()
p_ab = subparsers.add_parser('ab')
p_ab.add_argument(...)
p_cd = subparsers.add_parser('cd')
p_cd.add_argument(...)
Not ideal, but at least it gives you the good from argparse without too much ugly hackery. I ended up doing away with the switches and just using the subparser operations with required subarguments.
The argparse enhancement request referenced in #hpaulj's comment is still open after more than nine years, so I figured other people might benefit from the workaround I just discovered. Based on this comment in the enhancement request, I found I was able to add an option to two different mutually-exclusive groups using this syntax:
#!/usr/bin/env python
import argparse
import os
import sys
def parse_args():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument("-d", "--device", help="Path to UART device", default="./ttyS0")
mutex_group1 = parser.add_mutually_exclusive_group()
mutex_group2 = parser.add_mutually_exclusive_group()
mutex_group1.add_argument(
"-o",
"--output-file",
help="Name of output CSV file",
default="sensor_data_sent.csv",
)
input_file_action = mutex_group1.add_argument(
"-i", "--input-file", type=argparse.FileType("r"), help="Name of input CSV file"
)
# See: https://bugs.python.org/issue10984#msg219660
mutex_group2._group_actions.append(input_file_action)
mutex_group2.add_argument(
"-t",
"--time",
type=int,
help="How long to run, in seconds (-1 = loop forever)",
default=-1,
)
# Add missing ']' to usage message
usage = parser.format_usage()
usage = usage.replace('usage: ', '')
usage = usage.replace(']\n', ']]\n')
parser.usage = usage
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
print("Args parsed successfully...")
sys.exit(0)
This works well enough for my purposes:
$ ./fake_sensor.py -i input.csv -o output.csv
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -o/--output-file: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv -t 30
usage: fake_sensor.py [-h] [-d DEVICE] [-o OUTPUT_FILE | [-i INPUT_FILE | -t TIME]]
fake_sensor.py: error: argument -t/--time: not allowed with argument -i/--input-file
$ ./fake_sensor.py -i input.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv
Args parsed successfully...
$ ./fake_sensor.py -o output.csv -t 30
Args parsed successfully...
Accessing private members of argparse is, of course, rather brittle, so I probably wouldn't use this approach in production code. Also, an astute reader may notice that the usage message is misleading, since it implies that -o and -i can be used together when they cannot(!) However, I'm using this script for testing only, so I'm not overly concerned. (Fixing the usage message 'for real' would, I think, require much more time than I can spare for this task, but please comment if you know a clever hack for this.)
Subparsers?
Similar to unhammer's answer, but with more user control. Note: I have not actually tested this method, but it should work in theory and with the capabilities of python.
You can create two parsers, one for each of the two groups, and use conditionals to do the mutually exclusive part. Essentially using argparse for only part of the argument parsing. Using this method, you can go beyond the limitations of unhammer's answer as well.
# Python 3
import argparse
try:
parser = argparse.ArgumentParser()
parser.add_argument('-a')
parser.add_argument('-b')
args = parser.parse_args
except argparse.ArgumentError:
parser = argparse.ArgumentParser()
parser.add_argument('-c')
parser.add_argument('-d')
args = parser.parse_args