Develop Reference

python argparse stops parsing after it encounters '$'

I am trying to parse a command line using argparse
from argparse import ArgumentParser
argparser = ArgumentParser(prog="parse", description="desc")
create.add_argument("--name",dest="name",required=True,help="Name for element")
args = argparser.parse_args()
print(args)
When I execute this with below command
python argparser.py --name "input$output$"
The output is:
('args:', Namespace(name='input$'))
Expected Output:
('args:', Namespace(name='input$output$'))
Can you please help figure out what am I doing wrong ?
Why argparse stops parsing after encountering a special char?

This is because most shells consider strings starting with $ as a variable, and when quoted with double quotes, the shell tries to replace it with its value.
Jut open a terminal/console and type this in the shell (this works in both bash and fish):
echo "hi$test" # prints hi trying to interpolate the variables 'test'
echo 'hi$test' # prints hi$test no interpolation for single quotes
This happens before the shell starts application processes. So
I think when calling your application, you'd need to pass in the string quoted by single quotes, or escape the $ with a backslash.
echo "hi\$test" # prints hi$test since $ is escaped
If you want to see what Python actually receives from the shell as an argument, directly inspect sys.argv (that's where argparse and other modules a like read the command line arguments).
import sys
print sys.argv
In this specific case in the question, what happens is that your shell parses the input$output$ and tries to interpolate the variable $output, but there no such variable defined, so it gets replaced by an empty string. So what is actually being passed to Python as the argument is input$ (the last dollar sign stays in there because it's just a single dollar sign and can not be the name of a variable).

This may be related to your shell environment, since in bash, $ signals the start of a variable. $output would probably be substituted for the empty string. $ on its own won't be substituted.

running python in an argument of a program & how to give hex when string expected

I've come across this command and want to understand how it works. program is just simple c program command for mac or linux.
./program `python -c 'print "\xC8\xCE\xC5\x06"'`
1) Can someone explain how this command works?
2) is this the only way to give a hex value to a program when string is expected?

This is a way of evaluating python expressions from the command line in bash. It has nothing to do with C. The only python code here is print "\xC8\xCE\xC5\x06". The rest is bash code.
You can try this command in bash python -c "print 'Hello World'"
You can also read man python for more information on python command line flags.
In python strings \xHH is used to translate a hex value into characters. In this case u"\xC8\xCE\xC5" == u"ÈÎÅ". If you don't use unicode strings, the output will be some non-ascii characters, which program might make sense of, but that cannot be entered or printed in a regular bash session. program might not care if the string is printable, and instead just treat it as binary data.
The backticks in bash will run the enclosed command first, and then use the string as a regular bash expression in the parent scope. Another way of doing this in bash would be this:
./program $(python -c "print '\xC8\xCE\xC5\x06'")
To answer you second question: There are other ways of doing this as well. You could probably use printf instead of python. Like this:
./program $(printf "\xC8\xCE\xC5\x06")

Passing newline within string into a python script from the command line

I have a script that I run from the command line which I would like to be able to pass string arguments into. As in
script.py --string "thing1\nthing2"
such that the program would interpret the '\n' as a new line. If string="thing1\nthing2" I want to get
print string
to return:
thing1
thing2
rather than thing1\nthing2
If I simply hard-code the string "thing1\nthing2" into the script, it does this, but if it's entered as a command line argument via getopt, it doesn't recognize it. I have tried a number of approaches to this: reading in the cl string as r"%s" % arg, various ways of specifying it on the commandline, etc, and nothing seems to work. Ideas? Is this completely impossible?

From https://stackoverflow.com/a/4918413/478656 in Bash, you can use:
script.py --string $'thing1\nthing2'
e.g.
$ python test.py $'1\n2'
1
2
But that's Bash-specific syntax.

This is really a shell question since the shell does all the command parsing. Python doesn't care what's happening with that and only gets what comes through in the exec system call. If you're using bash, it doesn't do certain kinds of escaping between double quotes. If you want things like \n, \t, or \xnn to be escaped, the following syntax is a bash extension:
python test.py $'thing1\nthing2'
Note that the above example uses single quotes and not double quotes. That's important. Using double quotes causes different rules to apply. You can also do:
python test.py "thing1
thing2"
Here's some more info on bash quoting if you're interested. Even if you're not using bash, it's still good reading:
http://mywiki.wooledge.org/Quotes

This one is relatively simple and I am surprised no one has said it.
In your python script just write the following code
print string.replace("\\n", "\n")
and you will get the string printed with the new line and not the \n.

Passing a command line argument to Python whos string contains a metacharacter

I am attempting to pass in a string as input argument to a Python program, from the command line i.e. $python parser_prog.py <pos1> <pos2> --opt1 --opt2 and interpreting these using argparse. Of course if contains any metacharacters these are first interpreted by the shell, so it needs to be quoted.
This seems to work, strings are passed through literally, preserving the \*?! characters:
$ python parser_prog.py 'str\1*?' 'str2!'
However, when I attempt to pass through a '-' (hyphen) character, I cannot seem to mask it. It is interpreted as an invalid option.
$ python parser_prog.py 'str\1*?' '-str2!'
I have tried single and double quotes, is there a way to make sure Python interprets this as a raw string? (I'm not in the interpreter yet, this is on the shell command line, so I can't use pythonic expressions such as r'str1')
Thank you for any hints!

As you said yourself, Python only sees the strings after being processed by the shell. The command-line arguments '-f' and -f look identical to the called program, and there is no way to dsitinguish them. That said, I think that argparse supports a -- argument to denote the end of the options, and everything after this is treated as a positional argument.

Passing arguments to a Python script from a shell variable containing quoted spaces

I am calling a python script through a bash wrapper, but I'm having trouble dealing with arguments that contain quoted spaces.
I assemble the arguments to the python script into a bash variable, such as
opt="-c start.txt"
opt+="--self 'name Na'"
Then call the python script with something like:
python test_args.py $opt
When printing sys.argv in Python, I get
['test-args.py', '-c', 'start.txt', '--self', "'name", "Na'"]
instead of the expected
['test-args.py', '-c', 'start.txt', '--self', 'name Na']
I tried using an array when calling the script, such as
python test_args.py ${opt[#]}
but then I get
['test-args.py', "-c start.txt --self 'name Na'"]
Any other ideas?

Use an array, but store each argument as a separate element in the array:
opt=(-c start.txt)
opt+=(--self 'name Na')
python test_args.py "${opt[#]}"
See BashFAQ #050.

This is what the shlex module is for.
The shlex class makes it easy to write lexical analyzers for simple
syntaxes resembling that of the Unix shell. This will often be useful
for writing minilanguages, (for example, in run control files for
Python applications) or for parsing quoted strings.

Your instinct to embed spaces inside the variable's value was good, but when the value is simply expanded during the command line parsing their special meaning is lost as you saw. You need to expand the variable before the command line to your python script is parsed:
set -f
eval python test_args.py $opt
set +f
That will expand to:
python test_args.py -c start.txt --self 'name Na'
Which will then be parsed correctly with the quotes regaining their special meaning.
Edit: I've added set -f/+f (aka -/+o noglob) around the eval to disable file globbing although that wasn't an issue in the OP's example that's not an unheard of issue with eval. (Another, stronger caveat is to never eval user input unless you take extreme care to make sure it won't blow up into something nasty. If you don't control the value being eval-ed, you can't be sure what will happen.)