Develop Reference

Use python script to prepare command line options

I am writing a python script to prepare command line options for another command (lame, but the issue is interesting irrespective of that). My plan is to use my script in this way:
lame $(python my_script.py)
I don't want to call lame from within my script, because I still want to be able to add additional options.
My problem is that print-statements from within my script get masked. So if my_script.py looks like this:
print('--tt "Some title" "a quoted filename.wav" "a quoted filename.mp3"')
Then the quotes are ignored and lame receives the arguments ['--tt', '"Some', 'title"', '"a', 'quoted', 'filename.wav"', '"a', 'quoted', 'filename.mp3"'] and obviously is not able to make any sense of it. How should the print command in python look to make sure the outer command receives the arguments ['--tt', 'Some title', 'a quoted filename.wav', 'a quoted filename.mp3']?
Note: I'm writing the command line arguments as they would be passed to python's subprocess module.

Assuming the arguments (e.g. filenames) do not contain newline characters,
would you please try:
my_script.py:
args = ['--tt', 'Some title', 'a quoted filename.wav', 'a quoted filename.mp3']
print('\n'.join(args))
bash script:
readarray -t args < <(python my_script.py)
mylame "${args[#]}"
The python script merges the argument list with a newline character
as a delimiter,
then the bash script splits the string on the newline characters and
assigns an array to retrieve the list of arguments.
As an edge case, if the filenames may contain newline characters, you can
pick another character as the delimiter. The null character will be the
safest choice.

The problem is that bash only does quoting and expansion once, so it won't do quoting again on the output of your python script. To fix this do:
eval lame $(python my_script.py) more args
eval re-runs the quoting and expansion rules on the command line.

python argparse stops parsing after it encounters '$'

I am trying to parse a command line using argparse
from argparse import ArgumentParser
argparser = ArgumentParser(prog="parse", description="desc")
create.add_argument("--name",dest="name",required=True,help="Name for element")
args = argparser.parse_args()
print(args)
When I execute this with below command
python argparser.py --name "input$output$"
The output is:
('args:', Namespace(name='input$'))
Expected Output:
('args:', Namespace(name='input$output$'))
Can you please help figure out what am I doing wrong ?
Why argparse stops parsing after encountering a special char?

This is because most shells consider strings starting with $ as a variable, and when quoted with double quotes, the shell tries to replace it with its value.
Jut open a terminal/console and type this in the shell (this works in both bash and fish):
echo "hi$test" # prints hi trying to interpolate the variables 'test'
echo 'hi$test' # prints hi$test no interpolation for single quotes
This happens before the shell starts application processes. So
I think when calling your application, you'd need to pass in the string quoted by single quotes, or escape the $ with a backslash.
echo "hi\$test" # prints hi$test since $ is escaped
If you want to see what Python actually receives from the shell as an argument, directly inspect sys.argv (that's where argparse and other modules a like read the command line arguments).
import sys
print sys.argv
In this specific case in the question, what happens is that your shell parses the input$output$ and tries to interpolate the variable $output, but there no such variable defined, so it gets replaced by an empty string. So what is actually being passed to Python as the argument is input$ (the last dollar sign stays in there because it's just a single dollar sign and can not be the name of a variable).

This may be related to your shell environment, since in bash, $ signals the start of a variable. $output would probably be substituted for the empty string. $ on its own won't be substituted.

Ignore evaluation of '\n' character in sys.argv

Oddly enough, when running this program with the arguments of
program.py "(lp0\nS'cat'\np1\naI5\na."
With program.py being:
import sys,pickle
print sys.argv[1]=="(lp0\nS'cat'\np1\naI5\na."
False is printed... I have narrowed the difference in evaluation to the \n character however I can find no way of ignoring such.
Why is this and how can I fix it?

You need to use raw string literal like this:
sys.argv[1] == r"(lp0\nS'cat'\np1\naI5\na."
Also, you can use a string in the parameters without quotes.

It is because the syntax of strings in Python and in the shell (presumably Bash) is different.
You may want to run the program as
echo $'"(lp0\nS\'cat\'\np1\naI5\na.'
program.py $'"(lp0\nS\'cat\'\np1\naI5\na.'

Passing newline within string into a python script from the command line

I have a script that I run from the command line which I would like to be able to pass string arguments into. As in
script.py --string "thing1\nthing2"
such that the program would interpret the '\n' as a new line. If string="thing1\nthing2" I want to get
print string
to return:
thing1
thing2
rather than thing1\nthing2
If I simply hard-code the string "thing1\nthing2" into the script, it does this, but if it's entered as a command line argument via getopt, it doesn't recognize it. I have tried a number of approaches to this: reading in the cl string as r"%s" % arg, various ways of specifying it on the commandline, etc, and nothing seems to work. Ideas? Is this completely impossible?

From https://stackoverflow.com/a/4918413/478656 in Bash, you can use:
script.py --string $'thing1\nthing2'
e.g.
$ python test.py $'1\n2'
1
2
But that's Bash-specific syntax.

This is really a shell question since the shell does all the command parsing. Python doesn't care what's happening with that and only gets what comes through in the exec system call. If you're using bash, it doesn't do certain kinds of escaping between double quotes. If you want things like \n, \t, or \xnn to be escaped, the following syntax is a bash extension:
python test.py $'thing1\nthing2'
Note that the above example uses single quotes and not double quotes. That's important. Using double quotes causes different rules to apply. You can also do:
python test.py "thing1
thing2"
Here's some more info on bash quoting if you're interested. Even if you're not using bash, it's still good reading:
http://mywiki.wooledge.org/Quotes

This one is relatively simple and I am surprised no one has said it.
In your python script just write the following code
print string.replace("\\n", "\n")
and you will get the string printed with the new line and not the \n.

Space character in python that won't be interpreted by bash

I have a Python script script.py which I am using to generate the command line argument to another script exactly like so:
./main $(./script.py)
The output of script.py may contain spaces (e.g. foo bar) which are being unintentionally interpreted by the shell. I want the argument to ./main to be the single string "foo bar". Of course I can solve this problem if I quote the argument to ./main, like this:
./main "$(./script.py)"
But I can't and don't want to do that. (The reason is because ./main is being called without quotes from another script which I don't have control to edit.)
Is there an alternative representation of the space character that my Python script can use, and that bash won't interpret?

You can try to have ./script.py output a non-breaking space (U+00a0) instead of a regular space, which bash will not use for word-splitting. However, I would file a bug report to have the script that calls main add quotes to its argument. Whether this works depends on how main reacts to getting a string that consists of a two-byte UTF-8 sequence representing U+00a0 rather than a single space character.
A sample script.py:
#!/usr/bin/python
print u'foo\xa0bar'.encode('utf8')
A sample script a.bash:
#!/bin/bash
main () {
echo $#
}
main $(script.py)
And finally, a demonstration that main gets 1 argument from the output of script.py:
$ bash a.bash
1

You can change your python script to escape the spaces in the output if you cannot quote the calling code.
See this code snippet:
arg1() { echo "$1"; }
arg1 abc def
abc
arg1 abc\ def
abc def
So as you can see if you output abc\ def instead of abc def from python code it will be considered single argument.

Assuming that:
you have control over ./main and that it is a shell script
the entire output from the Python script is to be interpreted as a single parameter
simply use $* rather than $1 inside ./main