I have a bash code as follows
python "$TM"
The problem is that $TM can be whatever character, including ` characters. When $TM has `abc`, the bash tries to run abc as a command before giving it a parameter to python.
How can I prevent this? How can I pass the $TM literally without any interpretation?
ADDED
I need more explanation.
I'm using TextMate Bundle Editer so that the bash is called with a buffer ($TM_SELECTED_TEXT or $TM_CURRENT_LINE). The buffer is the selection I made in the TextMate editor. The bash code is as follows.
#!/bin/bash
if [ -n "$TM_SELECTED_TEXT" ]; then
TM="$TM_SELECTED_TEXT"
else
if [ -n "$TM_CURRENT_LINE" ]; then
TM="$TM_CURRENT_LINE"
fi
fi
/usr/bin/python /Users/smcho/smcho/works/prgtask/textmate/repeat.py "$TM"
The repeat.py is as follows
import sys
inputString = sys.stdin.read().decode('utf-8')
inputString = inputString.rstrip().lstrip()
content = inputString[0:-2]
mark = inputString[-1]
r_num = len(content)
string = "%s\n%s" % (content, mark * r_num)
sys.stdout.write(string)
sys.exit(0)
If the input is "abc:-", it will convert the string to "abc\n---".
The problem is that if the input contains `` character, bash evaluates it before sending it to python code as parameter.
I think you are getting it wrong. Bash didn't "expand" TM because it contained backticks (that would be a terrible security breach), the variable already contains the output of the command. You should quote the backticks to prevent the process substitution to occur:
$ TM="`ls`"
$ echo $TM
file1 file2
vs:
$ TM="\`ls\`" # or TM='`ls`'
$ echo $TM
`ls`
Why use bash as intermediary in the first place?
#!/usr/bin/env python
import os
tm = os.environ.get('TM_SELECTED_TEXT', "") or \
os.environ.get('TM_CURRENT_LINE', "")
and so on…
Your repeat.py wouldn't do anything with that argument anyway.
Your question is a bit vague but have you tried to see if quoting $TM to prevent word splitting solves your problem:
python "$TM"
python "$TM"
Related
I'm trying to use Python to extract info from some JSON (on a system where I can't install jq). My current approach runs afoul of the syntax restrictions described in Why can't use semi-colon before for loop in Python?. How can I modify this code to still work in light of this limitation?
My current code looks like the following:
$ SHIFT=$(aws ec2 describe-images --region "$REGION" --filters "Name=tag:Release,Values=$RELEASE_CODE_1.2003.2")
$ echo "$SHIFT" | python -c "import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];"
File "<string>", line 1
import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];
^
SyntaxError: invalid syntax
Since Python's syntax doesn't allow a for loop to be separated from a prior command with a semicolon, how can I work around this limitation?
There are several options here:
Pass your code as a multi-line string. Note that " is used to delimit Python strings rather than the original ' here for the sake of simplicity: A POSIX-compatible mechanism to embed a literal ' in a single-quoted string is possible, but quite ugly.
extractImageIds() {
python -c '
import sys, json
for image in json.load(sys.stdin)["Images"]:
print image["ImageId"]
' "$#"
}
Use bash's C-style escaped string syntax ($'') to embed newlines, as with $'\n'. Note that the leading $ is critical, and that this doesn't work with /bin/sh. See the bash-hackers' wiki on ANSI C-like strings for details.
extractImageIds() { python -c $'import sys, json\nfor image in json.load(sys.stdin)["Images"]:\n\tprint image["ImageId"]' "$#"; }
Use __import__() to avoid the need for a separate import command.
extractImageIds() { python -c 'for image in __import__("json").load(__import__("sys").stdin)["Images"]: print image["ImageId"]' "$#"; }
Pass the code on stdin and move the input onto argv; note that this only works if the input doesn't overwhelm your operating system's allowed maximum command-line size. Consider the following example:
extractImageIds() {
# capture function's input to a variable
local input=$(</dev/stdin) || return
# ...and expand that variable on the Python interpreter's command line
python - "$input" "$#" <<'EOF'
import sys, json
for image in json.loads(sys.argv[1])["Images"]:
print image["ImageId"]
EOF
}
Note that $(</dev/stdin) is a more efficient bash-only alternative to $(cat); due to shell builtin support, it works even on operating systems where /dev/stdin doesn't exist as a file.
All of these have been tested as follows:
extractImageIds <<<'{"Images": [{"ImageId": "one"}, {"ImageId": "two"}]}'
To efficiently provide stdin from a variable, one could run extractImageIds <<<"$variable" instead. Note that the "$#" elements in the wrapper are there to ensure that sys.argv is populated with arguments to the shell function -- where sys.argv isn't referenced by the Python code being run, this syntax is optional.
I am using a bash script to call google-api's upload_video.py (https://developers.google.com/youtube/v3/guides/uploading_a_video )
I have a mp4 called output.mp4 which I would like to upload.
The problem is I cannot get my array to work how I would like.
This new line character is "required" because my arguments to python script contain spaces.
Here is a simplified version of my bash script:
# Operator may change these
hold=100
location="Foo, Montana "
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_file=upload_file.txt
upload_movie=output.mp4
# Hit enter at end b/c \n not recognized
upload_title=$location' - '${file_array[0]}' - Hold '$hold' Sweeps
'
upload_description='The spectrum recording was made in at '$location'.
'
# Overwrite with 1st call > else apppend >>
echo "$upload_title" > $upload_file
echo "$upload_description" >> $upload_file
# Load each line of text file into array
IFS=$'\n'
cmd_google=$(<$upload_file)
unset IFS
nn=1
for i in "${cmd_google[#]}"
do
echo "$i"
# Delete last character: \n
#i=${i[-nn]%?}
#i=${i: : -nn}
#i=${i::${#i}-nn}
i=${i%?}
#i=${i#"\n"}
#i=${i%"\n"}
echo "$i"
done
python upload_video.py --file=$upload_movie --title="${cmd_google[0]}" --description="${cmd_google[1]}"
At first I attempted to remove the new line character, but it appears that the enter or \n is not working how I would like, each line is not separate. It writes the title and description as one line.
How do I modify my bash script to recognize a newline character?
This is much simpler than you are making it.
# Operator may change these
hold=100
location="Foo, Montana"
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_file=upload_file.txt
upload_movie=output.mp4
upload_title="$location - ${file_array[0]} - Hold $hold Sweeps"
upload_description="The spectrum recording was made in at $location."
cat <<EOF > "$upload_file"
$upload_title
$upload_description
EOF
# ...
readarray -t cmd_google < "$upload_file"
python upload_video.py --file="$upload_movie" --title="${cmd_google[0]}" --description="${cmd_google[1]}"
I suspect the readarray command is all you are really looking for, since much of the above code is simply creating a file that I assume you are receiving already created.
I figured it out with help from chepner's answer. My question hid the fact that I wanted to write new line characters into the video's description.
Instead of adding a new line character in the bash script, it is much easier to have a text file which contains the correctly formatted script and read it in, then concatenate it with run-time specific variable.
In my case the correctly formatted text is called description.txt:
Here is a snip of my description.txt which contains newline characters
Here is my final version of the script:
# Operator may change these
hold=100
location="Foo, Montana"
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_title="$location - ${file_array[0]} - Hold $hold Sweeps"
upload_description="The spectrum recording was made in at $location. "
# Read in script which contains newline
temp=$(<description.txt)
# Concatenate them
upload_description="$upload_description$temp"
upload_movie=output.mp4
python upload_video.py --file="$upload_movie" --title="$upload_title" --description="$upload_description"
I have written a batch script that prompts the user for an input and displays the help for a specific library ExternalLibrary. However, I receive the error below the code sample.
#echo off
set input=NUL
set /p input="Help: "
python -c "import sys; sys.path.append('D:\\lib'); import ExternalLibrary;lookup = ExternalLibrary.Presentation_control();if ('%input%' == 'NUL'):& help(lookup.%input%)&else:& help(lookup)"
A working version can be achieved by replacing the if statement with:
help(lookup.%input%)
Error in the column that starts with if
C:\Users\usah>Lib_Help.cmd
Help:
File "<string>", line 1
import sys; sys.path.append('D:\\lib'); import ExternalLibrary;lookup = ExternalLibrary.Presentation_control();if ('NUL' == 'NUL'):& help(lookup.NU
L)&else:& help(lookup)
^
SyntaxError: invalid syntax
Footnotes
1. I am note sure whether I should pass &, \n, or ; as newline
2. These related answers are not satisfying as they use workarounds.
Only simple statements can be written on single line separated by semicolons
This was more interesting than it seemed. For the sake of unambiguity you can't write conditional or any other compound statement on semicolon separated single line.
Simple statements
Compound statements
Your options
Rewrite it all to Python
This should not be so hard. Use sys.argv for arguments and
getattr(lookup, "something")
instead of your
lookup.%input%
sys.argv
getattr
Use linefeeds
Write multiline Python script on one cmd line. You can do it using !LF! special.
#echo off
setlocal enableDelayedExpansion
set LF=^
set input=NUL
set /p input="Help: "
python -c "import sys!LF!sys.path.append('D:\\lib')!LF!import ExternalLibrary!LF!lookup = ExternalLibrary.Presentation_control()!LF!if ('%input%' == 'NUL'):!LF! help(lookup)!LF!else:!LF! help(lookup.%input%)!LF!"
I am trying to run some piece of Python code in a Bash script, so i wanted to understand what is the difference between:
#!/bin/bash
#your bash code
python -c "
#your py code
"
vs
python - <<DOC
#your py code
DOC
I checked the web but couldn't compile the bits around the topic. Do you think one is better over the other?
If you wanted to return a value from Python code block to your Bash script then is a heredoc the only way?
The main flaw of using a here document is that the script's standard input will be the here document. So if you have a script which wants to process its standard input, python -c is pretty much your only option.
On the other hand, using python -c '...' ties up the single-quote for the shell's needs, so you can only use double-quoted strings in your Python script; using double-quotes instead to protect the script from the shell introduces additional problems (strings in double-quotes undergo various substitutions, whereas single-quoted strings are literal in the shell).
As an aside, notice that you probably want to single-quote the here-doc delimiter, too, otherwise the Python script is subject to similar substitutions.
python - <<'____HERE'
print("""Look, we can have double quotes!""")
print('And single quotes! And `back ticks`!')
print("$(and what looks to the shell like process substitutions and $variables!)")
____HERE
As an alternative, escaping the delimiter works identically, if you prefer that (python - <<\____HERE)
If you are using bash, you can avoid heredoc problems if you apply a little bit more of boilerplate:
python <(cat <<EoF
name = input()
print(f'hello, {name}!')
EoF
)
This will let you run your embedded Python script without you giving up the standard input. The overhead is mostly the same of using cmda | cmdb. This technique is known as Process Substitution.
If want to be able to somehow validate the script, I suggest that you dump it to a temporary file:
#!/bin/bash
temp_file=$(mktemp my_generated_python_script.XXXXXX.py)
cat > $temp_file <<EoF
# embedded python script
EoF
python3 $temp_file && rm $temp_file
This will keep the script if it fails to run.
If you prefer to use python -c '...' without having to escape with the double-quotes you can first load the code in a bash variable using here-documents:
read -r -d '' CMD << '--END'
print ("'quoted'")
--END
python -c "$CMD"
The python code is loaded verbatim into the CMD variable and there's no need to escape double quotes.
How to use here-docs with input
tripleee's answer has all the details, but there's Unix tricks to work around this limitation:
So if you have a script which wants to process its standard input, python -c is pretty much your only option.
This trick applies to all programs that want to read from a redirected stdin (e.g., ./script.py < myinputs) and also take user input:
python - <<'____HERE'
import os
os.dup2(1, 0)
print(input("--> "))
____HERE
Running this works:
$ bash heredocpy.sh
--> Hello World!
Hello World!
If you want to get the original stdin, run os.dup(0) first. Here is a real-world example.
This works because as long as either stdout or stderr are a tty, one can read from them as well as write to them. (Otherwise, you could just open /dev/tty. This is what less does.)
In case you want to process inputs from a file instead, that's possible too -- you just have to use a new fd:
Example with a file
cat <<'____HERE' > file.txt
With software there are only two possibilites:
either the users control the programme
or the programme controls the users.
____HERE
python - <<'____HERE' 4< file.txt
import os
for line in os.fdopen(4):
print(line.rstrip().upper())
____HERE
Example with a command
Unfortunately, pipelines don't work here -- but process substitution does:
python - <<'____HERE' 4< <(fortune)
import os
for line in os.fdopen(4):
print(line.rstrip().upper())
____HERE
I am trying to format the following awk command
awk -v OFS="\t" '{printf "chr%s\t%s\t%s\n", $1, $2-1, $2}' file1.txt > file2.txt
for use in python subprocess popen. However i am having a hard time formatting it. I have tried solutions suggested in similar answers but none of them worked. I have also tried using raw string literals. Also i would not like to use shell=True as this is not recommended
Edit according to comment:
The command i tried was
awk_command = """awk -v OFS="\t" '{printf "chr%s\t%s\t%s\n", $1, $2-1, $2}' file1.txt > file2.txt"""
command_execute = Popen(shlex.split(awk_command))
However i get the following error upon executing this
KeyError: 'printf "chr%s\t%s\t%s\n", $1, $2-1, $2'
googling the error suggests this happens when a value is requested for an undefined key but i do not understand its context here
> is the shell redirection operator. To implement it in Python, use stdout parameter:
#!/usr/bin/env python
import shlex
import subprocess
cmd = r"""awk -v OFS="\t" '{printf "chr%s\t%s\t%s\n", $1, $2-1, $2}'"""
with open('file2.txt', 'wb', 0) as output_file:
subprocess.check_call(shlex.split(cmd) + ["file1.txt"], stdout=output_file)
To avoid starting a separate process, you could implement this particular awk command in pure Python.
The simplest method, especially if you wish to keep the output redirection stuff, is to use subprocess with shell=True - then you only need to escape Python special characters. The line, as a whole, will be interpreted by the default shell.
WARNING: do not use this with untrusted input without sanitizing it first!
Alternatively, you can replace the command line with an argv-type sequence and feed that to subprocess instead. Then, you need to provide stuff as the program would see it:
remove all the shell-level escaping
remove the output redirection stuff and do the redirection yourself instead
Regarding the specific problems:
you didn't escape Python special characters in the string so \t and \n became the literal tab and newline (try to print awk_command)
using shlex.split is nothing different from shell=True - with an added unreliability since it cannot guarantee if would parse the string the same way your shell would in every case (not to mention the lack of transmutations the shell makes).
Specifically, it doesn't know or care about the special meaning of the redirection part:
>>> awk_command = """awk -v OFS="\\t" '{printf "chr%s\\t%s\\t%s\\n", $1, $2- 1, $2}' file1.txt > file2.txt"""
>>> shlex.split(awk_command)
['awk','-v','OFS=\\t','{printf "chr%s\\t%s\\t%s\\n", $1, $2-1, $2}','file1.txt','>','file2.txt']
So, if you wish to use shell=False, do construct the argument list yourself.