Develop Reference

How do i make my awk command into a python command

I have an awk command that works in bash, but im now trying to put it into a python script
I have tried both os.system, and subprocess.call both return the same error. sh: 1: Syntax error: "(" unexpected
os.system('awk \'FNR<=27{print;next} ++count%10==0{print;count}\' \'{0} > {1}\'.format(inputfile, outpufile)')
So this awk command will take the large inputfile and create an output file that leaves the first 27 lines of header, but then starting on line 28 it only takes every 10th line and puts it into the output file
Im using the .format() because it is within a python script where the input file will be different every times its run.
ive also tried
subprocess.call('awk \'FNR<=27{print;next} ++count%10==0{print;count}\' \'{0} > {1}\'.format(inputfile, outpufile)')
both come up with the same error above. What am I missing?

As per the comment above, probably more pythonic (and more manageable) to directly use python.
But, if you want to use awk then one way is to format your command with your variable filenames separately.
This works using a basic test text file:
import os
def awk_runner(inputfile, outputfile):
cmd = "awk 'FNR<=27{print;next} ++count%10==0{print;count}' " + inputfile + " > " + outputfile
os.system(cmd)
awk_runner('test1.txt', 'testout1.txt')

There are two main issues with your Python code:
format() is a python method call, it should not be put into the string of awk_cmd to execute under the shell
when calling format() method, braces {} are used to identify substitution target in the format string objects, they need to be escaped using {{ ... }}
See below a modified version of your code:
awk_cmd = "awk 'FNR<=7{{print;next}} ++count%10==0{{print;count}}' {0} > {1}".format(inputfile, outpufile)
os.system(awk_cmd)

How can I embed a Python program in a bash command if it needs both a loop and an import?

I'm trying to use Python to extract info from some JSON (on a system where I can't install jq). My current approach runs afoul of the syntax restrictions described in Why can't use semi-colon before for loop in Python?. How can I modify this code to still work in light of this limitation?
My current code looks like the following:
$ SHIFT=$(aws ec2 describe-images --region "$REGION" --filters "Name=tag:Release,Values=$RELEASE_CODE_1.2003.2")
$ echo "$SHIFT" | python -c "import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];"
File "<string>", line 1
import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];
^
SyntaxError: invalid syntax
Since Python's syntax doesn't allow a for loop to be separated from a prior command with a semicolon, how can I work around this limitation?

There are several options here:
Pass your code as a multi-line string. Note that " is used to delimit Python strings rather than the original ' here for the sake of simplicity: A POSIX-compatible mechanism to embed a literal ' in a single-quoted string is possible, but quite ugly.
extractImageIds() {
python -c '
import sys, json
for image in json.load(sys.stdin)["Images"]:
print image["ImageId"]
' "$#"
}
Use bash's C-style escaped string syntax ($'') to embed newlines, as with $'\n'. Note that the leading $ is critical, and that this doesn't work with /bin/sh. See the bash-hackers' wiki on ANSI C-like strings for details.
extractImageIds() { python -c $'import sys, json\nfor image in json.load(sys.stdin)["Images"]:\n\tprint image["ImageId"]' "$#"; }
Use __import__() to avoid the need for a separate import command.
extractImageIds() { python -c 'for image in __import__("json").load(__import__("sys").stdin)["Images"]: print image["ImageId"]' "$#"; }
Pass the code on stdin and move the input onto argv; note that this only works if the input doesn't overwhelm your operating system's allowed maximum command-line size. Consider the following example:
extractImageIds() {
# capture function's input to a variable
local input=$(</dev/stdin) || return
# ...and expand that variable on the Python interpreter's command line
python - "$input" "$#" <<'EOF'
import sys, json
for image in json.loads(sys.argv[1])["Images"]:
print image["ImageId"]
EOF
}
Note that $(</dev/stdin) is a more efficient bash-only alternative to $(cat); due to shell builtin support, it works even on operating systems where /dev/stdin doesn't exist as a file.
All of these have been tested as follows:
extractImageIds <<<'{"Images": [{"ImageId": "one"}, {"ImageId": "two"}]}'
To efficiently provide stdin from a variable, one could run extractImageIds <<<"$variable" instead. Note that the "$#" elements in the wrapper are there to ensure that sys.argv is populated with arguments to the shell function -- where sys.argv isn't referenced by the Python code being run, this syntax is optional.

python -c vs python -<< heredoc

I am trying to run some piece of Python code in a Bash script, so i wanted to understand what is the difference between:
#!/bin/bash
#your bash code
python -c "
#your py code
"
vs
python - <<DOC
#your py code
DOC
I checked the web but couldn't compile the bits around the topic. Do you think one is better over the other?
If you wanted to return a value from Python code block to your Bash script then is a heredoc the only way?

The main flaw of using a here document is that the script's standard input will be the here document. So if you have a script which wants to process its standard input, python -c is pretty much your only option.
On the other hand, using python -c '...' ties up the single-quote for the shell's needs, so you can only use double-quoted strings in your Python script; using double-quotes instead to protect the script from the shell introduces additional problems (strings in double-quotes undergo various substitutions, whereas single-quoted strings are literal in the shell).
As an aside, notice that you probably want to single-quote the here-doc delimiter, too, otherwise the Python script is subject to similar substitutions.
python - <<'____HERE'
print("""Look, we can have double quotes!""")
print('And single quotes! And `back ticks`!')
print("$(and what looks to the shell like process substitutions and $variables!)")
____HERE
As an alternative, escaping the delimiter works identically, if you prefer that (python - <<\____HERE)

If you are using bash, you can avoid heredoc problems if you apply a little bit more of boilerplate:
python <(cat <<EoF
name = input()
print(f'hello, {name}!')
EoF
)
This will let you run your embedded Python script without you giving up the standard input. The overhead is mostly the same of using cmda | cmdb. This technique is known as Process Substitution.
If want to be able to somehow validate the script, I suggest that you dump it to a temporary file:
#!/bin/bash
temp_file=$(mktemp my_generated_python_script.XXXXXX.py)
cat > $temp_file <<EoF
# embedded python script
EoF
python3 $temp_file && rm $temp_file
This will keep the script if it fails to run.

If you prefer to use python -c '...' without having to escape with the double-quotes you can first load the code in a bash variable using here-documents:
read -r -d '' CMD << '--END'
print ("'quoted'")
--END
python -c "$CMD"
The python code is loaded verbatim into the CMD variable and there's no need to escape double quotes.

How to use here-docs with input
tripleee's answer has all the details, but there's Unix tricks to work around this limitation:
So if you have a script which wants to process its standard input, python -c is pretty much your only option.
This trick applies to all programs that want to read from a redirected stdin (e.g., ./script.py < myinputs) and also take user input:
python - <<'____HERE'
import os
os.dup2(1, 0)
print(input("--> "))
____HERE
Running this works:
$ bash heredocpy.sh
--> Hello World!
Hello World!
If you want to get the original stdin, run os.dup(0) first. Here is a real-world example.
This works because as long as either stdout or stderr are a tty, one can read from them as well as write to them. (Otherwise, you could just open /dev/tty. This is what less does.)
In case you want to process inputs from a file instead, that's possible too -- you just have to use a new fd:
Example with a file
cat <<'____HERE' > file.txt
With software there are only two possibilites:
either the users control the programme
or the programme controls the users.
____HERE
python - <<'____HERE' 4< file.txt
import os
for line in os.fdopen(4):
print(line.rstrip().upper())
____HERE
Example with a command
Unfortunately, pipelines don't work here -- but process substitution does:
python - <<'____HERE' 4< <(fortune)
import os
for line in os.fdopen(4):
print(line.rstrip().upper())
____HERE

Python equivalent to perl -pe?

I need to pick some numbers out of some text files. I can pick out the lines I need with grep, but didn't know how to extract the numbers from the lines. A colleague showed me how to do this from bash with perl:
cat results.txt | perl -pe 's/.+(\d\.\d+)\.\n/\1 /'
However, I usually code in Python, not Perl. So my question is, could I have used Python in the same way? I.e., could I have piped something from bash to Python and then gotten the result straight to stdout? ... if that makes sense. Or is Perl just more convenient in this case?

Yes, you can use Python from the command line. python -c <stuff> will run <stuff> as Python code. Example:
python -c "import sys; print sys.path"
There isn't a direct equivalent to the -p option for Perl (the automatic input/output line-by-line processing), but that's mostly because Python doesn't use the same concept of $_ and whatnot that Perl does - in Python, all input and output is done manually (via raw_input()/input(), and print/print()).
For your particular example:
cat results.txt | python -c "import re, sys; print ''.join(re.sub(r'.+(\d\.\d+)\.\n', r'\1 ', line) for line in sys.stdin)"
(Obviously somewhat more unwieldy. It's probably better to just write the script to do it in actual Python.)

You can use:
$ python -c '<your code here>'

You can in theory, but Python doesn't have anywhere near as much regex magic that Perl does, so the resulting command will be much more unwieldy, especially as you can't use regular expressions without importing re (and you'll probably need sys for sys.stdin too).
The Python equivalent of your colleague's Perl one-liner is approximately:
import sys, re
for line in sys.stdin:
print re.sub(r'.+(\d\.\d+)\.\n', r'\1 ', line)

You have a problem which can be solved several ways.
I think you should consider using regular expression (what perl is doing in your example) directly from Python. Regular expressions are in the re module. An example would be:
import re
filecontent = open('somefile.txt').read()
print re.findall('.+(\d\.\d+)\.$', filecontent)
(I would prefer using $ instead of '\n' for line endings, because line endings are different between operational systems and file encodings)
If you want to call bash commands from inside Python, you could use:
import os
os.system(mycommand)
Where command is the bash command. I use it all the time, because some operations are better to perform in bash than in Python.
Finally, if you want to extract the numbers with grep, use the -o option, which prints only the matched part.

Perl (or sed) is more convenient. However it is possible, if ugly:
python -c 'import sys, re; print "\n".join(re.sub(".+(\d\.\d+)\.\n","\1 ", l) for l in sys.stdin)'

Quoting from https://stackoverflow.com/a/12259852/411282:
for ln in __import__("fileinput").input(): print ln.rstrip()
See the explanation linked above, but this does much more of what perl -p does, including support for multiple file names and stdin when no filename is given.
https://docs.python.org/3/library/fileinput.html#fileinput.input

You can use python to execute code directly from your bash command line, by using python -c, or you can process input piped to stdin using sys.stdin, see here.

How can I pass the bash variable literally without any interpretation?

I have a bash code as follows
python "$TM"
The problem is that $TM can be whatever character, including ` characters. When $TM has `abc`, the bash tries to run abc as a command before giving it a parameter to python.
How can I prevent this? How can I pass the $TM literally without any interpretation?
ADDED
I need more explanation.
I'm using TextMate Bundle Editer so that the bash is called with a buffer ($TM_SELECTED_TEXT or $TM_CURRENT_LINE). The buffer is the selection I made in the TextMate editor. The bash code is as follows.
#!/bin/bash
if [ -n "$TM_SELECTED_TEXT" ]; then
TM="$TM_SELECTED_TEXT"
else
if [ -n "$TM_CURRENT_LINE" ]; then
TM="$TM_CURRENT_LINE"
fi
fi
/usr/bin/python /Users/smcho/smcho/works/prgtask/textmate/repeat.py "$TM"
The repeat.py is as follows
import sys
inputString = sys.stdin.read().decode('utf-8')
inputString = inputString.rstrip().lstrip()
content = inputString[0:-2]
mark = inputString[-1]
r_num = len(content)
string = "%s\n%s" % (content, mark * r_num)
sys.stdout.write(string)
sys.exit(0)
If the input is "abc:-", it will convert the string to "abc\n---".
The problem is that if the input contains `` character, bash evaluates it before sending it to python code as parameter.

I think you are getting it wrong. Bash didn't "expand" TM because it contained backticks (that would be a terrible security breach), the variable already contains the output of the command. You should quote the backticks to prevent the process substitution to occur:
$ TM="`ls`"
$ echo $TM
file1 file2
vs:
$ TM="\`ls\`" # or TM='`ls`'
$ echo $TM
`ls`

Why use bash as intermediary in the first place?
#!/usr/bin/env python
import os
tm = os.environ.get('TM_SELECTED_TEXT', "") or \
os.environ.get('TM_CURRENT_LINE', "")
and so on…
Your repeat.py wouldn't do anything with that argument anyway.

Your question is a bit vague but have you tried to see if quoting $TM to prevent word splitting solves your problem:
python "$TM"

python "$TM"