Develop Reference

How to correctly escape special characters in python subprocess?

Im trying to run this bash command using python subprocess
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
output:-
helld.xl.py
parse_maillog.py
replace_pattern.py
split_text_match.py
ssh_bad_login.py
Here is what i have done in python2.7 way, but it gives the output where awk command filter is not working
>>> p1=subprocess.Popen(["find","/Users/johndoe/sandbox","-iname","*.py"],stdout=subprocess.PIPE)
>>> p2=subprocess.Popen(['awk','-F"/"','" {print $NF} "'],stdin=p1.stdout,stdout=subprocess.PIPE)
>>>p2.communicate()
('/Users/johndoe/sandbox/argparse.py\n/Users/johndoe/sandbox/custom_logic_substitute.py\n/Users/johndoe/sandbox/finditer_html_parse.py\n/Users/johndoe/sandbox/finditer_simple.py\n/Users/johndoe/sandbox/group_regex.py\n/Users/johndoe/sandbox/helo.py\n/Users/johndoe/sandbox/newdir/helld.xl.py\n/Users/johndoe/sandbox/parse_maillog.py\n/Users/johndoe/sandbox/replace_pattern.py\n/Users/johndoe/sandbox/split_text_match.py\n/Users/johndoe/sandbox/ssh_bad_login.py\n', None)
I could also get output by using p1 alone here like below,but i cant get the awk working here
list1=[]
result=p1.communicate()[0].split("\n")
for item in res:
a=item.rstrip('/').split('/')
list1.append(a[-1])
print list1

You are incorrectly passing in shell quoting (and extra shell quoting which isn't even required by the shell!) when you're not invoking a shell. Don't do that.
p2=subprocess.Popen(['awk', '-F/', '{print $NF}'], stdin=...
When you have shell=True you need extra quotes around some arguments to protect them from the shell, but there is no shell here, so putting them in is incorrect, and will cause parse errors by Awk.
However, you should almost never need to call Awk from Python, especially for trivial tasks which Python can easily do natively:
list1 = [line.split('/')[-1]
for line in subprocess.check_output(
["find", "/Users/johndoe/sandbox",
"-iname", "*.py"]).splitlines()]
In this particular case, note also that GNU find already has a facility to produce this result directly:
list1 = subprocess.check_output(
["find", "/Users/johndoe/sandbox",
"-iname", "*.py", "-printf", "%f\\n"]).splitlines()

Use this: p2.communicate()[0].split("\n").
It will output a list of lines.

if you don't have any reservation using shell=True , then this should be pretty simple solution
from subprocess import Popen
import subprocess
command='''
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
'''
process=Popen(command,shell=True,stdout=subprocess.PIPE)
result=process.communicate()
print result

Use awk in python script

I have to use awk print inside python script.
Below format I used
a = commands.getoutput(" ls -l | awk \'{print $1, $2}\' | awk \'{if(NR>3)print}\'"
I am getting below error:
KeyError: 'print $1, $2'
Can some one help me to fix.

Your Awk scripts can be simplified to a single script trivially. There is nothing here which really calls for using Awk anyway.
from subprocess import Popen, PIPE
with Popen(["ls", "-l"], stdout=PIPE, universal_newlines=True) as proc:
for index, line in enumerate(proc.stdout):
if index <= 2:
continue
print(' '.join(line.split()[0:2]))

"before" method returns unexpected newline character in Pexpect

Short scenario: I send telnet commands using Python and Pexpect library.
I encountered a "child.before" behavior that I can't change or understand. When I read the output of myResult=child.before I see the command that I sent(using child.sendline()), the result of the command and eventually the prompt.
Often,however,the command sent is not on a single line. If the command is long enough a newline character is introduced.
Example:
Code:
child.sendline('tail -f /myLocation | grep "something"')
child.expect('myPrompt',timeout=100)
myResult=child.before
print(myResult)
Result:
tail -f /myLocation | grep
"something"
Command output
Expected Result
tail -f /myLocation | grep "something"
Command output
Because of this I cannot safety use myResult.split('\n') method.
Any help will be appreciated.
Thank you!

How can I get my Python script to work using bash?

I am new to this site so hopefully this is the correct location to place this question.
I am trying to write a script using python for Linux, that:
creates a file file.txt
appends the output of the 'lsof' command to file.txt
read each line of the output and append them to an array.
then print each line.
I'm basically just doing this to familiarize myself with using python for bash, I'm new to this area so any help would be great. I'm not sure where to go from here. Also if there is a better way to do this I'm open to that!
#!/usr/bin/env python
import subprocess
touch = "touch file.txt"
subprocess.call(touch, shell=True)
xfile = "file.txt"
connection_count = "lsof -i tcp | grep ESTABLISHED | wc -l"
count = subprocess.call(connection_count, shell=True)
if count > 0:
connection_lines = "lsof -i tcp | grep ESTABLISHED >> file.txt"
subprocess.call(connection_lines, shell=True)
with open(subprocess.call(xfile, shell=True), "r") as ins:
array = []
for line in ins:
array.append(line)
for i in array:
print i

subprocess.call returns the return code for the process that was started ($? in bash). This is almost certainly not what you want -- and explains why this line almost certainly fails:
with open(subprocess.call(xfile, shell=True), "r") as ins:
(you can't open a number).
Likely, you want to be using subprocess.Popen with stdout=subprocess.PIPE. Then you can read the output from the pipe. e.g. to get the count, you probably want something like:
connection_count = "lsof -i tcp | grep ESTABLISHED"
proc = subprocess.POPEN(connection_count, shell=True, stdout=subprocess.PIPE)
# line counting moved to python :-)
count = sum(1 for unused_line in proc.stdout)
(you could also use Popen.communicate here)
Note, excessive use of shell=True is always a bit scary for me... It's much better to chain your pipes together as demonstrated in the documentation.

Escaping escape sequence in Python

I am kind of new to python. Goal is to execute a shell command using subprocess parse & retrive the printed output from shell. The execution errors out as shown in the sample output msg below. Also shown below is the sample code snippet
Code snippet:
testStr = "cat tst.txt | grep Location | sed -e '/.*Location: //g' "
print "testStr = "+testStr
testStrOut = subprocess.Popen([testStr],shell=True,stdout=subprocess.PIPE).communicate()[0]
Output:
testStr = cat tst.txt | grep Location | sed -e '/.*Location: //g'
cat: tst.txt: No such file or directory
sed: -e expression #1, char 15: unknown command: `/'
Is there a workaround or a function that could be used ?
Appreciate your help
Thanks

I suppose your main error is not python related. To be more precise, there are 3 of them:
You forgot to import subprocess.
It should be sed -e 's/.*Location: //g'. You wrote ///g instead of s///g.
tst.txt does not exist.

You should be passing testStr directly as the first argument, rather than enclosing it in a list. See subprocess.Popen, the paragraph that starts "On Unix, with shell=True: ...".