Develop Reference

How to correctly escape special characters in python subprocess?

Im trying to run this bash command using python subprocess
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
output:-
helld.xl.py
parse_maillog.py
replace_pattern.py
split_text_match.py
ssh_bad_login.py
Here is what i have done in python2.7 way, but it gives the output where awk command filter is not working
>>> p1=subprocess.Popen(["find","/Users/johndoe/sandbox","-iname","*.py"],stdout=subprocess.PIPE)
>>> p2=subprocess.Popen(['awk','-F"/"','" {print $NF} "'],stdin=p1.stdout,stdout=subprocess.PIPE)
>>>p2.communicate()
('/Users/johndoe/sandbox/argparse.py\n/Users/johndoe/sandbox/custom_logic_substitute.py\n/Users/johndoe/sandbox/finditer_html_parse.py\n/Users/johndoe/sandbox/finditer_simple.py\n/Users/johndoe/sandbox/group_regex.py\n/Users/johndoe/sandbox/helo.py\n/Users/johndoe/sandbox/newdir/helld.xl.py\n/Users/johndoe/sandbox/parse_maillog.py\n/Users/johndoe/sandbox/replace_pattern.py\n/Users/johndoe/sandbox/split_text_match.py\n/Users/johndoe/sandbox/ssh_bad_login.py\n', None)
I could also get output by using p1 alone here like below,but i cant get the awk working here
list1=[]
result=p1.communicate()[0].split("\n")
for item in res:
a=item.rstrip('/').split('/')
list1.append(a[-1])
print list1

You are incorrectly passing in shell quoting (and extra shell quoting which isn't even required by the shell!) when you're not invoking a shell. Don't do that.
p2=subprocess.Popen(['awk', '-F/', '{print $NF}'], stdin=...
When you have shell=True you need extra quotes around some arguments to protect them from the shell, but there is no shell here, so putting them in is incorrect, and will cause parse errors by Awk.
However, you should almost never need to call Awk from Python, especially for trivial tasks which Python can easily do natively:
list1 = [line.split('/')[-1]
for line in subprocess.check_output(
["find", "/Users/johndoe/sandbox",
"-iname", "*.py"]).splitlines()]
In this particular case, note also that GNU find already has a facility to produce this result directly:
list1 = subprocess.check_output(
["find", "/Users/johndoe/sandbox",
"-iname", "*.py", "-printf", "%f\\n"]).splitlines()

Use this: p2.communicate()[0].split("\n").
It will output a list of lines.

if you don't have any reservation using shell=True , then this should be pretty simple solution
from subprocess import Popen
import subprocess
command='''
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
'''
process=Popen(command,shell=True,stdout=subprocess.PIPE)
result=process.communicate()
print result

Python - Getting rid of unnecessary output

I am logging into a remote node using SSH, getting the status of a service and want to print it.
Running the bash command on my remote node yields.
[root#redis-1 ~]# redis-cli -a '!t3bmjEJss' info replication | grep role | cut -d':' -f2
slave
The python code that Ive written is
def serviceDetails(ip,svc):
if svc == 'redis-server':
ssh = subprocess.Popen(["ssh", "%s" % ip, "redis-cli -a '!t3Z9LJt2_wmUDbmjEJss' info replication | grep role | cut -d':' -f2"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
print (result)
else:
print ("Redis service is not running on this node")
The output that I am getting from result variable is:
[b'slave\r\n']
Why do all these extra characters appear ? And how can I get rid of them ?

The entire process of calling subprocess.Popen and then manually reading from its stdout property can be condensed into one call which will also automatically performs the bytes to string conversion:
subprocess.check_output([arg0, arg1, ...], encoding='utf-8')
If you also want to read stderr, then include a stderr=subprocess.STDOUT.
You can find the docs for subprocess.check_output here.

When you use .readlines(), it will return a list of lines. You can use .read() if you want it all in one string. It has the b there because it is a byte string. To get it to a normal string, you can use .decode('utf-8') in most cases. It may be a different encoding, but utf-8 will probably work. Then to get rid of the new line, you can use .strip(). Putting it all together, either of these would work:
result = ssh.stdout.read().decode('utf-8').strip()
print(result)
# slave
or
result = [line.decode('utf-8').strip() for line in ssh.stdout.readlines()]
print(result)
# ['slave']
Either one will work when you have only one line. If you have more than one line, the first will not work properly; it will have \r\n in the middle of the string.

python variables in awk

I like python and I like awk too, and I know that can use it via subprocess or command library, BUT I want to use awk with variables defined before in python, like this simple example:
file = 'file_i_want_read.list'
awk '{print $0}' file > another_file
anybody know how can I do it or something similar?

The easy way to do this is to not use the shell, and instead just pass a list of arguments to subprocess, so file is just one of those arguments.
The only trick is that if you don't use the shell, you can't use shell features like redirection; you have to use the equivalent subprocess features. Like this:
with open('another_file', 'wb') as output:
subprocess.check_call(['awk', '{print $0}', file], stdout=output)
If you really want to use shell redirection instead, then you have to build a shell command line. That's mainly just a matter of using your favorite Python string manipulation methods. But you need to be careful to make sure to quote and/or escape things—e.g., if file might be file i want read.list, then that will show up as 4 separate arguments unless you put it in quotes. shlex.quote can do that for you. So:
cmdline = "awk '{print $0}' %s > another_file" % (shlex.quote(file),)
subprocess.check_call(cmdline, shell=True)

How can I get my Python script to work using bash?

I am new to this site so hopefully this is the correct location to place this question.
I am trying to write a script using python for Linux, that:
creates a file file.txt
appends the output of the 'lsof' command to file.txt
read each line of the output and append them to an array.
then print each line.
I'm basically just doing this to familiarize myself with using python for bash, I'm new to this area so any help would be great. I'm not sure where to go from here. Also if there is a better way to do this I'm open to that!
#!/usr/bin/env python
import subprocess
touch = "touch file.txt"
subprocess.call(touch, shell=True)
xfile = "file.txt"
connection_count = "lsof -i tcp | grep ESTABLISHED | wc -l"
count = subprocess.call(connection_count, shell=True)
if count > 0:
connection_lines = "lsof -i tcp | grep ESTABLISHED >> file.txt"
subprocess.call(connection_lines, shell=True)
with open(subprocess.call(xfile, shell=True), "r") as ins:
array = []
for line in ins:
array.append(line)
for i in array:
print i

subprocess.call returns the return code for the process that was started ($? in bash). This is almost certainly not what you want -- and explains why this line almost certainly fails:
with open(subprocess.call(xfile, shell=True), "r") as ins:
(you can't open a number).
Likely, you want to be using subprocess.Popen with stdout=subprocess.PIPE. Then you can read the output from the pipe. e.g. to get the count, you probably want something like:
connection_count = "lsof -i tcp | grep ESTABLISHED"
proc = subprocess.POPEN(connection_count, shell=True, stdout=subprocess.PIPE)
# line counting moved to python :-)
count = sum(1 for unused_line in proc.stdout)
(you could also use Popen.communicate here)
Note, excessive use of shell=True is always a bit scary for me... It's much better to chain your pipes together as demonstrated in the documentation.

Why cannot pass arguments with subprocess.PIPE in python?

I'm trying to do something really easy in order to learn how to use subprocess in python
What I'm trying is this:
ll | egrep "*gz"
so after read the manual of python (which I didn't understand very well), I tried this:
lista = subprocess.Popen(['ls', '-alF'], stdout=subprocess.PIPE)
filtro = subprocess.Popen(['egrep', '"*gz"'], stdin=lista.stdout, stdout=subprocess.PIPE)
filtro.communicate()[0]
But all I get is '' and I don't really know how to do this, I've read this but it seems I didn't get it at all... could somebody explain to me how this works in order to use it after with other commands??
Thanks in advance!!

The problem might be the double set of quotes around the argument to egrep. Try this instead:
import subprocess
ls = subprocess.Popen(['ls', '-alF'], stdout=subprocess.PIPE)
egrep = subprocess.Popen(['egrep', '\.gz$'], stdin=ls.stdout, stdout=subprocess.PIPE)
print egrep.communicate()[0]
I am assuming you are looking for files ending in ".gz" here, as your initial regex does not make sense. If you are simply looking for files ending in "gz", you would use 'gz$' instead. And if you do not case where in the line "gz" appears, simply use 'gz'.
Edit: Here is a full example. In a directory containing the three files "pipe.py", "test1.py.gz" and "test2.py.gz", where "pipe.py" is the above script, I execute:
$ python pipe.py
With the result
-rw-r--r-- 1 amaurea amaurea 146 Jan 30 20:54 test1.py.gz
-rw-r--r-- 1 amaurea amaurea 150 Jan 30 20:54 test2.py.gz

On Unix, take advantage of shell argument:
lista = subprocess.Popen('ls -alF', stdout=subprocess.PIPE, shell=True)
filtro = subprocess.Popen('egrep "*gz"', stdin=lista.stdout, stdout=subprocess.PIPE, shell=True)
filtro.communicate()[0]
You can simply copy commands and do not wary about breaking them into argument lists.
Also you can specify shell explicitly:
subprocess.Popen('bash -c "ls -l"', stdout=subprocess.PIPE, shell=True)