Develop Reference

Running a C executable inside a python program

I have written a C code where I have converted one file format to another file format. To run my C code, I have taken one command line argument : filestem.
I executed that code using : ./executable_file filestem > outputfile
Where I have got my desired output inside outputfile
Now I want to take that executable and run within a python code.
I am trying like :
import subprocess
import sys
filestem = sys.argv[1];
subprocess.run(['/home/dev/executable_file', filestem , 'outputfile'])
But it is unable to create the outputfile. I think some thing should be added to solve the > issue. But unable to figure out. Please help.

subprocess.run has optional stdout argument, you might give it file handle, so in your case something like
import subprocess
import sys
filestem = sys.argv[1]
with open('outputfile','wb') as f:
subprocess.run(['/home/dev/executable_file', filestem],stdout=f)
should work. I do not have ability to test it so please run it and write if it does work as intended

You have several options:
NOTE - Tested in CentOS 7, using Python 2.7
1. Try pexpect:
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import pexpect
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
command_output, exitstatus = pexpect.run("/usr/bin/bash -c '{0}'".format(cmd), withexitstatus=True)
if exitstatus == 0:
print(command_output)
else:
print("Houston, we've had a problem.")
2. Run subprocess with shell=true (Not recommended):
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import sys
import subprocess
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
result = subprocess.check_output(shlex.split(cmd), shell=True) # or subprocess.call(cmd, shell=True)
print(result)
It works, but python.org frowns upon this, due to the chance of a shell injection: see "Security Considerations" in the subprocess documentation.
3. If you must use subprocess, run each command separately and take the SDTOUT of the previous command and pipe it into the STDIN of the next command:
p = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE)
stdout_data, stderr_data = p.communicate()
p = subprocess.Popen(cmd, stdin=stdout_data, stdout=PIPE)
etc...
Good luck with your code!

| Not Working In Subprocess.call

Whenever I use a command in a subprocess with "|" in it doesn't work it has an output of
Command "|" is unknown, try "in link help".
Or when I put this:
#!/usr/bin/python
from subprocess import call
from shlex import split
interface = call(split("ip -o link show | awk '{print $2}' | grep wl"))
It is giving the output of:
Error: either "dev" is duplicate, or "awk" is a garbage.

You can use subprocess.check_output method and Popen class though I wasn't able to chain both pipe operations. Partial solution:
from subprocess import check_output, Popen, PIPE
from shlex import split
process = Popen(split('ip -o link show'), stdout=PIPE)
output = check_output(('awk', '{print $2}'), stdin=process.stdout)
return_code = process.wait()
print(output, return_code)
So basically, awk is taking the process standard output, and result is saved in the output variable.

Python - subprocess with Quotes and Pipe Grep

im having an issue trying to get a simple grep command into python. I want to take the output of the following command in a file or a list.
grep -c 'some thing' /home/user/* | grep -v :0
This is what I have, but its not working at all...
thing = str(subprocess.Popen(['grep', '-c', 'some thing', '/home/user/*', '|', 'grep', '-v', ':0'], stdout=subprocess.PIPE)
Basically I need to search files in a directory and return a result if my string is missing from any of the files in the directory.
Working Code (Thanks!!):
thing = subprocess.Popen(('grep -c "some thing" /home/user/* | grep -v ":0"' ),shell=True, stdout=subprocess.PIPE)

The pipe | is a shell feature. You have to use Popen with shell=True to use it.

To emulate the shell pipeline in Python, see How do I use subprocess.Popen to connect multiple processes by pipes?:
#!/usr/bin/env python
import os
from glob import glob
from subprocess import Popen, PIPE
p1 = Popen(["grep", "-c", 'some thing'] + glob(os.path.expanduser('~/*')),
stdout=PIPE)
p2 = Popen(["grep", "-v", ":0"], stdin=p1.stdout)
p1.stdout.close()
p2.wait()
p1.wait()
To get output as a string, set stdout=PIPE and call output = p2.communicate()[0] instead of p2.wait().
To suppress error messages such as "grep: /home/user/dir: Is a directory", you could set stderr=DEVNULL.
You could implement the pipeline in pure Python:
import os
from glob import glob
for name in glob(os.path.expanduser('~/*')):
try:
count = sum(1 for line in open(name, 'rb') if b'some thing' in line)
except IOError:
pass # ignore
else:
if count: # don't print zero counts
print("%s:%d" % (name, count))

How to run " ps cax | grep something " in Python?

How do I run a command with a pipe | in it?
The subprocess module seems complex...
Is there something like
output,error = `ps cax | grep something`
as in shell script?

See Replacing shell pipeline:
import subprocess
proc1 = subprocess.Popen(['ps', 'cax'], stdout=subprocess.PIPE)
proc2 = subprocess.Popen(['grep', 'python'], stdin=proc1.stdout,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out, err = proc2.communicate()
print('out: {0}'.format(out))
print('err: {0}'.format(err))
PS. Using shell=True can be dangerous. See for example the warning in the docs.
There is also the sh module which can make subprocess scripting in Python a lot more pleasant:
import sh
print(sh.grep(sh.ps("cax"), 'something'))

You've already accepted an answer, but:
Do you really need to use grep? I'd write something like:
import subprocess
ps = subprocess.Popen(('ps', 'cax'), stdout=subprocess.PIPE)
output = ps.communicate()[0]
for line in output.split('\n'):
if 'something' in line:
...
This has the advantages of not involving shell=True and its riskiness, doesn't fork off a separate grep process, and looks an awful lot like the kind of Python you'd write to process data file-like objects.

import subprocess
process = subprocess.Popen("ps cax | grep something",
shell=True,
stdout=subprocess.PIPE,
)
stdout_list = process.communicate()[0].split('\n')

Drop that 'ps' subprocess and back away slowly! :)
Use the psutil module instead.

import os
os.system('ps -cax|grep something')
If you wanna replace grep argument with some variable:
os.system('ps -cax|grep '+your_var)

Parsing a stdout in Python

In Python I need to get the version of an external binary I need to call in my script.
Let's say that I want to use Wget in Python and I want to know its version.
I will call
os.system( "wget --version | grep Wget" )
and then I will parse the outputted string.
How to redirect the stdout of the os.command in a string in Python?

One "old" way is:
fin,fout=os.popen4("wget --version | grep Wget")
print fout.read()
The other modern way is to use a subprocess module:
import subprocess
cmd = subprocess.Popen('wget --version', shell=True, stdout=subprocess.PIPE)
for line in cmd.stdout:
if "Wget" in line:
print line

Use the subprocess module:
from subprocess import Popen, PIPE
p1 = Popen(["wget", "--version"], stdout=PIPE)
p2 = Popen(["grep", "Wget"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Use subprocess instead.

If you are on *nix, I would recommend you to use commands module.
import commands
status, res = commands.getstatusoutput("wget --version | grep Wget")
print status # Should be zero in case of of success, otherwise would have an error code
print res # Contains stdout