I am using the subprocess module in Python (python 2.6) with the Popen method.
What I am trying to achieve:
I am trying to use the following bash command with Popen that will return a match if it finds a file with "stderr" string.
The code:
bash
find . -exec grep "stderr" {} +
what I am doing in python
from subprocess import Popen, PIPE
command = "find . -exec grep 'stderr' {} +"
stream = Popen(command, stdout=PIPE, shell=True, cwd=dir)
stream_stdout, stream_stderr = stream.communicate()
What I get bacK:
It looks to have worked as stream_stdout and stream_stderr return what I suspect,
but I am getting this txt sent to the screen:
find: missing argument to `-exec'
Any ideas why?
*EDIT:
I did not have a space between {}+ this is why I was getting the above out. Apologies! *
Cheers,
Mike
Try making your command a list of arguments:
command = "find . -exec grep 'stderr' {} +".split(" ")
Edit: Sorry, I didn't realize Popen() could take a string. This answer is not correct.
Why are you using the subprocess module?
Would this code, from Mr. Beazley, do the job?
import os
import fnmatch
import re
def gen_find(filepat,top):
for path, dirlist, filelist in os.walk(top):
for name in fnmatch.filter(filelist,filepat):
yield os.path.join(path,name)
def grep(pat,lines):
patc = re.compile(pat)
for line in lines:
if patc.search(line): return True
for f in gen_find("*","."):
fopen = open(f, 'rb')
if grep("stderr",fopen.readlines()):
print f
Related
I have written a C code where I have converted one file format to another file format. To run my C code, I have taken one command line argument : filestem.
I executed that code using : ./executable_file filestem > outputfile
Where I have got my desired output inside outputfile
Now I want to take that executable and run within a python code.
I am trying like :
import subprocess
import sys
filestem = sys.argv[1];
subprocess.run(['/home/dev/executable_file', filestem , 'outputfile'])
But it is unable to create the outputfile. I think some thing should be added to solve the > issue. But unable to figure out. Please help.
subprocess.run has optional stdout argument, you might give it file handle, so in your case something like
import subprocess
import sys
filestem = sys.argv[1]
with open('outputfile','wb') as f:
subprocess.run(['/home/dev/executable_file', filestem],stdout=f)
should work. I do not have ability to test it so please run it and write if it does work as intended
You have several options:
NOTE - Tested in CentOS 7, using Python 2.7
1. Try pexpect:
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import pexpect
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
command_output, exitstatus = pexpect.run("/usr/bin/bash -c '{0}'".format(cmd), withexitstatus=True)
if exitstatus == 0:
print(command_output)
else:
print("Houston, we've had a problem.")
2. Run subprocess with shell=true (Not recommended):
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import sys
import subprocess
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
result = subprocess.check_output(shlex.split(cmd), shell=True) # or subprocess.call(cmd, shell=True)
print(result)
It works, but python.org frowns upon this, due to the chance of a shell injection: see "Security Considerations" in the subprocess documentation.
3. If you must use subprocess, run each command separately and take the SDTOUT of the previous command and pipe it into the STDIN of the next command:
p = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE)
stdout_data, stderr_data = p.communicate()
p = subprocess.Popen(cmd, stdin=stdout_data, stdout=PIPE)
etc...
Good luck with your code!
This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
I want to assign the output of a command I run using os.system to a variable and prevent it from being output to the screen. But, in the below code ,the output is sent to the screen and the value printed for var is 0, which I guess signifies whether the command ran successfully or not. Is there any way to assign the command output to the variable and also stop it from being displayed on the screen?
var = os.system("cat /etc/services")
print var #Prints 0
From this question which I asked a long time ago, what you may want to use is popen:
os.popen('cat /etc/services').read()
From the docs for Python 3.6,
This is implemented using subprocess.Popen; see that class’s
documentation for more powerful ways to manage and communicate with
subprocesses.
Here's the corresponding code for subprocess:
import subprocess
proc = subprocess.Popen(["cat", "/etc/services"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
print("program output:", out)
You might also want to look at the subprocess module, which was built to replace the whole family of Python popen-type calls.
import subprocess
output = subprocess.check_output("cat /etc/services", shell=True)
The advantage it has is that there is a ton of flexibility with how you invoke commands, where the standard in/out/error streams are connected, etc.
The commands module is a reasonably high-level way to do this:
import commands
status, output = commands.getstatusoutput("cat /etc/services")
status is 0, output is the contents of /etc/services.
For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code. Since you are only interested in the output, you can write a utility wrapper like this.
from subprocess import PIPE, run
def out(command):
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
return result.stdout
my_output = out("echo hello world")
# Or
my_output = out(["echo", "hello world"])
I know this has already been answered, but I wanted to share a potentially better looking way to call Popen via the use of from x import x and functions:
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
print cmdline("cat /etc/services")
print cmdline('ls')
print cmdline('rpm -qa | grep "php"')
print cmdline('nslookup google.com')
I do it with os.system temp file:
import tempfile, os
def readcmd(cmd):
ftmp = tempfile.NamedTemporaryFile(suffix='.out', prefix='tmp', delete=False)
fpath = ftmp.name
if os.name=="nt":
fpath = fpath.replace("/","\\") # forwin
ftmp.close()
os.system(cmd + " > " + fpath)
data = ""
with open(fpath, 'r') as file:
data = file.read()
file.close()
os.remove(fpath)
return data
Python 2.6 and 3 specifically say to avoid using PIPE for stdout and stderr.
The correct way is
import subprocess
# must create a file object to store the output. Here we are getting
# the ssid we are connected to
outfile = open('/tmp/ssid', 'w');
status = subprocess.Popen(["iwgetid"], bufsize=0, stdout=outfile)
outfile.close()
# now operate on the file
from os import system, remove
from uuid import uuid4
def bash_(shell_command: str) -> tuple:
"""
:param shell_command: your shell command
:return: ( 1 | 0, stdout)
"""
logfile: str = '/tmp/%s' % uuid4().hex
err: int = system('%s &> %s' % (shell_command, logfile))
out: str = open(logfile, 'r').read()
remove(logfile)
return err, out
# Example:
print(bash_('cat /usr/bin/vi | wc -l'))
>>> (0, '3296\n')```
I want only the wlan device name at a linux system with python. I could get the device name with shell scripting:
echo /sys/class/net/*/wireless | awk -F'/' '{ print $5 }'
So i want to use this at python with subprocess.
import shlex
import subprocess
def main():
echo = shlex.split('echo /sys/class/net/*/wireless')
echo_proc = subprocess.Popen(echo, shell=True, stdout=subprocess.PIPE)
awk = shlex.split("awk -F'/' '{ print $5 }'")
awk_proc = subprocess.Popen(awk, stdin=echo_proc.stdout)
print(awk_proc.stdout)
But I get only None as output. If it is possible, I would prefer a solution with subprocess.run(). So I replaced Popen with run. But then I get the error message AttributeError: 'bytes' object has no attribute 'fileno'.
A type glob and a pathname expansion by shell will be a headache.
In my environment, the following snippet works:
import subprocess
subprocess.run('echo /sys/class/net/*/wireless', shell=True)
But the following returns an empty string:
import subprocess
subprocess.run(['echo', '/sys/class/net/*/wireless'], shell=True)
Then please try the following as a starting point:
import subprocess
subprocess.run('echo /sys/class/net/*/wireless | awk -F"/" "{ print \$5 }"', shell=True)
which will bring your desired output.
[Update]
If you want to assign a variable to the output above, please try:
import subprocess
proc = subprocess.run('echo /sys/class/net/*/wireless | awk -F"/" "{ print \$5 }"', shell=True, stdout = subprocess.PIPE)
wlan = proc.stdout.decode("utf8").rstrip("\n")
print(wlan)
BTW if you don't stick to the subprocess module, why don't you go with a native way as:
import glob
list = glob.glob('/sys/class/net/*/wireless')
for elm in list:
print(elm.split('/')[4])
Hope this helps.
This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
Below is the script for searching a input data in a given OS.system command ( copied to file out.txt) and prints the line of the given input data. Now I want to put the output i.e line in another OS.system command. For example symaccess -sid 567 show -name xxx -type init where xxx is the output of the previous OS.system command i.e line.
Note - I can use only python 2.6.6 and the scripts related to storage
import os
os.system('symaccess -sid 456 list -type init > out.txt')
server = raw_input("server name:")
with open('out.txt', 'rt') as in_f:
for line in in_f:
if server in line:
print line
I used another method as below
import os server = raw_input("server name:")
var = "symaccess -sid 239 list -type init | grep \"{0}\"".format(server)
wwn = os.system(var)
init = 'symaccess -sid 239 -type init show {0}'.format(wwn) print init os.system(init)
above is the script i used to add a output of one os.system to another os.syste,. i got the first os.system executed but for the second one i.e os.system(init) is not coming because os.system(var) should be assigned as a variable.could someone tell how to assign a variable to os.system(init)
import subprocess
cmd= 'ls'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
std_out = p.communicate()
Here is a possible way to access the output in Python.
Try using "subprocess.checkoutput()".
import subprocess
import os
output = subprocess.check_output(['symaccess -sid 456 list -type init'])
os.system(output)
im having an issue trying to get a simple grep command into python. I want to take the output of the following command in a file or a list.
grep -c 'some thing' /home/user/* | grep -v :0
This is what I have, but its not working at all...
thing = str(subprocess.Popen(['grep', '-c', 'some thing', '/home/user/*', '|', 'grep', '-v', ':0'], stdout=subprocess.PIPE)
Basically I need to search files in a directory and return a result if my string is missing from any of the files in the directory.
Working Code (Thanks!!):
thing = subprocess.Popen(('grep -c "some thing" /home/user/* | grep -v ":0"' ),shell=True, stdout=subprocess.PIPE)
The pipe | is a shell feature. You have to use Popen with shell=True to use it.
To emulate the shell pipeline in Python, see How do I use subprocess.Popen to connect multiple processes by pipes?:
#!/usr/bin/env python
import os
from glob import glob
from subprocess import Popen, PIPE
p1 = Popen(["grep", "-c", 'some thing'] + glob(os.path.expanduser('~/*')),
stdout=PIPE)
p2 = Popen(["grep", "-v", ":0"], stdin=p1.stdout)
p1.stdout.close()
p2.wait()
p1.wait()
To get output as a string, set stdout=PIPE and call output = p2.communicate()[0] instead of p2.wait().
To suppress error messages such as "grep: /home/user/dir: Is a directory", you could set stderr=DEVNULL.
You could implement the pipeline in pure Python:
import os
from glob import glob
for name in glob(os.path.expanduser('~/*')):
try:
count = sum(1 for line in open(name, 'rb') if b'some thing' in line)
except IOError:
pass # ignore
else:
if count: # don't print zero counts
print("%s:%d" % (name, count))