I have the following shell script that I would like to write in Python (of course grep . is actually a much more complex command):
#!/bin/bash
(cat somefile 2>/dev/null || (echo 'somefile not found'; cat logfile)) \
| grep .
I tried this (which lacks an equivalent to cat logfile anyway):
#!/usr/bin/env python
import StringIO
import subprocess
try:
myfile = open('somefile')
except:
myfile = StringIO.StringIO('somefile not found')
subprocess.call(['grep', '.'], stdin = myfile)
But I get the error AttributeError: StringIO instance has no attribute 'fileno'.
I know I should use subprocess.communicate() instead of StringIO to send strings to the grep process, but I don't know how to mix both strings and files.
p = subprocess.Popen(['grep', '...'], stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
output, output_err = p.communicate(myfile.read())
Don't use bare except, it may catch too much. In Python 3:
#!/usr/bin/env python3
from subprocess import check_output
try:
file = open('somefile', 'rb', 0)
except FileNotFoundError:
output = check_output(cmd, input=b'somefile not found')
else:
with file:
output = check_output(cmd, stdin=file)
It works for large files (the file is redirected at the file descriptor level -- no need to load it into the memory).
If you have a file-like object (without a real .fileno()); you could write to the pipe directly using .write() method:
#!/usr/bin/env python3
import io
from shutil import copyfileobj
from subprocess import Popen, PIPE
from threading import Thread
try:
file = open('somefile', 'rb', 0)
except FileNotFoundError:
file = io.BytesIO(b'somefile not found')
def write_input(source, sink):
with source, sink:
copyfileobj(source, sink)
cmd = ['grep', 'o']
with Popen(cmd, stdin=PIPE, stdout=PIPE) as process:
Thread(target=write_input, args=(file, process.stdin), daemon=True).start()
output = process.stdout.read()
The following answer uses shutil as well --which is quite efficient--,
but avoids a running a separate thread, which in turn never ends and goes zombie when the stdin ends (as with the answer from #jfs)
import os
import subprocess
import io
from shutil import copyfileobj
file_exist = os.path.isfile(file)
with open(file) if file_exists else io.StringIO("Some text here ...\n") as string_io:
with subprocess.Popen("cat", stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True) as process:
copyfileobj(string_io, process.stdin)
# the subsequent code is not executed until copyfileobj ends,
# ... but the subprocess is effectively using the input.
process.stdin.close() # close or otherwise won't end
# Do some online processing to process.stdout, for example...
for line in process.stdout:
print(line) # do something
Alternatively to close and parsing, if the output is known to fit in memory:
...
stdout_text , stderr_text = process.communicate()
Related
I'm trying to read the content of a file with python 3.8.5 but the output is empty, I don't understand what I'm doing wrong.
Here is the code:
import subprocess
import os
filename = "ls.out"
ls_command = "ls -la"
file = open(filename, "w")
subprocess.Popen(ls_command, stdout=file, shell=True)
file.close()
# So far, all is ok. The file "ls.out" is correctly created and filled with the output of "ls -la" command"
file = open(filename, "r")
for line in file:
print(line)
file.close()
The output of this script is empty, it doesn't print anything. I'm not able to see the content of ls.out.
What is not correct here ?
Popen creates a new process and launches it but returns immediately. So the end result is that you've forked your code and have both processes running at once. Your python code in executing faster than the start and finish of ls. Thus, you need to wait for the process to finish by adding a call to wait():
import subprocess
import os
filename = "ls.out"
ls_command = "ls -la"
file = open(filename, "w")
proc = subprocess.Popen(ls_command, stdout=file, shell=True)
proc.wait()
file.close()
file = open(filename, "r")
for line in file:
print(line)
file.close()
Popen merely starts the subprocess. Chances are the file is not yet populated when you open it.
If you want to wait for the Popen object to finish, you have to call its wait method, etc; but a much better and simpler solution is to use subprocess.check_call() or one of the other higher-level wrappers.
If the command prints to standard output, why don't you read it drectly?
import subprocess
import shlex
result = subprocess.run(
shlex.split(ls_command), # avoid shell=True
check=True, text=True, capture_output=True)
line = result.stdout
Is there any possible way to communicate with the cmd and at the same time save all its output to a file?
I mean that after every command the output will be saved, not at the end of the sub-process.
I want it to be something like this:
import subprocess
process = subprocess.Popen('C:\\Windows\\system32\\cmd.exe', stdout=subprocess.PIPE,
stdin=subprocess.PIPE)
while True:
with open("log.txt", "a+") as myfile:
myfile.write(process.stdout.readlines())
process.stdin(raw_input())
You have two ways of doing this, either by creating an iterator from the read or readline functions and do:
import subprocess
import sys
with open('test.log', 'w') as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), ''):
sys.stdout.write(c)
f.write(c)
or
import subprocess
import sys
with open('test.log', 'w') as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
for line in iter(process.stdout.readline, ''):
sys.stdout.write(line)
f.write(line)
I am trying to display a output from system . But, my script produces the result only when I run it two times. Below is the script. Using subprocess.Popen at both the places does not produce any out put and same with subprocess.call.
#!/usr/bin/env python
import subprocess
import re
contr = 0
spofchk='su - dasd -c "java -jar /fisc/dasd/bin/srmclient.jar -spof_chk"'
res22 = subprocess.call("touch /tmp/logfile",shell=True,stdout=subprocess.PIPE)
fp = open("/tmp/logfile","r+")
res6 =subprocess.Popen(spofchk,shell=True,stdout=fp)
fil_list=[]
for line in fp:
line = line.strip()
fil_list.append(line)
fp.close()
for i in fil_list[2:]:
if contr % 2 == 0:
if 'no SPOF' in i:
flag=0
#print(flag)
#print(i)
else:
flag = 1
else:
continue
#Incrementing the counter by 2 so that we will only read line with spof and no SPOF
contr+=2
The child process has its own file descriptor and therefore you may close the file in the parent as soon as the child process is started.
To read the whole child process' output that is redirected to a file, wait until it exits:
import subprocess
with open('logfile', 'wb', 0) as file:
subprocess.check_call(command, stdout=file)
with open('logfile') as file:
# read file here...
If you want to consume the output while the child process is running, use PIPE:
#!/usr/bin/env python3
from subprocess import Popen, PIPE
with Popen(command, stdout=PIPE) as process, open('logfile', 'wb') as file:
for line in process.stdout: # read b'\n'-separated lines
# handle line...
# copy it to file
file.write(line)
Here's a version for older Python versions and links to fix other possible issues.
Since subprocess open a new shell , so in first time it is not possible to create the file and the file and write the output of another subprocess at the same time
.. So only solution for this is to use os. System ..
I'm looking for a better way to do this, if possible:
import subprocess
f = open('temp.file', 'w+')
f.write('hello world')
f.close()
out = subprocess.check_output(['cat', 'temp.file'])
print out
subprocess.check_output(['rm', 'temp.file'])
In this example I'm creating a file and passing it as input to cat (in reality it's not cat I'm running but some other program that parses an input pcap file).
What I'm wondering is, is there a way in Python I can create a 'file-like object' with some content, and pipe this file-like object as input to a command-line program. If it is possible, I reckon it would be more efficient than writing a file to the disk and then deleting that file.
check_output takes a stdin input argument to specify a file-like object to connect to the process's standard input.
with open('temp.file') as input:
out = subprocess.check_output(['cat'], stdin=input)
Also, there's no need to shell out to run rm; you can remove the file directly from Python:
os.remove('temp.file')
You can write to a TemporaryFile
import subprocess
from tempfile import TemporaryFile
f = TemporaryFile("w")
f.write("foo")
f.seek(0)
out = subprocess.check_output(['cat'],stdin=f)
print(out)
b'foo'
If you just want to write to a file like object and get the content:
from io import StringIO
f = StringIO()
f.write("foo")
print(f.getvalue())
If the program is configured to read from stdin, you can use Popen.communicate:
>>> from subprocess import Popen, PIPE
>>> p = Popen('cat', stdout=PIPE, stdin=PIPE, stderr=PIPE)
>>> out, err = p.communicate(input=b"Hello world!")
>>> out
'Hello world!'
If the command accepts only filenames, if it doesn't read input from its stdin i.e., if you can't use stdin=PIPE + .communicate() or stdin=real_file then you could try /dev/fd/# filenames:
#!/usr/bin/env python3
import os
import subprocess
import threading
def pump_input(pipe):
with pipe:
for i in range(3):
print(i, file=pipe)
r, w = os.pipe()
try:
threading.Thread(target=pump_input, args=[open(w, 'w')]).start()
out = subprocess.check_output(['cat', '/dev/fd/'+str(r)], pass_fds=[r])
finally:
os.close(r)
print('got:', out)
No content touches the disk. The input is passed to the subprocess via the pipe directly.
If you have a file-like object that is not a real file (otherwise, just pass its name as the command-line argument) then pump_input() could look like:
import shutil
def pump_input(pipe):
with pipe:
shutil.copyfileobj(file_like_object, pipe)
I use external library, like this:
from some_lib import runThatProgram
infile = '/tmp/test'
outfile = '/tmp/testout'
runThatProgram(infile, outfile)
while runThatProgram is:
def runThatProgram(infile, outfile):
os.system("%s %s > %s" % ('thatProgram', infile, outfile))
The problem is that 'thatProgram' returns lots of stuff on STDERR, I want to redirect it to a file, but I cannot edit runThatProgram code because it is in third party lib!
To illustrate what Rosh Oxymoron said, you can hack the code like this :
from some_lib import runThatProgram
infile = '/tmp/test'
outfile = '/tmp/testout 2>&1'
runThatProgram(infile, outfile)
with this, it will call
thatProgram /tmp/test > /tmp/testout 2>&1
that will redirected stderr (2) to stdout (1), and everything will be logged in your outfile.
To elaborate on using subprocess, you can open it, give it a pipe and then work from there so
import subprocess
program = "runthatprogram.py".split()
process = subprocess.Popen(program, stdout = subprocess.PIPE, stderr = open('stderr','w')) #stderr to fileobj
process.communicate()[0] #display stdout