is there a "nice" way to iterate over the output of a shell command?
I'm looking for the python equivalent for something like:
ls | while read file; do
echo $file
done
Note that 'ls' is only an example for a shell command which will return it's result in multiple lines and of cause 'echo' is just: do something with it.
I known of these alternatives: Calling an external command in Python but I don't know which one to use or if there is a "nicer" solution to this. (In fact "nicer" is the main focus of this question.)
This is for replacing some bash scripts with python.
you can open a pipe ( see doc ):
import os
with os.popen('ls') as pipe:
for line in pipe:
print (line.strip())
as in the document this syntax is depreciated and is replaced with more complicated subprocess.Popen
from subprocess import Popen, PIPE
pipe = Popen('ls', shell=True, stdout=PIPE)
for line in pipe.stdout:
print(line.strip())
check_output will give you back a string you can parse. call will simply re-use your existing stdout/stderr/stdin and simply return the process exit code.
Related
I'm just learning Python but have about 16 years experience with PERL and PHP.
I'm trying to get the output of ngrep and write it to a log file using Python while also tailing the log file. I've seen some examples online but some seem old and outdated and others use shell=True which is discouraged.
In perl I just use something similar to the following
#!/usr/bin/perl
open(NGFH,"ngrep -iW byline $filter");
while ($line = <NGFH>) {
open(LOG,">> /path/to/file.log")
// highlighting, filtering, other sub routine calls
print LOG $line
}
I've gotten tail to work but ngrep doesn't. I'd like to be able to run this infinately and output the stream from ngrep to the log file after filtering. I couldn't get the output from ngrep to show in stdout so that's as far as I've gotten. I was expecting to be able to see the data file tail as the log file was updated and see the output from ngrep. For now i was just using bash to run the following.
echo "." >> /path/to/ngrep.log
Thanks!
Here's what I got so far...
Updated
This seems to work now. I wouldn't know how to improve on it though.
import subprocess
import select
import re
log = open('/path/to/ngrep.log','a+',0)
print log.name
n = subprocess.Popen(['ngrep', '-iW', 'byline'],\
stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p = select.poll()
p.register(n.stdout)
f = subprocess.Popen(['tail','-F','-n','0','/path/to/tailme.log'],\
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p2 = select.poll()
p2.register(f.stdout)
def srtrepl(match):
if match.group(0) == 'x.x.x.x':
# do something
if match.group(0) == 'x.x.y.y':
# do something else
return '\033[92m'+ match.group(0) + '\033[0m'
while True:
if p.poll(1):
line = n.stdout.readline()
s = re.compile(r'(8.8.(4.4|8.8)|192.168.[0-9]{1,3}.[0-9]{1,3})' )
print s.sub( srtrepl, line )
log.write(n.stdout.readline())
if p2.poll(1):
print f.stdout.readline().rstrip('\n')
To emulate your perl code in Python:
#!/usr/bin/env python3
from subprocess import Popen, PIPE
with Popen("ngrep -iW byline".split() + [filter_], stdout=PIPE) as process, \
open('/path/to/file.log', 'ab') as log_file:
for line in process.stdout: # read b'\n'-separated lines
# highlighting, filtering, other function calls
log_file.write(line)
It starts ngrep process passing filter_ variable and appends the output to the log file while allowing you to modify it in Python. See Python: read streaming input from subprocess.communicate() (there could be buffering issues: check whether ngrep supports --line-buffered option like grep and if you want to tail file.log then pass buffering=1 to open(), to enable line-buffering (only usable in the text-mode) or call log_file.flush() after log_file.write(line)).
You could emulate ngrep in pure Python too.
If you want to read output from several processes concurrently (ngrep, tail in your case) then you need to able to read pipes without blocking e.g., using threads, async.io.
I am trying to run the following bash script in Python and store the readlist output. The readlist that I want to be stored as a python list, is a list of all files in the current directory ending in *concat_001.fastq.
I know it may be easier to do this in python (i.e.
import os
readlist = [f for f in os.listdir(os.getcwd()) if f.endswith("concat_001.fastq")]
readlist = sorted(readlist)
However, this is problematic, as I need Python to sort the list in EXACTLY the same was as bash, and I was finding that bash and Python sort certain things in different orders (eg Python and bash deal with capitalised and uncapitalised things differently - but when I tried
readlist = np.asarray(sorted(flist, key=str.lower))
I still found that two files starting with ML_ and M_ were sorted in different order with bash and Python. Hence trying to run my exact bash script through Python, then to use the sorted list generated with bash in my subsequent Python code.
input_suffix="concat_001.fastq"
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g" `
readlist="$(echo $ender)"
I have tried
proc = subprocess.call(command1, shell=True, stdout=subprocess.PIPE)
proc = subprocess.call(command2, shell=True, stdout=subprocess.PIPE)
proc = subprocess.Popen(command3, shell=True, stdout=subprocess.PIPE)
But I just get: subprocess.Popen object at 0x7f31cfcd9190
Also - I don't understand the difference between subprocess.call and subprocess.Popen. I have tried both.
Thanks,
Ruth
So your question is a little confusing and does not exactly explain what you want. However, I'll try to give some suggestions to help you update it, or in my effort, answer it.
I will assume the following: your python script is passing to the command line 'input_suffix' and that you want your python program to receive the contents of 'readlist' when the external script finishes.
To make our lives simpler, and allow things to be more complicated, I would make the following bash script to contain your commands:
script.sh
#!/bin/bash
input_suffix=$1
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g"`
readlist="$(echo $ender)"
echo $readlist
You would execute this as script.sh "concat_001.fastq", where $1 takes in the first argument passed on the command line.
To use python to execute external scripts, as you quite rightly found, you can use subprocess (or as noted by another response, os.system - although subprocess is recommended).
The docs tell you that subprocess.call:
"Wait for command to complete, then return the returncode attribute."
and that
"For more advanced use cases when these do not meet your needs, use the underlying Popen interface."
Given you want to pipe the output from the bash script to your python script, let's use Popen as suggested by the docs. As I posted the other stackoverflow answer, it could look like the following:
import subprocess
from subprocess import Popen, PIPE
# Execute out script and pipe the output to stdout
process = subprocess.Popen(['script.sh', 'concat_001.fastq'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# Obtain the standard out, and standard error
stdout, stderr = process.communicate()
and then:
>>> print stdout
*concat_001.fastq
Currently, I am using the following command to do this
$ python scriptName.py <filePath
This command uses "<" to stdin the file to script.
and it works fine, I can use sys.stdin.read to get the file data.
But, what if I want to pass file data as a string,
I don't want to pass file path in operator "<".
Is there is anyway, where I can pass String as stdin to a python script?
Thanks,
Kamal
The way I read your question, you currently have some file abc.txt with content
Input to my program
And you execute it this way:
python scriptName.py <abc.txt
Now you no longer want to go by way of this file, and instead type the input as part of the command, while still reading from stdin. Working on the windows command line you may do it like this:
echo Input to my program | python scriptName.py
while on Linux/Mac you'd better quote it to avoid shell expansion:
echo "Input to my program" | python scriptName.py
This only works for single-line input on windows (AFAIK), while on linux (and probably Mac) you can use the -e switch to insert newlines:
echo -e "first line\nsecond line" | python scriptName.py
There is raw_input which you can use make the program prompt for input and you can send in a string. And yes, it is mentioned in the first few pages of the tutorial at http://www.python.org.
>>> x = raw_input()
Something # you type
>>> x
'Something'
And sending the input via < the shell redirection operation is the property of shell and not python.
I could be wrong, but the way that I read the OP's question, I think he may currently be calling an os command to run a shell script inside of his python script, and then using a < operator to pass a file's contents into this shell script, and he is just hard coding the < and filename.
What he really desires to do is a more dynamic approach where he can pass a string defined in Python to this shell script.
If this is the case, the method I would suggest is this:
import subprocess;
script_child = subprocess.Popen(['/path/to/script/myScript.sh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
stdout, stderr = clone_child.communicate("String to pass to the script.")
print "Stdout: ", stdout
print "Stderr: ", stderr
Alternatively, you can pass arguments to the script in the initial Popen like so:
script_child = subprocess.Popen(['/path/to/script/myScript.sh', '-v', 'value', '-fs'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
I'm calling rtmpdump via subprocess and trying to redirect its output to a file. The problem is that I simply can't redirect it.
I tried first setting up the sys.stdout to the opened file. This works for, say, ls, but not for rtmpdump. I also tried setting the sys.stderr just to make sure and it also didn't work.
I tried then using a ">> file" with the command line argument but again it doesn't seem to work.
Also for the record, for some reason, Eclipse prints rtmpdump's output even if I use subprocess.call instead of subprocess.check_output, and without having to call the print method. This is black magic!
Any suggestions?
Edit: Here's some sample code.
# /!\ note: need to use os.chdir first to get to the folder with rtmpdump!
command = './rtmpdump -r rtmp://oxy.videolectures.net/video/ -y 2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01 -a video -s http://media.videolectures.net/jw-player/player.swf -w ffa4f0c469cfbe1f449ec42462e8c3ba16600f5a4b311980bb626893ca81f388 -x 53910 -o test.flv'
split_command = shlex.split(command)
subprocess.call(split_command)
sys.stdout is the python's idea of the parent's output stream.
In any case you want to change the child's output stream.
subprocess.call and subprocess.Popen take named parameters for the output streams.
So open the file you want to output to and then pass that as the appropriate argument to subprocess.
f = open("outputFile","wb")
subprocess.call(argsArray,stdout=f)
Your talk of using >> suggest you are using shell=True, or think you are passing your arguments to the shell. In any case it is better to use the array form of subprocess, which avoid an unnecessary process, and any weirdness from the shell.
EDIT:
So I downloaded RTMPDump and tried it out, it would appear the messages are appearing on stderr.
So with the following program, nothing appears on the programs output, and the rtmpdump logs when into the stderr.txt file:
#!/usr/bin/env python
import os
import subprocess
RTMPDUMP="./rtmpdump"
assert os.path.isfile(RTMPDUMP)
command = [RTMPDUMP,'-r','rtmp://oxy.videolectures.net/video/',
'-y','2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01',
'-a','video','-s',
'http://media.videolectures.net/jw-player/player.swf',
'-w','ffa4f0c469cfbe1f449ec42462e8c3ba16600f5a4b311980bb626893ca81f388'
,'-x','53910','-o','test.flv']
stdout = open("stdout.txt","wb")
stderr = open("stderr.txt","wb")
subprocess.call(command,stdout=stdout,stderr=stderr)
See the link on getting the output from subprocess on SO
Getting the entire output from subprocess.Popen
https://stackoverflow.com/questions/tagged/subprocess
I guess the way would be to collect the output and write it to a file directly or provide file descriptors to which you output can be written.
Something like this:
f = open('dump.txt', 'wb')
p = subprocess.Popen(args, stdout=f, stderr=subprocess.STDOUT, shell=True)
In a script , I want to run a .exe with some command line parameters as "-a",and then
redirect the standard output of the program to a file?
How can I implement that?
You can redirect directly to a file using subprocess.
import subprocess
with open('output.txt', 'w') as output_f:
p = subprocess.Popen('Text/to/execute with-arg',
stdout=output_f,
stderr=output_f)
Easiest is os.system("the.exe -a >thefile.txt"), but there are many other ways, for example with the subprocess module in the standard library.
You can do something like this
e.g. to read output of ls -l (or any other command)
p = subprocess.Popen(["ls","-l"],stdout=subprocess.PIPE)
print p.stdout.read() # or put it in a file
you can do similar thing for stderr/stdin
but as Alex mentioned if you just want it in a file, just redirect the cmd output to a file
If you just want to run the executable and wait for the results, Anurag's solution is probably the best. I needed to respond to each line of output as it arrived, and found the following worked:
1) Create an object with a write(text) method. Redirect stdout to it (sys.stdout = obj). In your write method, deal with the output as it arrives.
2) Run a method in a seperate thread with something like the following code:
p = subprocess.Popen('Text/to/execute with-arg', stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=False)
while p.poll() is None:
print p.stdout.readline().strip()
Because you've redirected stdout, PIPE will send the output to your write method line by line. If you're not certain you're going to get line breaks, read(amount) works too, I believe.
3) Remember to redirect stdout back to the default: sys.stdout = __sys.stdout__
Although the title (.exe) sounds like it's a problem on Windows.
I had to share that the accepted answer (subprocess.Popen() with stdout/stderr arguments) didn't work for me on Mac OS X (10.8) with python 2.7.
I had to use subprocess.check_output() (python 2.7 and above) to make it work. Example:
import subprocess
cmd = 'ls -l'
out = subprocess.check_output(cmd, shell=True)
with open('my.log', 'w') as f:
f.writelines(out)
f.close()
Note that this solution writes all the accumulated output out when the program finishes.
If you want to monitor the log file during the run. You may want to try something else.
In my own case, I only cared about the end result.