I am trying to get the stdout of a python script to be shell-piped in as stdin to another python script like so:
find ~/test -name "*.txt" | python my_multitail.py | python line_parser.py
It should print an output but nothing comes out of it.
Please note that this works:
find ~/test -name "*.txt" | python my_multitail.py | cat
And this works too:
echo "bla" | python line_parser.py
my_multitail.py prints out the new content of the .txt files:
from multitail import multitail
import sys
filenames = sys.stdin.readlines()
# we get rid of the trailing '\n'
for index, filename in enumerate(filenames):
filenames[index] = filename.rstrip('\n')
for fn, line in multitail(filenames):
print '%s: %s' % (fn, line),
sys.stdout.flush()
When a new line is added to the .txt file ("hehe") then my_multitail.py prints:
/home/me/test2.txt: hehe
line_parser.py simply prints out what it gets on stdin:
import sys
for line in sys.stdin:
print "line=", line
There is something I must be missing. Please community help me :)
There's a hint if you run your line_parser.py interactively:
$ python line_parser.py
a
b
c
line= a
line= b
line= c
Note that I hit ctrl+D to provoke an EOF after entering the 'c'. You can see that it's slurping up all the input before it starts iterating over the lines. Since this is a pipeline and you're continuously sending output through to it, this doesn't happen and it never starts processing. You'll need to choose a different way of iterating over stdin, for example:
import sys
line = sys.stdin.readline()
while line:
print "line=", line
line = sys.stdin.readline()
Related
I am using a subprocess call in python where I have to print a file contents using cat. The file name is a variable that I generate in the python code itself. This is my code:
pid = str(os.getpid())
tmp_file_path = "/tmp/" + pid + "/data_to_synnet"
synnet_output = subprocess.check_output(["cat echo '%s'"%tmp_file_path], shell=True)
The above code throws an error saying cat: echo: No such file or directory.
However, when I use only subprocess.check_output(["echo '%s'"%tmp_file_path], shell=True), the variable name is printed correctly.
Also, I tried doing this (cat echo $tmp_file_name) in the command line and it works. Can someone please tell what is wrong?
The command you want is this:
"cat '%s'"%tmp_file_path
Just get rid of the "echo" word.
Alternatively,
synnet_output = subprocess.check_output(["cat", tmp_file_path], shell=False)
I am trying to accomplish an idea which i really don't know how to do it. Basically i am trying to capture the value through grep command as shown below
p = subprocess.Popen('tail -f /data/qantasflight/run/tomcat/logs/localhost_access_log.2016-02-29.txt | grep /qantas-ui/int/price?', stdout=subprocess.PIPE, shell = True)
stdout = p.communicate()[0]
Process the stdout value and then push the value as shown below
f = urllib.urlopen("http://162.16.1.90:9140/TxnService", params2)
param2 is the value where i will process the result given by the subprocess.Popen
In a nutshell i want the following :
-- Wait for the new value --> -- Process the value --> -- Push the value -->
This should be realtime and the python script will keep on getting the new value, process it and then push the value.
i try code somethink as you want, but im only printing new lines in log, you can process line and push
from __future__ import print_function
import subprocess
from time import sleep
f = open('test.log', 'r')
while True:
line = ''
while len(line) == 0 or line[-1] != '\n':
tail = f.readline()
if tail == '':
continue
line = tail
print(line, end='')
this print new line to console, just edit and use :) maybe i help you :)
My goal is to compare two data one is from text file and one is from directory and after comparing it this is will notify or display in the console what are the data that is not found for example:
ls: /var/patchbundle/rpms/:squid-2.6.STABLE21-7.el5_10.x86_64.rpm NOT FOUND!
ls: /var/patchbundle/rpms/:tzdata-2014j-1.el5.x86_64.rpm
ls: /var/patchbundle/rpms/:tzdata-java-2014j-1.el5.x86_64.rpm
ls: /var/patchbundle/rpms/:wireshark-1.0.15-7.el5_11.x86_64.rpm
ls: /var/patchbundle/rpms/:wireshark-gnome-1.0.15-7.el5_11.x86_64.rpm
ls: /var/patchbundle/rpms/:yum-updatesd-0.9-6.el5_10.noarch.rpm NOT FOUND
It must be like that. So Here's my python code.
import package, sys, os, subprocess
path = '/var/tools/tools/newrpms.txt'
newrpms = open(path, "r")
fds = newrpms.readline()
def checkrc(rc):
if(rc != 0):
sys.exit(rc)
cmd = package.Errata()
for i in newrpms:
rc = cmd.execute("ls /var/patchbundle/rpms/ | grep %newrpms ")
if ( != 0):
cmd.logprint ("%s not found !" % i)
checkrc(rc)
sys.exit(0)
newrpms.close
Please see the shell script. This script its executing file but because I want to use another language that's why Im trying python
retval=0
for i in $(cat /var/tools/tools/newrpms.txt)
do
ls /var/patchbundle/rpms/ | grep $i
if [ $? != 0 ]
then
echo "$i NOT FOUND!"
retval=255
fi
done
exit $retval
Please see my Python code. What is wrong because it is not executing like the shell executing it.
You don't say what the content of "newrpms.txt" is; you say the script is not executing how you want - but you don't say what it is doing; I don't know what package or package.Errata are, so I'm playing guess-the-problem; but lots of things are wrong.
if ( != 0): is a syntax error. If {empty space} is not equal to zero?
cmd.execute("ls /var/patchbundle/rpms/ | grep %newrpms ") is probably not doing what you want. You can't put a variable in a string in Python like that, and if you could newrpms is the file handle not the current line. That should probably be ...grep %s" % (i,)) ?
The control flow is doing:
Look in this folder, try to find files
Call checkrc()
Only quit with an error if the last file was not found
newrpms.close isn't doing anything, it would need to be newrpms.close() to call the close method.
You're writing shell-script-in-Python. How about:
import os, sys
retval=0
for line in open('/var/tools/tools/newrpms.txt'):
rpm_path = '/var/patchbundle/rpms/' + line.strip()
if not os.path.exists(rpm_path):
print rpm_path, "NOT FOUND"
retval = 255
else:
print rpm_path
sys.exit(retval)
Edited code slightly, and an explanation:
The code is almost a direct copy of the shell script into Python. It loops over every line in the text file, and calls line.strip() to get rid of the newline character at the end. It builds rpm_path which will be something like "/var/patchbundle/rpms/:tzdata-2014j-1.el5.x86_64.rpm".
Then it uses sys.path.exists() which tests if a file exists and returns True if it does, False if it does not, and uses that test to set the error value and print the results like the shell script prints them. This replaces the "ls ... | grep " part of your code for checking if a file exists.
I'm trying to write a simple python script where
it takes values from stdin
replaces a specific matched word
passes on the output with the NEW value back to stdout
I only have the part where it takes the values from stdin and looks for the matching words, I'm a bit stuck after that.
import re
import sys
for line in sys.stdin:
matchObj = re.search(r'<something>(.*)</something>',line)
if matchObj:
oldWord = matchObj.group(1)
print oldWord
Contents of foo
<something>REPLACEME</something>
<blah>UNTOUCH</blah>
Ideally if I run this command
cat foo | ./test.py
I would get something like this
<something>NEWWORD</something
<blah>UNTOUCH</blah>
Are you looking for re.sub?
import re
import sys
for line in sys.stdin:
sys.stdout.write(re.sub(r'(<something>)REPLACEME(</something>)',
r'\1NEWWORD\2',
line))
Running the above on your example data:
$ echo '<something>REPLACEME</something>\n<something>UNTOUCH</something>' | python2 test.py
<something>NEWWORD</something>
<blah>UNTOUCH</blah>
Note that parsing XML with regular expressions is probably a bad idea. The Python standard library comes with a number of XML modules.
Here's an example:
import sys
import xml.etree.ElementTree
tree = xml.etree.ElementTree.parse(sys.stdin)
root = tree.getroot()
for node in root.iter('something'):
if node.text == 'REPLACEME':
node.text == 'NEWWORD'
tree.write(sys.stdout)
The above would work just the same:
$ echo '<root><something>REPLACEME</something>\n<blah>UNTOUCH</blah></root>' | python2 test.py
<root><something>REPLACEME</something>
<blah>UNTOUCH</blah></root>
firs if you run cat foo | ./test.py you got test.py: command not found , you need to run this : cat foo |python ./test.py .
then the output of your code will be :
REPLACEME
but for the output that you want, you need to use re.sub():
import re
import sys
for line in sys.stdin:
matchObj = re.sub(r'<something>(.*)</something>','<something>NEWWORD</something>',line)
if matchObj:
print matchObj
output :
<something>NEWWORD</something>
<blah>UNTOUCH</blah>
Also as a pythonic way you can use The ElementTree XML API
I want to pipe the output of ps -ef to python line by line.
The script I am using is this (first.py) -
#! /usr/bin/python
import sys
for line in sys.argv:
print line
Unfortunately, the "line" is split into words separated by whitespace. So, for example, if I do
echo "days go by and still" | xargs first.py
the output I get is
./first.py
days
go
by
and
still
How to write the script such that the output is
./first.py
days go by and still
?
Instead of using command line arguments I suggest reading from standard input (stdin). Python has a simple idiom for iterating over lines at stdin:
import sys
for line in sys.stdin:
sys.stdout.write(line)
My usage example (with above's code saved to iterate-stdin.py):
$ echo -e "first line\nsecond line" | python iterate-stdin.py
first line
second line
With your example:
$ echo "days go by and still" | python iterate-stdin.py
days go by and still
What you want is popen, which makes it possible to directly read the output of a command like you would read a file:
import os
with os.popen('ps -ef') as pse:
for line in pse:
print line
# presumably parse line now
Note that, if you want more complex parsing, you'll have to dig into the documentation of subprocess.Popen.
Another approach is to use the input() function (the code is for Python 3).
while True:
try:
line = input()
print('The line is:"%s"' % line)
except EOFError:
# no more information
break
The difference between the answer and the answer got by Dr. Jan-Philip Gehrcke is that now each of the lines is without a newline (\n) at the end.
I know this is really out-of-date, but you could try
#! /usr/bin/python
import sys
print(sys.argv, len(sys.argv))
if len(sys.argv) == 1:
message = input()
else:
message = sys.argv[1:len(sys.argv)]
print('Message:', message)
and I tested it thus:
$ ./test.py
['./test.py'] 1
this is a test
Message: this is a test
$ ./test.py this is a test
['./test.py', 'this', 'is', 'a', 'test'] 5
Message: ['this', 'is', 'a', 'test']
$ ./test.py "this is a test"
['./test.py', 'this is a test'] 2
Message: ['this is a test']
$ ./test.py 'this is a test'
['./test.py', 'this is a test'] 2
Message: ['this is a test']
$ echo "This is a test" | ./test.py
['./test.py'] 1
Message: This is a test
Or, if you wanted the message to be one string, each and every time, then
message = ' '.join(sys.argv[1:len(sys.argv)])
would do the trick on line 8