I want to pipe the output of ps -ef to python line by line.
The script I am using is this (first.py) -
#! /usr/bin/python
import sys
for line in sys.argv:
print line
Unfortunately, the "line" is split into words separated by whitespace. So, for example, if I do
echo "days go by and still" | xargs first.py
the output I get is
./first.py
days
go
by
and
still
How to write the script such that the output is
./first.py
days go by and still
?
Instead of using command line arguments I suggest reading from standard input (stdin). Python has a simple idiom for iterating over lines at stdin:
import sys
for line in sys.stdin:
sys.stdout.write(line)
My usage example (with above's code saved to iterate-stdin.py):
$ echo -e "first line\nsecond line" | python iterate-stdin.py
first line
second line
With your example:
$ echo "days go by and still" | python iterate-stdin.py
days go by and still
What you want is popen, which makes it possible to directly read the output of a command like you would read a file:
import os
with os.popen('ps -ef') as pse:
for line in pse:
print line
# presumably parse line now
Note that, if you want more complex parsing, you'll have to dig into the documentation of subprocess.Popen.
Another approach is to use the input() function (the code is for Python 3).
while True:
try:
line = input()
print('The line is:"%s"' % line)
except EOFError:
# no more information
break
The difference between the answer and the answer got by Dr. Jan-Philip Gehrcke is that now each of the lines is without a newline (\n) at the end.
I know this is really out-of-date, but you could try
#! /usr/bin/python
import sys
print(sys.argv, len(sys.argv))
if len(sys.argv) == 1:
message = input()
else:
message = sys.argv[1:len(sys.argv)]
print('Message:', message)
and I tested it thus:
$ ./test.py
['./test.py'] 1
this is a test
Message: this is a test
$ ./test.py this is a test
['./test.py', 'this', 'is', 'a', 'test'] 5
Message: ['this', 'is', 'a', 'test']
$ ./test.py "this is a test"
['./test.py', 'this is a test'] 2
Message: ['this is a test']
$ ./test.py 'this is a test'
['./test.py', 'this is a test'] 2
Message: ['this is a test']
$ echo "This is a test" | ./test.py
['./test.py'] 1
Message: This is a test
Or, if you wanted the message to be one string, each and every time, then
message = ' '.join(sys.argv[1:len(sys.argv)])
would do the trick on line 8
Related
I want to pipe the output of ps -ef to python line by line.
The script I am using is this (first.py) -
#! /usr/bin/python
import sys
for line in sys.argv:
print line
Unfortunately, the "line" is split into words separated by whitespace. So, for example, if I do
echo "days go by and still" | xargs first.py
the output I get is
./first.py
days
go
by
and
still
How to write the script such that the output is
./first.py
days go by and still
?
Instead of using command line arguments I suggest reading from standard input (stdin). Python has a simple idiom for iterating over lines at stdin:
import sys
for line in sys.stdin:
sys.stdout.write(line)
My usage example (with above's code saved to iterate-stdin.py):
$ echo -e "first line\nsecond line" | python iterate-stdin.py
first line
second line
With your example:
$ echo "days go by and still" | python iterate-stdin.py
days go by and still
What you want is popen, which makes it possible to directly read the output of a command like you would read a file:
import os
with os.popen('ps -ef') as pse:
for line in pse:
print line
# presumably parse line now
Note that, if you want more complex parsing, you'll have to dig into the documentation of subprocess.Popen.
Another approach is to use the input() function (the code is for Python 3).
while True:
try:
line = input()
print('The line is:"%s"' % line)
except EOFError:
# no more information
break
The difference between the answer and the answer got by Dr. Jan-Philip Gehrcke is that now each of the lines is without a newline (\n) at the end.
I know this is really out-of-date, but you could try
#! /usr/bin/python
import sys
print(sys.argv, len(sys.argv))
if len(sys.argv) == 1:
message = input()
else:
message = sys.argv[1:len(sys.argv)]
print('Message:', message)
and I tested it thus:
$ ./test.py
['./test.py'] 1
this is a test
Message: this is a test
$ ./test.py this is a test
['./test.py', 'this', 'is', 'a', 'test'] 5
Message: ['this', 'is', 'a', 'test']
$ ./test.py "this is a test"
['./test.py', 'this is a test'] 2
Message: ['this is a test']
$ ./test.py 'this is a test'
['./test.py', 'this is a test'] 2
Message: ['this is a test']
$ echo "This is a test" | ./test.py
['./test.py'] 1
Message: This is a test
Or, if you wanted the message to be one string, each and every time, then
message = ' '.join(sys.argv[1:len(sys.argv)])
would do the trick on line 8
I have a requirement to fetch the count the occurrence of '|' in each line of a file then match the count with given inputcount, needs to throw exception when the count is wrong.
Say if the inputcount=3 and the file has following content
s01|test|aaa|hh
S02|test|bbb
so3|test|ccc|oo
then exception should get thrown on executing the line 2 and it should exit the file.
Tried below Awk command to fetch the count for each lines, but I was not sure how to compare and throw the exception, when it not matches
awk ' {print (split($0,a,"\|")-1) }' test.dat
Can anyone please help me with it?
You may use this awk:
awk -v inputcount=3 -F '\\|' 'NF && NF != inputcount+1 {exit 1}' file &&
echo "good" || echo "bad"
Details:
-F '\\|' sets | as input field separator
NF != inputcount+1 will return true if any line doesn't have inputcount pipe delimiters.
$ inputcount=3
$ awk -v c="$inputcount" 'gsub(/\|/,"&") != c{exit 1}' file
$ echo $?
1
As you also tagged the post with python I will write a python answer that could be a simple script.
The core is:
with open(filename) as f:
for n, line in enumerate(f):
if line.count("|") != 3:
print(f"Not valid file at line {n + 1}")
Than you can add some boilerplate:
import fileinput
import sys
with fileinput.input() as f:
for n, line in enumerate(f):
if line.count("|") != 3:
print(f"Not valid file at line {n + 1}")
sys.exit(1)
And with fileinput you can accept almost any sort of input: see Piping to a python script from cmd shell
Maybe try
awk -F '[|]' -v cols="$inputcount" 'NF != cols+1 {
print FILENAME ":" FNR ":" $0 >"/dev/stderr"; exit 1 }' test.dat
The -F argument says to split on this delimiter; the number of resulting fields NF will be one more than there are delimiters, so we scream and die when that number is wrong.
I would like to simulate a coding session (for video recording session : I am not a touch typist :-)
For example, I have a shell script like this (test.sh)
hello="Hello"
world="world"
echo $hello", "$world
And I have a python script like this (Simulate_KeyPresses.py) :
import sys
import time
import subprocess
def send_letter(letter):
# V1 : simple print
sys.stdout.write(letter)
sys.stdout.flush()
# V2: Test with expect (apt-get install expect)
# cmd = """echo 'send "{}"' | expect""".format(c)
# subprocess.run(cmd, shell=True)
def simulate_keypresses(content):
lines = content.split("\n")
for line in lines:
for c in line:
send_letter(c)
time.sleep(0.03)
send_letter("\n")
time.sleep(0.5)
if __name__ == "__main__":
filename = sys.argv[1]
with open(filename, "r") as f:
content = f.read()
simulate_keypresses(content)
Which I can invoke like this :
python Simulate_KeyPresses.py test.sh
And it works beautifully.
However when I pipe it to bash, like this:
python Simulate_KeyPresses.py test.sh | /bin/bash
I get
Hello, world
i.e I only get stdout and the key presses are not shown.
What I would like to see:
hello="Hello"
world="world"
echo $hello", "$world
Hello, world
I found a related answer (Simulate interactive python session), but it only handle python coding sessions.
I tried to use Expect, but it does not work as intended (does not show stdin also).
Help would be appreciated!
You can use the program tee as:
python Simulate_KeyPresses.py test.sh | tee /dev/tty | /bin/bash
How about adding this to your script:
subprocess.call("./{}".format(filename), shell=True)
The result will be
hello="Hello"
world="world"
echo $hello", "$world
Hello, world
I am trying to get the stdout of a python script to be shell-piped in as stdin to another python script like so:
find ~/test -name "*.txt" | python my_multitail.py | python line_parser.py
It should print an output but nothing comes out of it.
Please note that this works:
find ~/test -name "*.txt" | python my_multitail.py | cat
And this works too:
echo "bla" | python line_parser.py
my_multitail.py prints out the new content of the .txt files:
from multitail import multitail
import sys
filenames = sys.stdin.readlines()
# we get rid of the trailing '\n'
for index, filename in enumerate(filenames):
filenames[index] = filename.rstrip('\n')
for fn, line in multitail(filenames):
print '%s: %s' % (fn, line),
sys.stdout.flush()
When a new line is added to the .txt file ("hehe") then my_multitail.py prints:
/home/me/test2.txt: hehe
line_parser.py simply prints out what it gets on stdin:
import sys
for line in sys.stdin:
print "line=", line
There is something I must be missing. Please community help me :)
There's a hint if you run your line_parser.py interactively:
$ python line_parser.py
a
b
c
line= a
line= b
line= c
Note that I hit ctrl+D to provoke an EOF after entering the 'c'. You can see that it's slurping up all the input before it starts iterating over the lines. Since this is a pipeline and you're continuously sending output through to it, this doesn't happen and it never starts processing. You'll need to choose a different way of iterating over stdin, for example:
import sys
line = sys.stdin.readline()
while line:
print "line=", line
line = sys.stdin.readline()
I'm trying to write a simple python script where
it takes values from stdin
replaces a specific matched word
passes on the output with the NEW value back to stdout
I only have the part where it takes the values from stdin and looks for the matching words, I'm a bit stuck after that.
import re
import sys
for line in sys.stdin:
matchObj = re.search(r'<something>(.*)</something>',line)
if matchObj:
oldWord = matchObj.group(1)
print oldWord
Contents of foo
<something>REPLACEME</something>
<blah>UNTOUCH</blah>
Ideally if I run this command
cat foo | ./test.py
I would get something like this
<something>NEWWORD</something
<blah>UNTOUCH</blah>
Are you looking for re.sub?
import re
import sys
for line in sys.stdin:
sys.stdout.write(re.sub(r'(<something>)REPLACEME(</something>)',
r'\1NEWWORD\2',
line))
Running the above on your example data:
$ echo '<something>REPLACEME</something>\n<something>UNTOUCH</something>' | python2 test.py
<something>NEWWORD</something>
<blah>UNTOUCH</blah>
Note that parsing XML with regular expressions is probably a bad idea. The Python standard library comes with a number of XML modules.
Here's an example:
import sys
import xml.etree.ElementTree
tree = xml.etree.ElementTree.parse(sys.stdin)
root = tree.getroot()
for node in root.iter('something'):
if node.text == 'REPLACEME':
node.text == 'NEWWORD'
tree.write(sys.stdout)
The above would work just the same:
$ echo '<root><something>REPLACEME</something>\n<blah>UNTOUCH</blah></root>' | python2 test.py
<root><something>REPLACEME</something>
<blah>UNTOUCH</blah></root>
firs if you run cat foo | ./test.py you got test.py: command not found , you need to run this : cat foo |python ./test.py .
then the output of your code will be :
REPLACEME
but for the output that you want, you need to use re.sub():
import re
import sys
for line in sys.stdin:
matchObj = re.sub(r'<something>(.*)</something>','<something>NEWWORD</something>',line)
if matchObj:
print matchObj
output :
<something>NEWWORD</something>
<blah>UNTOUCH</blah>
Also as a pythonic way you can use The ElementTree XML API