Develop Reference

How to open and print the contents of a file line by line using subprocess?

I am trying to write a python script which SSHes into a specific address and dumps a text file. I am currently having some issues. Right now, I am doing this:
temp = "cat file.txt"
need = subprocess.Popen("ssh {host} {cmd}".format(host='155.0.1.1', cmd=temp),shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(need)
This is the naive approach where I am basically opening the file, saving its output to a variable and printing it. However, this really messes up the format when I print "need". Is there any way to simply use subprocess and read the file line by line? I have to be SSHed into the address in order to dump the file otherwise the file will not be detected, that is why I am not simply doing
f = open(temp, "r")
file_contents = f.read()
print (file_contents)
f.close()
Any help would be appreciated :)

You don't need to use the subprocess module to print the entire file line by line. You can use pure python.
f = open(temp, "r")
file_contents = f.read()
f.close()
# turn the file contents into a list
file_lines = file_contents.split("\n")
# print all the items in the list
for file_line in file_lines:
print(file_line)

python infinite loop getting slowdown while getting data from other text file

I am new to python,actually I tried a simple program. I am reading a text file which is updating data every time from logs so I need to copy or append that updated data only from that text file to another text file and do some process.For this I am done with below code.
with open("Log.txt") as f:
data = f.readlines()
i=len(data)#find length of file
def check(i):
with open("Log.txt") as f2:
count=f2.readlines()
x=len(count)#finds length of file after update
print(x)
j=i
i=x
while(j<x):
with open("Log.txt") as f1:
count=f1.readlines()
msg=count[j]#get updated text
j=j+1
with open('Output1.txt', 'a') as f3:
f3.write(str(msg))#append updated text to file
process()#calling a function which do some process on text file
while True:
check(i)
By using above code I am getting updated data but the problem is its getting slow down while doing infinite loops.Means Initially If I have a data in Log.text file upto 12:00pm it will append all data,After looping for 5 min the data in log.text file will increase but after same 5 min time I will get only 3 min data into output file. it was slow there is so much delay in getting data from text file.why??how can I get same updated text into output file instantly.

Try following code:
def read_and_copy():
with open("/var/log/messages") as input:
with open('/tmp/output', 'a+') as output:
while True:
# read line by line
data = input.readline()
# check if line is not empty
# also if needed - provide necessary checks
if data:
# write to another file
output.write(data)
# Flush the write buffers of the stream if applicable.
# This does nothing for read-only and non-blocking streams.
output.flush()
read_and_copy()
Keep in mind, that each call of read_and_copy will read entire file again and overwrite output file.

I think what you should use is follow, which follows a file. Whenever new lines are added to Log.txt, the below code updates it to Output1.txt.
Try this
import os, time
logfile = "Log.txt"
thefile = open(logfile)
def follow(thefile):
thefile.seek(0,2) # Go to the end of the file
while True:
line = thefile.readline()
if not line:
time.sleep(1) # Sleep briefly
else:
with open('Output1.txt', 'a') as f3:
f3.write(line)
loglines = follow(thefile)

How to get data from txt file in python log analysis?

I am beginner to python, I am trying to do log analysis, but I do not know how to get the txt file.
This is the code for outputting date, but these dates must be taken from the txt file :
import sys
import re
file = open ('desktop/trail.txt')
for line_string in iter(sys.stdin.readline,''):
line = line_string.rstrip()
date = re.search(r'date=[0-9]+\-[0-9]+\-[0-9]+', line)
date = date.group()
print date

You can use with statement to open a file safely and read each line with a readlines method. readlines returns a list of string.
Below code should work in your case:
import sys
import re
with open('desktop/trail.txt', 'r') as f:
for line in f.readlines():
line = line_string.rstrip()
date = re.search(r'date=[0-9]+\-[0-9]+\-[0-9]+', line)
date = date.group()
print date

you can do something like
for line in file.readlines():
don't forget about file closing! You can do it with file.close()

Use process substitution as input file to Python twice

Consider the following python script
#test.py
import sys
inputfile=sys.argv[1]
with open(inputfile,'r') as f:
for line in f.readlines():
print line
with open(inputfile,'r') as f:
for line in f.readlines():
print line
Now I want to run test.py on a substituted process, e.g.,
python test.py <( cat file | head -10)
It seems the second f.readlines returns empty. Why is that and is there a way to do it without having to specify two input files?

Why is that.
Process substitution works by creating a named pipe. So all the data consumed at the first open/read loop.
Is there a way to do it without having to specify two input files.
How about buffering the data before using it.
Here is a sample code
import sys
import StringIO
inputfile=sys.argv[1]
buffer = StringIO.StringIO()
# buffering
with open(inputfile, 'r') as f:
buffer.write(f.read())
# use it
buffer.seek(0)
for line in buffer:
print line
# use it again
buffer.seek(0)
for line in buffer:
print line

readlines() will read all available lines from the input at once. This is why the second call returns nothing because there is nothing left to read. You can assign the result of readlines() to a local variable and use it as many times as you want:
import sys
inputfile=sys.argv[1]
with open(inputfile,'r') as f:
lines = f.readlines()
for line in lines:
print line
#use it again
for line in lines:
print line

Reading specific lines from a file and writing to another file in python

I have a file containing 1000 lines. First 15 lines will be header information.
I am trying to do the below :-
1) Read the file
2)Get the number of lines with header information. It will return 15
3)write lines 1-15 to a text file.
I am able to do 1 and 2 correctly, but not the 3rd step. Any inputs please?
Below is my code
#!/usr/bin/python
import numpy;
import os;
import math;
import cPickle; import time
from numpy import arange, sign
import copy
import re
import sys
import string
import mmap
head_lines = 0;
count=1
fide = open("text1.txt","r");
while (count==1): #We skip header
head_lines = head_lines+1;
line = fide.readline();
if 'END OF HEADER' in line:
print 'End of the Header reached'
break
print "hello world"
print head_lines
nlines = head_lines;
key=1;
while (key < nlines):
file1 = open("Header.txt","w")
lines = fide.readline()
file1.write(lines)
key = key+1;
print "done"

with open("input.txt") as f1:
with open("output.txt","w") as f2:
for _ in range(15):
f2.write(f1.readline())
is that what you are asking for?
(in python2.7 I think you can do with open('f1') as f1,open('f2','w') as f2: ...)

There are two problems in your code:
What ever you are writing in the file header.txt you'll be overwriting it every time you loop on the second while because you are re-opening the file which will put the file pointer to its origin i.e. the start of the file.
The second issue is similar but on the other file fide. You open it putting the file pointer to the start of the file and read one line until the end of the headers. And in the second while loop you keep reading lines from the same file pointer fide so you're reading the next nlines.
You could store the header's lines in a list and then write those strings in the output file.