Develop Reference

Parsing Logs with Regular Expressions Python

Coding and Python lightweight :)
I've gotta iterate through some logfiles and pick out the ones that say ERROR. Boom done got that. What I've gotta do is figure out how to grab the following 10 lines containing the details of the error. Its gotta be some combo of an if statement and a for/while loop I presume. Any help would be appreciated.
import os
import re
# Regex used to match
line_regex = re.compile(r"ERROR")
# Output file, where the matched loglines will be copied to
output_filename = os.path.normpath("NodeOut.log")
# Overwrites the file, ensure we're starting out with a blank file
#TODO Append this later
with open(output_filename, "w") as out_file:
out_file.write("")
# Open output file in 'append' mode
with open(output_filename, "a") as out_file:
# Open input file in 'read' mode
with open("MXNode1.stdout", "r") as in_file:
# Loop over each log line
for line in in_file:
# If log line matches our regex, print remove later, and write > file
if (line_regex.search(line)):
# for i in range():
print(line)
out_file.write(line)

There is no need for regex to do this, you can just use the in operator ("ERROR" in line).
Also, to clear the content of the file without opening it in w mode, you can simply place the cursor at the beginning of the file and truncate.
import os
output_filename = os.path.normpath("NodeOut.log")
with open(output_filename, 'a') as out_file:
out_file.seek(0, 0)
out_file.truncate(0)
with open("MXNode1.stdout", 'r') as in_file:
line = in_file.readline()
while line:
if "ERROR" in line:
out_file.write(line)
for i in range(10):
out_file.write(in_file.readline())
line = in_file.readline()
We use a while loop to read lines one by one using in_file.readline(). The advantage is that you can easily read the next line using a for loop.
See the doc:
f.readline() reads a single line from the file; a newline character (\n) is left at the end of the string, and is only omitted on the last line of the file if the file doesn’t end in a newline. This makes the return value unambiguous; if f.readline() returns an empty string, the end of the file has been reached, while a blank line is represented by '\n', a string containing only a single newline.

Assuming you would only want to always grab the next 10 lines, then you could do something similar to:
with open("MXNode1.stdout", "r") as in_file:
# Loop over each log line
lineCount = 11
for line in in_file:
# If log line matches our regex, print remove later, and write > file
if (line_regex.search(line)):
# for i in range():
print(line)
lineCount = 0
if (lineCount < 11):
lineCount += 1
out_file.write(line)
The second if statement will help you always grab the line. The magic number of 11 is so that you grab the next 10 lines after the initial line that the ERROR was found on.

How do I split each line into two strings and print without the comma?

I'm trying to have output to be without commas, and separate each line into two strings and print them.
My code so far yields:
173,70
134,63
122,61
140,68
201,75
222,78
183,71
144,69
But i'd like it to print it out without the comma and the values on each line separated as strings.
if __name__ == '__main__':
# Complete main section of code
file_name = "data.txt"
# Open the file for reading here
my_file = open('data.txt')
lines = my_file.read()
with open('data.txt') as f:
for line in f:
lines.split()
lines.replace(',', ' ')
print(lines)

In your sample code, line contains the full content of the file as a str.
my_file = open('data.txt')
lines = my_file.read()
You then later re-open the file to iterate the lines:
with open('data.txt') as f:
for line in f:
lines.split()
lines.replace(',', ' ')
Note, however, str.split and str.replace do not modify the existing value, as strs in python are immutable. Also note you are operating on lines there, rather than the for-loop variable line.
Instead, you'll need to assign the result of those functions into new values, or give them as arguments (E.g., to print). So you'll want to open the file, iterate over the lines and print the value with the "," replaced with a " ":
with open("data.txt") as f:
for line in f:
print(line.replace(",", " "))
Or, since you are operating on the whole file anyway:
with open("data.txt") as f:
print(f.read().replace(",", " "))
Or, as your file appears to be CSV content, you may wish to use the csv module from the standard library instead:
import csv
with open("data.txt", newline="") as csvfile:
for row in csv.reader(csvfile):
print(*row)

with open('data.txt', 'r') as f:
for line in f:
for value in line.split(','):
print(value)

while python can offer us several ways to open files this is the prefered one for working with files. becuase we are opening the file in lazy mode (this is the prefered one espicialy for large files), and after exiting the with scope (identation block) the file io will be closed automaticly by the system.
here we are openening the file in read mode. files folow the iterator polices, so we can iterrate over them like lists. each line is a true line in the file and is a string type.
After getting the line, in line variable, we split (see str.split()) the line into 2 tokens, one before the comma and the other after the comma. split return new constructed list of strings. if you need to omit some unwanted characters you can use the str.strip() method. usualy strip and split combined together.
elegant and efficient file reading - method 1
with open("data.txt", 'r') as io:
for line in io:
sl=io.split(',') # now sl is a list of strings.
print("{} {}".format(sl[0],sl[1])) #now we use the format, for printing the results on the screen.
non elegant, but efficient file reading - method 2
fp = open("data.txt", 'r')
line = None
while (line=fp.readline()) != '': #when line become empty string, EOF have been reached. the end of file!
sl=line.split(',')
print("{} {}".format(sl[0],sl[1]))

Python Splitting Text file based on a keyword

I am trying to write a python program that will constantly read a text file line by line and each time it comes across a line with the word 'SPLIT' it will write the contents to a new text file.
Please could someone point me in the right direction of writing a new text file each time the script comes across the word 'split'. I have no problem reading a text file with Python, I'm unsure how to split on the keyword and create an individual text file each time.
THE SCRIPT BELOW WORKS IN 2.7.13
file_counter = 0
done = False
with open('test.txt') as input_file:
# with open("test"+str(file_counter)+".txt", "w") as out_file:
while not done:
for line in input_file:
if "SPLIT" in line:
done = True
file_counter += 1
else:
print(line)
out_file = open("test"+str(file_counter)+".txt", "a")
out_file.write(line)
#out_file.write(line.strip()+"\n")
print file_counter

You need to have two loops. One which iterates the filenames of the output files then another inside to write the input contents to the current active output until "split" is found:
out_n = 0
done = False
with open("test.txt") as in_file:
while not done: #loop over output file names
with open(f"out{out_n}.txt", "w") as out_file: #generate an output file name
while not done: #loop over lines in inuput file and write to output file
try:
line = next(in_file).strip() #strip whitespace for consistency
except StopIteration:
done = True
break
if "SPLIT" in line: #more robust than 'if line == "SPLIT\n":'
break
else:
out_file.write(line + '\n') #must add back in newline because we stripped it out earlier
out_n += 1 #increment output file name integer

for line in text.splitlines():
if " SPLIT " in line:
# write in new file.
pass
To write in new file check here:
https://www.tutorialspoint.com/python/python_files_io.htm
or
https://docs.python.org/3.6/library/functions.html#open

Insert text in between file lines in python

I have a file that I am currently reading from using
fo = open("file.txt", "r")
Then by doing
file = open("newfile.txt", "w")
file.write(fo.read())
file.write("Hello at the end of the file")
fo.close()
file.close()
I basically copy the file to a new one, but also add some text at the end of the newly created file. How would I be able to insert that line say, in between two lines separated by an empty line? I.e:
line 1 is right here
<---- I want to insert here
line 3 is right here
Can I tokenize different sentences by a delimiter like \n for new line?

First you should load the file using the open() method and then apply the .readlines() method, which splits on "\n" and returns a list, then you update the list of strings by inserting a new string in between the list, then simply write the contents of the list to the new file using the new_file.write("\n".join(updated_list))
NOTE: This method will only work for files which can be loaded in the memory.
with open("filename.txt", "r") as prev_file, open("new_filename.txt", "w") as new_file:
prev_contents = prev_file.readlines()
#Now prev_contents is a list of strings and you may add the new line to this list at any position
prev_contents.insert(4, "\n This is a new line \n ")
new_file.write("\n".join(prev_contents))

readlines() is not recommended because it reads the whole file into memory. It is also not needed because you can iterate over the file directly.
The following code will insert Hello at line 2 at line 2
with open('file.txt', 'r') as f_in:
with open('file2.txt','w') as f_out:
for line_no, line in enumerate(f_in, 1):
if line_no == 2:
f_out.write('Hello at line 2\n')
f_out.write(line)
Note the use of the with open('filename','w') as filevar idiom. This removes the need for an explicit close() because it closes the file automatically at the end of the block, and better, it does this even if there is an exception.

For Large file
with open ("s.txt","r") as inp,open ("s1.txt","w") as ou:
for a,d in enumerate(inp.readlines()):
if a==2:
ou.write("hi there\n")
ou.write(d)

U could use a marker
#FILE1
line 1 is right here
<INSERT_HERE>
line 3 is right here
#FILE2
some text
with open("FILE1") as file:
original = file.read()
with open("FILE2") as input:
myinsert = input.read()
newfile = orginal.replace("<INSERT_HERE>", myinsert)
with open("FILE1", "w") as replaced:
replaced.write(newfile)
#FILE1
line 1 is right here
some text
line 3 is right here

How do I write a string to a specific line number?

I am having trouble finding the answer to this after quite a bit of searching.
What I want to do is, do a string search and write on the line above or below it, depending on my string.
Here is something I've done so far:
file = open('input.txt', 'r+')
f = enumerate(file)
for num, line in f:
if 'string' in line:
linewrite = num - 1
???????
EDIT EXTENSION OF INITIAL QUESTION:
I already picked the answer that best solved my initial question. But now using Ashwini's method where I rewrote the file, how can I do a search AND REPLACE a string. To be more specific.
I have a text file with
SAMPLE
AB
CD
..
TYPES
AB
QP
PO
..
RUNS
AB
DE
ZY
I want to replace AB with XX, ONLY UNDER lines SAMPLE and RUNS
I've already tried multiple ways of using replace(). I tried something like
if 'SAMPLE' in line:
f1.write(line.replace('testsample', 'XX'))
if 'RUNS' in line:
f1.write(line.replace('testsample', 'XX'))
and that didn't work

The following can be used as a template:
import fileinput
for line in fileinput.input('somefile', inplace=True):
if 'something' in line:
print 'this goes before the line'
print line,
print 'this goes after the line'
else:
print line, # just print the line anyway

You may have to read all the lines in a list first, and if the condition is matched you can then store your string at a particular index using list.insert
with open('input.txt', 'r+') as f:
lines = f.readlines()
for i, line in enumerate(lines):
if 'string' in line:
lines.insert(i,"somedata") # inserts "somedata" above the current line
f.truncate(0) # truncates the file
f.seek(0) # moves the pointer to the start of the file
f.writelines(lines) # write the new data to the file
or without storing all the lines you'll need a temporary file to store the data, and then
rename the temporary file to the original file:
import os
with open('input.txt', 'r') as f, open("new_file",'w') as f1:
for line in f:
if 'string' in line:
f1.write("somedate\n") # Move f1.write(line) above, to write above instead
f1.write(line)
os.remove('input.txt') # For windows only
os.rename("newfile", 'input.txt') # Rename the new file