Develop Reference

Problem skipping line whilst iterating using previous line and current line comparison

I have a list of sorted data arranged so that each item in the list is a csv line to be written to file.
The final step of the script checks the contents of each field and if all but the last field match then it will copy the current line's last field onto the previous line's last field.
I would like to as I've found and processed one of these matches skip the current line where the field was copied from thus only leaving one of the lines.
Here's an example set of data
field1,field2,field3,field4,something
field1,field2,field3,field4,else
Desired output
field1,field2,field3,field4,something else
This is what I have so far
output_csv = ['field1,field2,field3,field4,something',
'field1,field2,field3,field4,else']
# run through the output
# open and create a csv file to save output
with open('output_table.csv', 'w') as f:
previous_line = None
part_duplicate_line = None
part_duplicate_flag = False
for line in output_csv:
part_duplicate_flag = False
if previous_line is not None:
previous = previous_line.split(',')
current = line.split(',')
if (previous[0] == current[0]
and previous[1] == current[1]
and previous[2] == current[2]
and previous[3] == current[3]):
print(previous[0], current[0])
previous[4] = previous[4].replace('\n', '') + ' ' + current[4]
part_duplicate_line = ','.join(previous)
part_duplicate_flag = True
f.write(part_duplicate_line)
if part_duplicate_flag is False:
f.write(previous_line)
previous_line = line
ATM script adds the line but doesn't skip the next line, I've tried various renditions of continue statements after part_duplicate_line is written to file but to no avail.

Looks like you want one entry for each combination of the first 4 fields
You can use a dict to aggregate data -
#First we extract the key and values
output_csv_keys = list(map(lambda x: ','.join(x.split(',')[:-1]), output_csv))
output_csv_values = list(map(lambda x: x.split(',')[-1], output_csv))
#Then we construct a dictionary with these keys and combine the values into a list
from collections import defaultdict
output_csv_dict = defaultdict(list)
for key, value in zip(output_csv_keys, output_csv_values):
output_csv_dict[key].append(value)
#Then we extract the key/value combinations from this dictionary into a list
for_printing = [','.join([k, ' '.join(v)]) for k, v in output_csv_dict.items()]
print(for_printing)
#Output is ['field1,field2,field3,field4,something else']
#Each entry of this list can be output to the csv file

I propose to encapsulate what you want to do in a function where the important part obeys this logic:
either join the new info to the old record
or output the old record and forget it
of course at the end of the loop we have in any case a dangling old record to output
def join(inp_fname, out_fname):
'''Input file contains sorted records, when two (or more) records differ
only in the last field, we join the last fields with a space
and output only once, otherwise output the record as-is.'''
######################### Prepare for action ##########################
from csv import reader, writer
with open(inp_fname) as finp, open(out_fname, 'w') as fout:
r, w = reader(finp), writer(fout)
######################### Important Part starts here ##############
old = next(r)
for new in r:
if old[:-1] == new[:-1]:
old[-1] += ' '+new[-1]
else:
w.writerow(old)
old = new
w.writerow(old)
To check what I've proposed you can use these two snippets (note that these records are shorter than yours, but it's an example and it doesn't matter because we use only -1 to index our records).
The 1st one has a "regular" last record
open('a0.csv', 'w').write('1,1,2\n1,1,3\n1,2,0\n1,3,1\n1,3,2\n3,3,0\n')
join('a0.csv', 'a1.csv')
while the 2nd has a last record that must be joined to the previous one.
open('b0.csv', 'w').write('1,1,2\n1,1,3\n1,2,0\n1,3,1\n1,3,2\n')
join('b0.csv', 'b1.csv')
If you run the snippets, as I have done before posting, in the environment where you have defined join you should get what you want.

Dividing an array in a for loop?

I have been given a task by my teacher and one of the questions wants me to divide everything in the array by 26.22(A full marathon). I have been working on this all day and am totally stuck could someone please show me how to make this work?
this is what I have so far
import string
forename = []
surname = []
distance = []
farthest_walk = []
marathon = []
#Opening the text file and sorting variables
data = open("members.txt","r")
for line in data:
value = line.split(',')
forename.append(value[0])
surname.append(value[1])
distance.append(value[2])
#Closing the text file
data.close()
Results = open("Results.txt","w+")
Results.write("The number of whole marathons walked be each member is:\n")
for count in range(len(distance)):
if float(distance[count])/ 26.22 = temp:
marathon.append
Results.write(forename[count]+":")
Results.write(surname[count]+":")
Results.write(marathon[count])
Results.close()
It is supposed to end up as Forename, Surname, WholeMarathosRun but I don't see how it could get there.

You almost got there.
For each name, you need to compute how many marathons he ran, which can be achieved with the following operation:
temp = float(distance[count])/ 26.22
This doesn't need to be in an if statement.
Then you need to write this value in the output file after the names:
Results.write(forename[count]+":")
Results.write(surname[count]+":")
Results.write(temp)
# line break such that each result stay in one line
Results.write("\n")
All those lines go inside the last for loop that you already have.

How to output five lists from a loop?

This is my code:
import re
with open("C:\\Corpora\\record-13.txt") as f:
concepts = f.readlines()
j = 0
for line in concepts:
PATTERN = re.compile(r'''((?:[^ "]|"[^"]*")+)''')
TokCurrLineCon = PATTERN.split(line)[1::2]
temp = TokCurrLineCon[1].split(':')
StartLineNum[j] = temp[0]
StartOffset[j] = temp[1]
temp = TokCurrLineCon[2].split('||')
EndOfCon[j] = temp[0]
TypeOfCon[j] = temp[1]
temp = EndOfCon[j].split(':')
EndLineNum[j] = temp[0]
EndOffset[j] = temp[1]
temp = TypeOfCon[j].split('"')
TypeOfCon[j] = temp[1]
j +=1
I need 5 lists as the end (StartLineNum, StartOffset, EndLineNum, EndOffset, TypeOfCon), but when I run it I face the error StartLineNum[j] = temp[0]
TypeError: 'str' object does not support item assignment
Any idea how to fix it?

The error message is telling you that StartLineNum is a str, so StartLineNum[j] = <anything> is illegal.
From your description, it sounds like you expected StartLineNum to be a list. So presumably the problem is that you constructed a string instead of a list somewhere in the code above. Since we can't see that code, we can't fix it, beyond saying that you should create a list if you want a list.
However, I suspect there's another problem in your code. For this to work, StartLineNum would have to be not just a list, but a list that's already got as many members as the file has lines. But you can't know how many that is until you've read the whole file in. A better solution would be to use the append method on lists. (Then you don't need the j variable, either.) For example:
StartLineNum = []
for line in concepts:
# blah blah
StartLineNum.append(temp[0])
# etc.

reusable function: substituting the values returned by another function

Below is the snippet: I'm parsing job log and the output is the formatted result.
def job_history(f):
def get_value(j,n):
return j[n].split('=')[1]
lines = read_file(f)
for line in lines:
if line.find('Exit_status=') != -1:
nLine = line.split(';')
jobID = '.'.join(nLine[2].split('.',2)[:-1]
jData = nLine[3].split(' ')
jUsr = get_value(jData,0)
jHst = get_value(jData,9)
jQue = get_value(jData,3)
eDate = job_value(jData,14)
global LJ,LU,LH,LQ,LE
LJ = max(LJ, len(jobID))
LU = max(LU, len(jUsr))
LH = max(LH, len(jHst))
LQ = max(LQ, len(jQue))
LE = max(LE, len(eDate))
print "%-14s%-12s%-14s%-12s%-10s" % (jobID,jUsr,eDate,jHst,jQue)
return LJ,LU,LE,LH,LQ
In principle, I should have another function like this:
def fmt_print(a,b,c,d,e):
print "%-14s%-12s%-14s%-12s%-10s\n" % (a,b,c,d,e)
to print the header and call the functions like this to print the complete result:
fmt_print('JOB ID','OWNER','E_DATE','R_HOST','QUEUE')
job_history(inFile)
My question is: how can I make fmt_print() to print both the header and the result using the values LJ,LU,LE,LH,LQ for the format spacing. the job_history() will parse a number of log files from the log-directory. The length of the field of similar type will differ from file to file and I don't wanna go static with the spacing (assuming the max length per field) for this as there gonna be lot more columns to print (than the example). Thanks in advance for your help. Cheers!!
PS. For those who know my posts: I don't have to use python v2.3 anymore. I can use even v2.6 but I want my code to be v2.4 compatible to go with RHEL5 default.
Update: 1
I had a fundamental problem in my original script. As I mentioned above that the job_history() will read the multiple files in a directory in a loop, the max_len were being calculated per file and not for the entire result. After modifying
unutbu's script a little bit and following xtofl's (if this is what it meant) suggestion, I came up with this, which seems to be working.
def job_history(f):
result=[]
for line in lines:
if line.find('Exit_status=') != -1:
....
....
global LJ,LU,LH,LQ,LE
LJ = max(LJ, len(jobID))
LU = max(LU, len(jUsr))
LH = max(LH, len(jHst))
LQ = max(LQ, len(jQue))
LE = max(LE, len(eDate))
result.append((jobID,jUsr,eDate,jHst,jQue))
return LJ,LU,LH,LQ,LE,result
# list of log files
inFiles = [ m for m in os.listdir(logDir) ]
saved_ary = []
for inFile in sorted(inFiles):
LJ,LU,LE,LH,LQ,result = job_history(inFile)
saved_ary += result
# format printing
fmt_print = "%%-%ds %%-%ds %%-%ds %%-%ds %%-%ds" % (LJ,LU,LE,LH,LQ)
print_head = fmt_print % ('Job Id','User','End Date','Exec Host','Queue')
print '%s\n%s' % (print_head, len(print_head)*'-')
for lines in saved_ary:
print fmt_print % lines
I'm sure there are lot other better ways of doing this, so suggestion(s) are welcomed. cheers!!
Update: 2
Sorry for brining up this "solved" post again. Later discovered, I was even wrong with my updated script, so I thought I'd post another update for future reference. Even though it appeared to be working, actually length_data were overwritten with the new one for every file in the loop. This works correctly now.
def job_history(f):
def get_value(j,n):
return j[n].split('=')[1]
lines = read_file(f)
for line in lines:
if "Exit_status=" in line:
nLine = line.split(';')
jobID = '.'.join(nLine[2].split('.',2)[:-1]
jData = nLine[3].split(' ')
jUsr = get_value(jData,0)
....
result.append((jobID,jUsr,...,....,...))
return result
# list of log files
inFiles = [ m for m in os.listdir(logDir) ]
saved_ary = []
LJ = 0; LU = 0; LE = 0; LH = 0; LQ = 0
for inFile in sorted(inFiles):
j_data = job_history(inFile)
saved_ary += j_data
for ix in range(len(saved_ary)):
LJ = max(LJ, len(saved_ary[ix][0]))
LU = max(LU, len(saved_ary[ix][1]))
....
# format printing
fmt_print = "%%-%ds %%-%ds %%-%ds %%-%ds %%-%ds" % (LJ,LU,LE,LH,LQ)
print_head = fmt_print % ('Job Id','User','End Date','Exec Host','Queue')
print '%s\n%s' % (print_head, len(print_head)*'-')
for lines in saved_ary:
print fmt_print % lines
The only problem is it's taking a bit of time to start printing the info on the screen, just because, I think, as it's putting all the in the array first and then printing. Is there any why can it be improved? Cheers!!

Since you don't know LJ, LU, LH, LQ, LE until the for-loop ends, you have to complete this for-loop before you print.
result=[]
for line in lines:
if line.find('Exit_status=') != -1:
...
LJ = max(LJ, len(jobID))
LU = max(LU, len(jUsr))
LH = max(LH, len(jHst))
LQ = max(LQ, len(jQue))
LE = max(LE, len(eDate))
result.append((jobID,jUsr,eDate,jHst,jQue))
fmt="%%-%ss%%-%ss%%-%ss%%-%ss%%-%ss"%(LJ,LU,LE,LH,LQ)
for jobID,jUsr,eDate,jHst,jQue in result:
print fmt % (jobID,jUsr,eDate,jHst,jQue)
The fmt line is a bit tricky. When you use string interpolation, each %s gets replaced by a number, and %% gets replaced by a single %. This prepares the correct format for the subsequent print statements.

Since the column header and column content are so closely related, why not couple them into one structure, and return an array of 'columns' from your job_history function? The task of that function would be to
output the header for each colum
create the output for each line, into the corresponding column
remember the maximum width for each column, and store it in the column struct
Then, the prinf_fmt function can 'just'
iterate over the column headers, and print them using the respective width
iterate over the 'rest' of the output, printing each cell with 'the respective width'
This design will separate output definition from actual formatting.
This is the general idea. My python is not that good; but I may think up some example code later...

Depending on how many lines are there, you could:
read everything first to figure out the maximum field lengths, then go through the lines again to actually print out the results (if you have only a handful of lines)
read one page of results at a time and figure out maximum length for the next 30 or so results (if you can handle the delay and have many lines)
don't care about the format and output in a csv or some database format instead - let the final person / actual report generator worry about importing it

Embedded for loop with regex

def find_string(header,file_1,counter):
ab = re.compile(str(header))
for line in file_1:
if re.search(ab,line) !=None:
print line
counter+=1
return counter
file_1 = open("text_file_with_headers.txt",'r')
header_array = []
header_array.append("header1")
header_array.append("header2")
# ...
counter = 0
for header in header_array:
counter = find_string(header,file_1,counter)
Every time I run this it searches for only one of the headers and I cannot figure out why.

Because when the loop for line in file_1: has ended for the first header, the file's pointer is at the end of the file. You must move this pointer to the file's beginning again, that is done with function seek() . You must add seek(0,0) like that
counter = 0
for header in header_array:
counter = find_string(header,file_1,counter)
f1.seek(0,0)
.
EDIT
1) ab is a compiled regex, then you can write ab.search(line)
2) bool(None) is False, then you can write if ab.search(line): no need of != None
3)
def find_string(header,file_1,counter):
lwh = re.compile('^.*?'+header+'.*$',re.MULTILINE)
lines_with_header = lwh.findall(file-1.read())
print ''.join(lines_with_header)
return counter + 1
and even
def find_string(header,file_1,counter):
lwh = re.compile('^.*?'+header+'.*$',re.MULTILINE)
print ''.join(matline.group() for matline in lwh.finditer(file-1.read()) )
return counter + 1
4)
def find_string(header,file_1):
lwh = re.compile('^.*?'+header+'.*$',re.MULTILINE)
lines_with_header = lwh.findall(file-1.read())
print ''.join(lines_with_header)
file_1 = open("text_file_with_headers.txt",'r')
header_list = ["header1","header2",....]
for counter,header in header_list:
find_string(header,file_1)
file_1.seek(0,0)
counter += 1 # because counter began at 0
5) You run through file_1 as many times that there are headers in header_list.
You should run through it only one time and recording each line containing one of the headers in a list being one of the values of a dictionary whose keys should be the headers. It would be faster.
6) An array in Python is an array

The file object keeps track of your position in the file, and after you've gone through the outer loop once, you're at the end of the file and there are no more lines to read.
If I were you, I would reverse the order in which your loops are nested: I would iterate through the file line by line, and for each line, iterate through the list of strings you want to find. That way, I would only have to read each line from the file once.