Develop Reference

Extract and take average of csv column based on condition

I'm taking a class on Python, so I'm definitely not using it on a day to day basis but am trying my best to learn. One of my assignments is to read data from a csv file and get the average beginning weight of males and females in the file. I'm not allowed to use Pandas or any other external packages so am just importing csv to read the data. My issue is, I can do the calculations for the first condition but when it gets to the second condition, it's returning the error, 'division by zero.' I have no idea what I'm doing wrong and was hoping someone could help me. I have confirmed the values for males and females by looking at the file and there is data for both.
This is the code that's returning 'division by zero.' If I reverse the rows and do males first instead of females, it does the same for females. If I just print out f_weight and m_weight, f_weight is populated, m_weight returns [].
import csv
def avg_start_weight(csv_data):
with open(csv_data, newline='') as csv_file:
csv_reader = csv.reader(csv_file, delimiter='|')
f_weight = [float(row[1]) for row in csv_reader if row[0] == 'F']
m_weight = [float(cell[1]) for cell in csv_reader if cell[0] == 'M']
f_average = sum(f_weight) / len(f_weight)
m_average = sum(m_weight) / len(m_weight)
print(f'2. The average female beginning weight is {f_average:.2f} and the average male beginning weight is {m_average:.2f}')
csv_file.close()
csv_data = 'freshman_kgs.csv'
avg_start_weight(csv_data)
I did get it to work this way but am guessing I shouldnt need to close and re-open the file each time I want to test a different condition, so wanted to see if I could get some help figuring out what Im doing wrong:
import csv
def avg_start_weight(csv_data):
with open(csv_data, newline='') as csv_file:
csv_reader = csv.reader(csv_file, delimiter='|')
f_weight = [float(row[1]) for row in csv_reader if row[0] == 'F']
f_average = sum(f_weight) / len(f_weight)
print(f'2. Average female beginning weight: {f_average:.2f}')
csv_file.close()
with open(csv_data, newline='') as csv_file:
csv_reader = csv.reader(csv_file, delimiter='|')
m_weight = [float(row[1]) for row in csv_reader if row[0] == 'M']
m_average = sum(m_weight) / len(m_weight)
print(f'Average male beginning weight: {m_average:.2f}')
csv_file.close()
csv_data = "freshman_kgs.csv"
avg_start_weight(csv_data)
The last thing I tried was this and it also returns 'division by zero' depending on which calculation I put second:
import csv
def avg_start_weight(csv_data):
with open(csv_data, newline='') as csv_file:
csv_reader = csv.reader(csv_file, delimiter='|')
m_weight = [float(row[1]) for row in csv_reader if row[0] == 'M']
m_weight_avg = sum(m_weight) / len(m_weight)
print(f'The average beginning weight for males was {m_weight_avg:.2f}')
f_weight = [float(row[1]) for row in csv_reader if row[0] == 'F']
f_weight_avg = sum(f_weight) / len(f_weight)
print(f'The average beginning weight for females was {f_weight_avg:.2f}')
csv_data = "freshman_kgs.csv"
avg_start_weight(csv_data)
I'm not at all asking for someone to do my homework for me, I'm trying my best to understand this and figure it out myself but am stuck. I really appreciate any help that I get. Ive been looking at others questions and most of them are using Pandas so weren't of much use to me unfortunately.

I see a couple of thing wrong with your code.
the line csv_reader = csv.reader(csv_file, delimiter='|') produces an iterable which you need to process to get at specific lines of data, also when you utilize the with open construct, it isn't necessary to close the file, since that is automatically handled for you. Here is how I would write the required code as a reference:
def avg_start_weight(csv_data, newline=''):
weights = {'M':[], 'F':[]}
with open(csv_data, mode='r') as file:
csv_file = csv.reader(file)
for lines in csv_file:
if lines[0] == 'M' or lines[0] == 'F':
weights[lines[0]].append(int(lines[1]))
print(f"Average Female Weight = {sum(weights['F'])/len(weights['F']):.2f}")
print(f"Average Male Weight = {sum(weights['M'])/len(weights['F']):.2f}")
Notice: I used a dictionary structure to hold a list of Male and female weights. While reading the lines of the csv file, I add new values to the appropriate dictionary key as defined by the first item in each line. Also note that I converted the input data from a string to an integer as I was adding each item to it's appropriate list.

CSV reading and writing; outputted CSV is blank

My program needs a function that reads data from a csv file ("all.csv") and extracts all the data pertaining to 'Virginia' (extract each row that has 'Virginia in it), then writes the extracted data to another csv file named "Virginia.csv" The program runs without error; however, when I open the "Virginia.csv" file, it is blank. My guess is that the issue is with my nested for loop, but I am not entirely sure what is causing the issue.
Here is the data within the all.csv file:
https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv
Here is my code:
import csv
input_file = 'all.csv'
output_file = 'Virginia.csv'
state = 'Virginia'
mylist = []
def extract_records_for_state (input_file, output_file, state):
with open(input_file, 'r') as infile:
contents = infile.readlines()
with open(output_file, 'w') as outfile:
writer = csv.writer(outfile)
for row in range(len(contents)):
contents[row] = contents[row].split(',') #split elements
for row in range(len(contents)):
for word in range(len(contents[row])):
if contents[row][2] == state:
writer.writerow(row)
extract_records_for_state(input_file,output_file,state)

I ran your code and it gave me an error
Traceback (most recent call last):
File "c:\Users\Dolimight\Desktop\Stack Overflow\Geraldo\main.py", line 27, in
extract_records_for_state(input_file, output_file, state)
File "c:\Users\Dolimight\Desktop\Stack Overflow\Geraldo\main.py", line 24, in extract_records_for_state
writer.writerow(row)
_csv.Error: iterable expected, not int,
I fixed the error by putting the contents of the row [contents[row]] into the writerow() function and ran it again and the data showed up in Virginia.csv. It gave me duplicates so I also removed the word for-loop.
import csv
input_file = 'all.csv'
output_file = 'Virginia.csv'
state = 'Virginia'
mylist = []
def extract_records_for_state(input_file, output_file, state):
with open(input_file, 'r') as infile:
contents = infile.readlines()
with open(output_file, 'w') as outfile:
writer = csv.writer(outfile)
for row in range(len(contents)):
contents[row] = contents[row].split(',') # split elements
print(contents)
for row in range(len(contents)):
if contents[row][2] == state:
writer.writerow(contents[row]) # this is what I changed
extract_records_for_state(input_file, output_file, state)

You have two errors. The first is that you try to write the row index at writer.writerow(row) - the row is contents[row]. The second is that you leave the newline in the final column on read but don't strip it on write. Instead you could leverage the csv module more fully. Let the reader parse the rows. And instead of reading into a list, which uses a fair amount of memory, filter and write row by row.
import csv
input_file = 'all.csv'
output_file = 'Virginia.csv'
state = 'Virginia'
mylist = []
def extract_records_for_state (input_file, output_file, state):
with open(input_file, 'r', newline='') as infile, \
open(output_file, 'w', newline="") as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
# add header
writer.writerow(next(reader))
# filter for state
writer.writerows(row for row in reader if row[2] == state)
extract_records_for_state(input_file,output_file,state)

Looking at your code two things jump out at me:
I see a bunch of nested statements (logic)
I see you reading a CSV as plain text, then interpreting it as CSV yourself (contents[row] = contents[row].split(',')).
I recommend two things:
break up logic into distinct chunks: all that nesting can be hard to interpret and debug; do one thing, prove that works; do another thing, prove that works; etc...
use the CSV API to its fullest: use it to both read and write your CSVs
I don't want to try and replicate/fix your code, instead I'm offering this general approach to achieve those two goals:
import csv
# Read in
all_rows = []
with open('all.csv', 'r', newline='') as f:
reader = csv.reader(f)
next(reader) # discard header (I didn't see you keep it)
for row in reader:
all_rows.append(row)
# Process
filtered_rows = []
for row in all_rows:
if row[2] == 'Virginia':
filtered_rows.append(row)
# Write out
with open('filtered.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(filtered_rows)
Once you understand both the logic and the API of those discrete steps, you can move on (advance) to composing something more complex, like the following which reads a row, decides if it should be written, and if so, writes it:
import csv
with open('filtered.csv', 'w', newline='') as f_out:
writer = csv.writer(f_out)
with open('all.csv', 'r', newline='') as f_in:
reader = csv.reader(f_in)
next(reader) # discard header
for row in reader:
if row[2] == 'Virginia':
writer.writerow(row)
Using either of those two pieces of code on this (really scaled-down) sample of all.csv:
date,county,state,fips,cases,deaths
2020-03-09,Fairfax,Virginia,51059,4,0
2020-03-09,Virginia Beach city,Virginia,51810,1,0
2020-03-09,Chelan,Washington,53007,1,1
2020-03-09,Clark,Washington,53011,1,0
gets me a filtered.csv that looks like:
2020-03-09,Fairfax,Virginia,51059,4,0
2020-03-09,Virginia Beach city,Virginia,51810,1,0
Given the size of this dataset, the second approach of write-on-demand-inside-the-read-loop is both faster (about 5x faster on my machine) and uses significantly less memory (about 40x less on my machine) because there's no intermediate storage with all_rows.
But, please take the time to run both, read them carefully, and see how each works the way it does.

getting average of some digits from a csv file as input and Write the averages in an output csv file in python 3

I am learning python3 :), and I am trying to read a CSV file with different rows
and take the average of the scores for each person(in each row)
and write it in a CSV file as an output in python 3.
The input file is like below:
David,5,2,3,1,6
Adele,3,4,1,5,2,4,2,1
...
The output file should seem like below:
David,4.75
Adele,2.75
...
It seems that I am reading the file correctly, as I print
the average for each name in the terminal, but in CSV
output file it prints only the average of the last name
of the input file, while I want to print all names and
corresponding averages in CSV output file.
Anybody can help me with it?
import csv
from statistics import mean
these_grades = []
name_list = []
reader = csv.reader(open('input.csv', newline=''))
for row in reader:
name = row[0]
name_list.append(name)
with open('result.csv', 'w', newline='\n') as f:
writer = csv.writer(f,
delimiter=',',
quotechar='"',
quoting=csv.QUOTE_MINIMAL)
for grade in row[1:]:
these_grades.append(int(grade))
for item in name_list:
writer.writerow([''.join(item), mean(these_grades)])
print('%s,%f' % (name , mean(these_grades)))

There are several issues in your code:
You're not using a context manager (with) when you read the input file. There's no reason to use it when writing but not when reading - you consequently don't close the "input.csv" file
You're using a list to store data from rows. This doesn't easily distinguish between the person's name and the scores associated with the person. It would be better to use a dictionary in which the key is the person's name, and the values stored against that key are the individual scores
You repeatedly open the file within a for loop in 'w' mode. Every time you open a file in write mode, it just wipes all the previous contents. You actually do write each row to the file, but you just wipe it again when you open the file on the next iteration.
You can use:
import csv
import statistics
# use a context manager to read the data in too, not just for writing
with open('input.csv') as infile:
reader = csv.reader(infile)
data = list(reader)
# Create a dictionary to store the scores against the name
scores = {}
for row in data:
scores[row[0]] = row[1:] # First item in the row is the key (name) and the rest is values
with open('output.csv', 'w', newline='') as outfile:
writer = csv.writer(outfile)
# Now we need to iterate the dictionary and average the score on each iteration
for name, scores in scores.items():
ave_score = statistics.mean([int(item) for item in scores])
writer.writerow([name, ave_score])
This can be further consolidated, but it's less easy to see what's happening:
with open('input.csv') as infile, open('output.csv', 'w', newline='') as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
for row in reader:
name = row[0]
values = row[1:]
ave_score = statistics.mean(map(int, values))
writer.writerow([name, ave_score])

Parsing CSV files using Python 2.7

I'm trying to write a script that will open a CSV file and write rows from that file to a new CSV file based on the match criteria of a unique telephone number in column 4 of csv.csv. The phone numbers are always in column 4, and are often duplicated in the file, however the other columns are often unique, thus each row is inherently unique.
A row from the csv file I'm reading looks like this: (the TN is 9259991234)
2,PPS,2015-09-17T15:44,9259991234,9DF51758-A2BD-4F65-AAA2
I hit an error with the code below saying that '_csv.writer' is not iterable and I'm not sure how to modify my code to solve the problem.
import csv
import sys
import os
os.chdir(r'C:\pTest')
with open(r'csv.csv', 'rb') as f:
reader = csv.reader(f, delimiter=',')
with open (r'new_csv.csv', 'ab') as new_f:
writer = csv.writer(new_f, delimiter=',')
for row in reader:
if row[3] not in writer:
writer.writerow(new_f)

Your error stems from this expression:
row[3] not in writer
You cannot test for membership against a csv.writer() object. If you wanted to track if you already have processed a phone number, use a separate set() object to track those:
with open(r'csv.csv', 'rb') as f:
reader = csv.reader(f, delimiter=',')
with open (r'new_csv.csv', 'ab') as new_f:
writer = csv.writer(new_f, delimiter=',')
seen = set()
for row in reader:
if row[3] not in seen:
seen.add(row[3])
writer.writerow(row)
Note that I also changed your writer.writerow() call; you want to write the row, not the file object.

Finding string in row to overwrite this row of CSV using Python 2.7

I'm following some feedback from another thread, but have gotten stuck. I'm looking to search an existing csv file to locate the row in which a string occurs. I am then looking to update this row with new data.
What I have so far gives me an "TypeError: unhasable type: 'list'":
allLDR = []
with open('myfile.csv', mode='rb') as f:
reader = csv.reader(f)
#allLDR.extend(reader)
for num, row in enumerate(reader):
if myField in row[0]:
rowNum = row
line_to_override = {rowNum:[nMaisonField, entreeField, indiceField, cadastreField]}
with open('myfile.csv', 'wb') as ofile:
writer = csv.writer(ofile, quoting=csv.QUOTE_NONE, delimiter=',')
#for line, row in enumerate(allLDR):
for line, row in enumerate(reader):
data = line_to_override.get(line, row)
writer.writerow(data)

The line allDR.extend(reader) consumes all of the input lines from the csv.reader object. Therefore, the for loop never runs, and rowNum=row is never executed, and {rowNum:blah} generates an exception.
Try commenting out the allDR.extend(reader) line.
As a debugging aid, try adding print statements inside the for loop and inside the conditional.
Here is a program which does what I think you want your program to do: it reads in myfile.csv, modifies rows conditionally based on the content of the first cell, and writes the file back out.
import csv
with open('myfile.csv', mode='rb') as ifile:
allDR = list(csv.reader(ifile))
for row in allDR:
if 'fowl' in row[0]:
row[:] = ['duck', 'duck', 'goose']
with open('myfile.csv', 'wb') as ofile:
csv.writer(ofile).writerows(allDR)