Develop Reference

Select column from multiple DataFrames based on same header prefix

I have a function that iterates over the rows of a csv for the Age column and if an age is negative, it will print the Key and the Age value to a text file.
def neg_check():
results = []
file_path = input('Enter file path: ')
file_data = pd.read_csv(file_path, encoding = 'utf-8')
for index, row in file_data.iterrows():
if row['Age'] < 0:
results.append((row['Key'], row['Age']))
with open('results.txt', 'w') as outfile:
outfile.write("\n".join(map(str, results)))
outfile.close()
In order to make this code repeatable, how can I modify it so it will iterate the rows if the column starts with "Age"? My files have many columns that start with "Age" but end differently. . I tried the following...
if row.startswith['Age'] < 0:
and
if row[row.startswith('Age')] < 0:
but it throws AttributeError: 'Series' object has no attribute 'startswith' error.
My csv files:
sample 1
Key Sex Age
1 Male 46
2 Female 34
sample 2
Key Sex AgeLast
1 Male 46
2 Female 34
sample 3
Key Sex AgeFirst
1 Male 46
2 Female 34

I would do this in one step, but there are a few options. One is filter:
v = df[df.filter(like='AgeAt').iloc[:, 0] < 0]
Or,
c = df.columns[df.columns.str.startswith('AgeAt')][0]
v = df[df[c] < 0]
Finally, to write to CSV, use
if not v.empty:
v.to_csv('invalid.csv')
Looping over your data is not necessary with pandas.

CSV file for calculating passing of failing grade from a set of marks

I have an CSV file, which has the current format below.
A two row example of this would be:
first_Name last_Name test1 test2 test3 test4
Alex Brian 11 17 13 24
Pete Tong 19 14 12 30
Now my current code does not work, Simply put I am not sure if I am on the right track.
My current code:
def grader(test1, test2, test3, finalExam):
first = test1 * .20
second = test2 * .20
third = test3 * .20
fourth = finalExam *.40
finalGrade = first + second + third + fourth
return finalGrade
def gradeScores(FinalGrade):
if FinalGrade >= 90 and FinalGrade <= 100:
return("You received an A")
elif FinalGrade >= 80 and FinalGrade < 90:
return("You received a B")
elif FinalGrade >= 70 and FinalGrade < 80:
return("You received a C")
elif FinalGrade >= 60 and FinalGrade < 70:
return("You received a D")
else:
return("Sorry, you received an F")
I also have this line of code which is to read the CSV file, and displays in the output window.
with open("studentGradeFrom.csv") as csvfile:
readFile = csv.reader(csvfile, delimiter=",", quotechar="¦")
for row in readFile:
print(row)
However since I am new to Python, I am looking for help to create a python script, that will look at the results and do a calculation which will tell me if the student has passed or failed.
I would like this to be done in a separate file. So I am guessing that I will need to read and write to a different CSV file, to show if a student has failed or has an overall passing percentage.
with open("studentGradeTo.csv", 'w') as avg: #used to write to after the calculation is complete
loadeddata = open("studentGradeFrom.csv", 'r') #used to read the data from the CSV before calculation.
writer=csv.writer(avg)
readloaded=csv.reader(loadeddata)
listloaded=list(readloaded)
Now my question: How would I go about doing this, from looking at data from one file which roughly 50 different students. While not changing the read CSV with the student grades, and only changing the CSV file which shows passing or failing grades. Any help would be appreciated.
Edit: I forgot to mention that the first test would be work 20% of the final grade, the same with the second test and third test. these three totalling to 60% of final grade. While the fourth test is worth 40% of the final grade.

Here is a quick example of the concepts using only the csv library (you could certainly optimize a lot of this, but it should work for the example).
import csv
student_grades = []
# First open up your file containing the raw student grades
with open("studentGradeFrom.csv", "r") as file:
# Setup your reader with whatever your settings actually are
csv_file = csv.DictReader(file, delimiter=",", quotechar='"')
# Cycle through each row of the csv file
for row in csv_file:
# Calculate the numerical grade of the student
grade = grader(
int(row["test1"]),
int(row["test2"]),
int(row["test3"]),
int(row["test4"])
)
# Calculate the letter score for the student
score = gradeScores(grade)
# Assemble all the data into a dictionary
# Only need to save fields you need in the final output
student_grades.append({
"first_name": row["first_name"],
"last_name": row["last_name"],
"test1": row["test1"],
"test2": row["test2"],
"test3": row["test3"],
"test4": row["test4"],
"grade": grade,
"score": score
})
# Open up a new csv file to save all the grades
with open("studentGradeFrom.csv", "w", newline="") as file:
# List of column names to use as a header for the file
# These will be used to match the dictionary keys set above
# Only need to list the fields you saved above
column_names = [
"first_name", "last_name", "test1", "test2", "test3",
"test4", "grade", "score"
]
# Create the csv writer, using the above headers
csv_file = csv.DictWriter(file, column_names)
# Write the header
csv_file.writeheader()
# Write each dictionary to the csv file
for student in student_grades:
csv_file.writerow(student)
You would need to fine tune this to your exact requirements, but it will hopefully get you going in the right direction. Most of this is documented in the official documentation if you need a specific reference: https://docs.python.org/3.6/library/csv.html.

This kind of task is suited for the pandas library.
Here is one solution, which is adaptable should your requirements change.
import pandas as pd
df = pd.read_csv('studentGradeFrom.csv')
# first_Name last_Name test1 test2 test3 test4
# 0 Alex Brian 11 17 13 24
# 1 Pete Tong 19 14 12 30
boundaries = {(90, 100.01): 'A',
(80, 90): 'B',
(70, 80): 'C',
(60, 70): 'D',
(0, 60): 'F'}
def grade_calc(x, b):
return next((v for k, v in b.items() if k[0] <= x <= k[1]), None)
df['FinalMark'] = 0.2*df['test1'] + 0.2*df['test2'] + 0.2*df['test3'] + 0.4*df['test4']
df['FinalGrade'] = df['FinalMark'].apply(grade_calc, b=boundaries)
# first_Name last_Name test1 test2 test3 test4 FinalMark FinalGrade
# 0 Alex Brian 11 17 13 24 17.8 F
# 1 Pete Tong 19 14 12 30 21.0 F
df.to_csv('studentGradeTo.csv', index=False)

how can i extract elements from lists in python

I am trying to extract elements from list.
I've looked up a lot of data, but I do not know..
this is my test.txt (text file)
[ left in the table = time, right in the table = value ]
0 81
1 78
2 76
3 74
4 81
5 79
6 80
7 81
8 83
9 83
10 83
11 82
.
.
22 81
23 80
If the current time is equal to the time in the table, i want to extract the value of that time.
this is my demo.py (python file)
import datetime
now = datetime.datetime.now())
current_hour = now.hour
with open('test.txt') as f:
lines = f.readlines()
time = [int(line.split()[0]) for line in lines]
value = [int(line.split()[1]) for line in lines]
>>>time = [0,1,2,3,4,5,....,23]
>>>value = [81,78,76,......,80]

You could make a loop where you iterate over the list, looking for the current hour at every position on the list.
Starting at position 0, it will compare it with the current hour. If it's the same value, it will assign the value at the position it was found in "time" to the variable extractedValue, then it will break the loop.
If it isn't the same value, it will increase by 1 the pos variable, which we use to look into the list. So it will keep searching until the first if is True or the list ends.
pos=0
for i in time:
if(current_hour==time[pos]):
extractedValue=value[pos]
break
else:
pos+=1
pass
Feel free to ask if you don't understand something :)

Assuming unique values for the time column:
import datetime
with open('text.txt') as f:
lines = f.readlines()
#this will create a dictionary with time value from test.txt as the key
time_data_dict = { l.split(' ')[0] : l.split(' ')[1] for l in lines }
current_hour = datetime.now().hour
print(time_data_dict[current_hour])

import datetime
import csv
data = {}
with open('hour.csv', newline='') as f:
reader = csv.reader(f)
for row in reader:
k, v = row
data[k] = v
hour = str(datetime.datetime.now().hour)
print(data[str(hour)])

How to parse csv file and compute stats based on that data

I have a task which requires me to make a program in python that reads a text file which has information about people (name, weight and height).
Then I need the program to ask for the user to enter a name then look for that name in the text file and print out the line which includes that name and the person's height and weight.
Then the program has to work out the average weight of the people and average height.
The text file is:
James,73,1.82,M
Peter,78,1.80,M
Jay,90,1.90,M
Beth,65,1.53.F
Mags,66,1.50,F
Joy,62,1.34,F
So far I have this code which prints out the line using the name that has been typed by the user but I don't know to assign the heights and the weights:
search = input("Who's information would you like to find?")
with open("HeightAndWeight.txt", "r") as f:
for line in f:
if search in line:
print(line)

Using the pandas library as suggested, you can do as follows:
import pandas as pd
df = pd.read_csv('people.txt', header=None, index_col=0)
df.columns = ['weight', 'height', 'sex']
print(df)
weight height sex
0
James 73 1.82 M
Peter 78 1.80 M
Jay 90 1.90 M
Beth 65 1.53 F
Mags 66 1.50 F
Joy 62 1.34 F
print(df.mean())
weight 72.333333
height 1.648333

You could use Python's built in csv module to split each line in the file into a list of columns as follows:
import csv
with open('HeightAndWeight.txt', 'rb') as f_input:
csv_input = csv.reader(f_input)
total_weight = 0
total_height = 0
for index, row in enumerate(csv_input, start=1):
total_weight += float(row[1])
total_height += float(row[2])
print "Average weight: {:.2f}".format(total_weight / index)
print "Average height: {:.2f}".format(total_height / index)
This would display the following output:
Average weight: 72.33
Average height: 1.65

The answer is actually in your question's title : use the standard lib's csv module to parse your file

Use:
splitted_line = line.split(',', 4)
to split the line you just found into four parts, using the comma , as a delimiter. You then can get the first part (the name) with splitted_line[0], the second part (age) with splitted_line[1] and so on. So, to print out the persons name, height and weight:
print('The person %s is %s years old and %s meters tall.' % (splitted_line[0], splitted_line[1], splitted_line[2]))
To get the average on height and age, you need to know how many entries are in your file, and then just add up age and height and divide it by the number of entries/persons. The whole thing would look like:
search = input("Who's information would you like to find?")
total = 0
age = 0
height = 0
with open("HeightAndWeight.txt", "r") as f:
for line in f:
total += 1
splitted_line = line.split(',', 4)
age += int(splitted_line[1])
height += int(splitted_line[2])
if search in line:
print('The person %s is %s years old and %s meters tall.' % (splitted_line[0], splitted_line[1], splitted_line[2]))
average_age = age / total
average_height = height / total
That's one straightforward way to do it, and hopefully easy to understand as well.

search = input("Who's information would you like to find?")
heights = []
weights = []
with open("HeightAndWeight.txt", "r") as f:
for line in f:
if search in line:
print(line)
heights.append(int(line.split(',')[2]))
weights.append(int(line.split(',')[1]))
# your calculation stuff

how do I match a specific number into number set efficiently?

I have a number set which contains 2375013 unique numbers in txt file. The data structure looks like this:
11009
900221
2
3
4930568
293
102
I want to match a number in a line from another data to the number set for extracting data what I need. So, I coded like this:
6 def get_US_users_IDs(filepath, mode):
7 IDs = []
8 with open(filepath, mode) as f:
9 for line in f:
10 sp = line.strip()
11 for id in sp:
12 IDs.append(id.lower())
13 return IDs
75 IDs = "|".join(get_US_users_IDs('/nas/USAuserlist.txt', 'r'))
76 matcher = re.compile(IDs)
77 if matcher.match(user_id):
78 number_of_US_user += 1
79 text = tweet.split('\t')[3]
But it takes a lot of time for running. Is there any idea to reduce run time?

What I understood is that you have a huge number of ids in a file and you want to know if a specific user_id is in this file.
You can use a python set.
fd = open(filepath, mode);
IDs = set(int(id) for id in fd)
...
if user_id in IDs:
number_of_US_user += 1
...