Develop Reference

How to iterate through all rows and save the data

I am working on a Python script to do some steps.
Right now everything is working except the part that it has to go through all table rows and save each line as a JSON object. The problem is: It only saves the last line. So it's not saving the previous lines. I know where is the problem but don't know how to fix.
Here is the code:
url = 'http://website.com/group1.html'
htmlFile= urlopen(urlGroup1)
soup= BeautifulSoup(htmlGroup1, 'html.parser')
table = soup2.find_all("table", {"class": "sortable employeeList result-table"})[0]
rows = table.find_all('tr')
Filegroup1 = open('group1.csv', 'wt+')
Datagroup1 = csv.writer(Filegroup1)
jsonFilePath = r'group1.json'
def make_json(Filegroup1, jsonFilePath):
data2 = {}
with open('group1.csv', encoding='utf-8') as csvf:
csvReader = csv.DictReader(csvf)
for rows in csvReader:
data2 = rows
with open(jsonFilePath, 'w', encoding='utf-8') as jsonf:
jsonf.write(simplejson.dumps([data2], indent=4))
try:
for row in rows:
FilteredRow = []
for cell in row.find_all(['td', 'th']):
FilteredRow.append(cell.get_text().strip())
Datagroup1.writerow(FilteredRow)
finally:
Filegroup1.close()
make_json(Filegroup1, jsonFilePath)
The issue is here:
for rows in csvReader:
data2 = rows
If I change it to the following, it will work!! But it will group each object by Employee ID. Which I don't want that.
for rows in csvReader:
key = rows['Employee Name']
data3[key] = rows

Why Python don't save csv results

I don't understand why Python don't save results correctly, while it prints it correct. Code look like this:
import csv
with open("dataset_1.csv", "r") as WBI:
data = csv.reader(WBI, delimiter = ";")
data = list(data)
header = data[0]
data = data[1:]
WaterBandIndex = []
for row in data:
WaterBandIndex.append(float(row[54])/float(row[83]))
print (WaterBandIndex)
with open("WBI.csv", "w+") as WBI:
csvwriter = csv.writer(WaterBandIndex, delimiter = "|", lineterminator = "\n")
csvwriter.writerows(WaterBandIndex)
Printed results are correct, but saves to csv nothing.
I'm green in programming.

The code should work if your variable WaterBandIndex is not empty.
import csv
WaterBandIndex = ['1','2','3']
with open("WBI.csv", "w") as f:
csvwriter = csv.writer(f , delimiter = "|", lineterminator = "\n")
csvwriter.writerows(WaterBandIndex)

Python CSV Row Loop

I am very new to Python programming and decided on a small project to learn the language.
Basically I am trying to:
Read the first cell of a CSV file.
Ask if that cell value is "liked".
If liked, write to the column next to the cell on 1., "1".
Else, write "0".
Repeat on next row until end of list.
My code right now:
import csv
reader = csv.reader(open("mylist.csv"), delimiter=',')
data = []
for row in reader:
data.append(row)
ask = (data[0][0])
ans = input("Do you like {}? ".format(ask))
if ans == ("y"):
f = open('mylist.csv', 'r')
reader = csv.reader(f)
data = list(reader)
f.close()
data[0][1] = '1'
my_new_list = open('mylist.csv', 'w', newline='')
csv_writer = csv.writer(my_new_list)
csv_writer.writerows(data)
my_new_list.close()
else:
f = open('mylist.csv', 'r')
reader = csv.reader(f)
data = list(reader)
f.close()
data[0][1] = '0'
my_new_list = open('mylist.csv', 'w', newline='')
csv_writer = csv.writer(my_new_list)
csv_writer.writerows(data)
my_new_list.close()
So basically, I am stuck trying to get the content of the next row.
FYI, I am looking to implement machine learning to this process.
First learning how to do this in a basic manner.
Any help is welcome.
Thank you!

You shouldn't read from and write to the same file/list/dict at the same time. If you do, references to data may change. You can start with something like this for your task. However, note that as the file grows you code becomes slower.
import csv
reader = csv.reader(open("test.csv", 'r'), delimiter=',')
content = []
for row in reader:
item = row[0]
ans = raw_input("Do you like {}? ".format(item))
if ans == 'y':
content.append([item, 1])
else:
content.append([item, 0])
writer = csv.writer(open('test.csv', 'w'))
writer.writerows(content)

In my last work with csv I opened the file so:
import csv
with open(name) as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
data.append(row)

If you want the resultant csv file to contain all of the data from the input file but with the question results added in, you could use something like this.
It will insert you answer (0 or 1) after the first item in each record.
import csv
reader = csv.reader(open("mylist.csv", 'r'), delimiter=',')
data = []
for row in reader:
data.append(row)
for row in data:
ans = raw_input("Do you like {}? ".format(row[0]))
if ans == 'y':
row[1:1] = "1"
else:
row[1:1] = "0"
writer = csv.writer(open('myresult.csv', 'w'))
writer.writerows(data)

CSV writing cannot be done

CSV writing cannot be done.The list of "li_result" has data result and I want to write this data in csv file.
This is the code
fp = open('dataResult.csv', 'w')
w = csv.writer(fp, delimiter=',')
csvwrite = unicode(li_result)
csvwrite_result = csvwrite.encode('sjis')
w.writerow(csvwrite_result)
But dataResult.csv is empty. Nothing error happen so I do not know what is wrong.
And I want to write the data in sjis code in csv file.(Now I use python2.7 so unicode is used to write letters,right?) I deleted these codes
csvwrite = unicode(li_result)
csvwrite_result = csvwrite.encode('sjis')
Still nothing is written.
What should I do to fix this?
Sample codes
fp = open(CSV_FILE_NAME_ACCOUNT, 'aw')
w = csv.writer(fp, delimiter=',')
title = 'abc'
name = 'hoge'
time = '2010-04-20 0:0:0'
u_title = unicode(title)
u_name = unicode(name)
u_time = unicode(time)
s_title = u_title.encode('sjis')
s_name = u_name.encode('sjis')
s_time = u_time.encode('sjis')
list = [s_title, s_name, s_time]
w.writerow(list)

import csv
fp = open('system path to your file on which data to read', 'w')
w = csv.writer(fp, delimiter=',')
title = 'abc'
name = 'hoge'
time = '2010-04-20 0:0:0'
list = [title, name, time]
w.writerow(list)

JSON like data to CSV file in python - not showing headers correctly

I am transforming JSON like data to CSV and having a few issues.
The code is here:
import json
import csv
def parse_file(inputed_file):
with open(input_file, 'r') as inputed_file:
content = inputed_file.readlines()
split_file = open('test.csv', 'w')
for line in content:
lines = line.split('\t')
data = json.loads(lines[0])
writer = csv.DictWriter(split_file, fieldnames = ["title", "firstname"], delimiter = ',')
writer.writeheader()
The problem is this is adding a header on each row for the data, I want to only have the header displayed once. Then add this for the data to go below the headers:
writer.writerow(data)
I have looked at this and tried it but failed: How can I convert JSON to CSV?.

Create the DictWriter outside the loop, and just call writer.writeheader() there. Then call writer.writerow() inside the loop.
def parse_file(inputed_file):
with open(input_file, 'r') as inputed_file:
content = inputed_file.readlines()
split_file = open('test.csv', 'w')
writer = csv.DictWriter(split_file, fieldnames = ["title", "firstname"], delimiter = ',')
writer.writeheader()
for line in content:
lines = line.split('\t')
data = json.loads(lines[0])
writer.writerow(data)