I scrape a website and try to store it into Csv format but when i did it is store only single row of data.
how to write multiples Row of data in csv.
for lis in lists:
title = lis.find('a', class_="title").text
tag = lis.find('span', class_="etc-mark").text
datetime = lis.find('span', class_="datetime").text
address = lis.find('div', class_="middle-xs").text
img = lis.find('span', class_="https://thecryptobasic.com")
data = [title, tag, datetime,address,img]
print(data)
# create the csv writer
writer = csv.writer(f)
# write a row to the csv file
writer.writerow(header)
writer.writerow(data)
# close the file
f.close()
data gets overwritten in every iteration, so after the loop it contains the data from the last iteration only.
Initialize data before the loop, the append to it in every iteration.
Use writerows instead of writerow.
data = []
for lis in lists:
title = lis.find('a', class_="title").text
tag = lis.find('span', class_="etc-mark").text
datetime = lis.find('span', class_="datetime").text
address = lis.find('div', class_="middle-xs").text
img = lis.find('span', class_="https://thecryptobasic.com")
data.append([title, tag, datetime, address, img])
...
writer.writerows(data)
while refactoring this piece of code, it will also better to let with manage the opening and closing of the file:
...
with open(..., 'w') as f:
writer = csv.writer(f)
writer.writerow(header)
writer.writerows(data)
We can even merge the write calls by pre-initializing data with header:
data = [header]
for lis in lists:
title = lis.find('a', class_="title").text
tag = lis.find('span', class_="etc-mark").text
datetime = lis.find('span', class_="datetime").text
address = lis.find('div', class_="middle-xs").text
img = lis.find('span', class_="https://thecryptobasic.com")
data.append([title, tag, datetime, address, img])
with open(..., 'w') as f:
writer = csv.writer(f)
writer.writerows(data)
Related
I'm attempting to convert yelps data set that is in JSON to a csv format. The new csv file that is created is empty.
I've tried different ways to iterate through the JSON but they all give me a zero bytes file.
The json file looks like this:
{"business_id":"1SWheh84yJXfytovILXOAQ","name":"Arizona Biltmore Golf Club","address":"2818 E Camino Acequia Drive","city":"Phoenix","state":"AZ","postal_code":"85016","latitude":33.5221425,"longitude":-112.0184807,"stars":3.0,"review_count":5,"is_open":0,"attributes":{"GoodForKids":"False"},"categories":"Golf, Active Life","hours":null}
import json
import csv
infile = open("business.json","r")
outfile = open("business2.csv","w")
data = json.load(infile)
infile.close()
out = csv.writer(outfile)
out.writerow(data[0].keys())
for row in data:
out.writerow(row.values())
I get an "extra data" message when the code runs. The new business2 csv file is empty and the size is zero bytes.
if you JSON has only one row.. then try this
infile = open("business.json","r")
outfile = open("business2.csv","w")
data = json.load(infile)
infile.close()
out = csv.writer(outfile)
#print(data.keys())
out.writerow(data.keys())
out.writerow(data.values())
Hi Please try the below code, by using with command the file access will automatically get closed when the control moves out of scope of with
infile = open("business.json","r")
outfile = open("business2.csv","w")
data = json.load(infile)
infile.close()
headers = list(data.keys())
values = list(data.values())
with open("business2.csv","w") as outfile:
out = csv.writer(outfile)
out.writerow(headers)
out.writerow(values)
You need to use with to close file.
import json
import csv
infile = open("business.json","r")
data = json.load(infile)
infile.close()
with open("business2.csv","w") as outfile:
out = csv.writer(outfile)
out.writerow(list(data.keys()))
out.writerow(list(data.values()))
i want to write the result of for loop which is PMID = Id of litrature ,Date = date of publication ,title = title of article,Abstract = abtract of artilce in csv file but it is saving only one element of the output no all
import numpy as np
from Bio import Entrez
from Bio import Medline
import csv
import pandas as pd
Entrez.email = "shayezkarimcide#gmail.com"
handle = Entrez.esearch(db="pmc",
term = "Antimicrobial resistance Drug Resistance",
rettype = "medline",retmode = "txt",
retmax= "200",sort = "pub date")
result = Entrez.read(handle)
handle.close()
Id = result ['IdList']
print (Id)
handle2 = Entrez.efetch(db="pmc",
id=Id, rettype="medline",
retmode="text")
records = Medline.parse(handle2)
header = ['ID','Date','Title','Abstract']
for result in records :
PMID = result['PMID']
Abstract = result['AB']
title = result['TI']
Date = result['DP']
print (PMID,Date,title,Abstract)
fields = [PMID, title,Date,Abstract]
rows = [PMID,Date,title,Abstract]
with open ('/home/shayez/Desktop/karim.csv','wt') as csvfile:
writer = csv.writer(csvfile, delimiter ="\t" )
writer.writerow(header)
writer.writerow(rows)
handle2.close()
You are opening the file, writing and closing it inside the loop (the with makes sure the file is closed after the with's scope is done) so it is replacing the previous file for each element in the loop.
Try opening the file only once, before the loop:
with open ('/home/shayez/Desktop/karim.csv','wt') as csvfile:
writer = csv.writer(csvfile, delimiter ="\t" )
writer.writerow(header)
for result in records :
PMID = result['PMID']
Abstract = result['AB']
title = result['TI']
Date = result['DP']
print (PMID,Date,title,Abstract)
fields = [PMID, title,Date,Abstract]
rows = [PMID,Date,title,Abstract]
writer.writerow(rows)
I have a Problem with continues writing my datas in a csv-file. I want a program that detects, if there is a csv-file for my measurements-data. If not it would be generated. When the csv-file is new generated the datas are written in the csv-file on the column after the header with the variable cycle = 0.
If the csv-file exists, the datas should be written continuously after the last line of the csv. Also the variable cycle should continue.
I have written a program that can detect if there is a file or not but with the continuously lines I have problems.
I hope someone can help me.
# mes = Array with 20 spaces filled with the Numbers 0-19
date = time.strftime("%d/%m/%Y")
def write(cycle, mes):
if os.path.exists('/home/pi/Documents/Ventilatorprüfstand_Programm/out.csv') is True: #does the out.csv existate?
print("Do something")
out = open('out.csv', 'w')
data = [[cycle, mes[0],mes[1],mes[2],mes[3],mes[4],mes[5],mes[6],mes[7],mes[8],mes[9],mes[10],mes[11],mes[12],mes[13],mes[14],mes[15],mes[16],mes[17],mes[18],mes[19], date]]
line = cycle+1
for row in data:
for line in row:
out.write('%s;' % line)
out.write('\n')
out.close()
else:
print("Do another something")
header = lookuptable.names()
out = open('out.csv', 'w')
for row in header:
for column in row:
out.write('%s' % column)
out.write('\t')
out.write('\n')
data = [[cycle, mes[0],mes[1],mes[2],mes[3],mes[4],mes[5],mes[6],mes[7],mes[8],mes[9],mes[10],mes[11],mes[12],mes[13],mes[14],mes[15],mes[16],mes[17],mes[18],mes[19], date]]
for row in data:
for column in row:
out.write('%s;' % column)
out.write('\n')
out.close()`
When opening the file with open() there is the option 'a' to append the new lines to the end:
'a' open for writing, appending to the end of the file if it exists
Here is an example using the csv Python standard library:
import csv
import os
import random
headers = ['cycle', 'date', 'speed', 'temp', 'power']
new_data = [[random.randint(0, 100) for _ in range(3)] for _ in range(2)]
date = '00/01/02'
cycle = 1
# Copy the data and include the date and the cycle number:
full_rows = [ [cycle, date, *row] for row in new_data ]
filename = 'example.csv'
# Check if the file exist, if not create the file with header
if not os.path.exists(filename):
print('creating a new file')
with open(filename, 'w') as csvfile:
csvwriter = csv.writer(csvfile, delimiter=',')
csvwriter.writerow(headers) # add the header
# Append the data to the file
with open(filename, 'a', newline='') as csvfile: # note the 'a' option
csvwriter = csv.writer(csvfile, delimiter=',')
csvwriter.writerows(full_rows)
CSV writing cannot be done.The list of "li_result" has data result and I want to write this data in csv file.
This is the code
fp = open('dataResult.csv', 'w')
w = csv.writer(fp, delimiter=',')
csvwrite = unicode(li_result)
csvwrite_result = csvwrite.encode('sjis')
w.writerow(csvwrite_result)
But dataResult.csv is empty. Nothing error happen so I do not know what is wrong.
And I want to write the data in sjis code in csv file.(Now I use python2.7 so unicode is used to write letters,right?) I deleted these codes
csvwrite = unicode(li_result)
csvwrite_result = csvwrite.encode('sjis')
Still nothing is written.
What should I do to fix this?
Sample codes
fp = open(CSV_FILE_NAME_ACCOUNT, 'aw')
w = csv.writer(fp, delimiter=',')
title = 'abc'
name = 'hoge'
time = '2010-04-20 0:0:0'
u_title = unicode(title)
u_name = unicode(name)
u_time = unicode(time)
s_title = u_title.encode('sjis')
s_name = u_name.encode('sjis')
s_time = u_time.encode('sjis')
list = [s_title, s_name, s_time]
w.writerow(list)
import csv
fp = open('system path to your file on which data to read', 'w')
w = csv.writer(fp, delimiter=',')
title = 'abc'
name = 'hoge'
time = '2010-04-20 0:0:0'
list = [title, name, time]
w.writerow(list)
I am trying to read an excel file, extract some data, and write it out as a csv. This is pretty new to me and I'm messing up somewhere: I keep getting an empty csv. I'm sure I'm missing something very basic, but darned if I can see it. Here is the code:
```
import xlrd
import os
import csv
from zipfile import ZipFile
import datetime
datafile = "./2013_ERCOT_Hourly_Load_Data.xls"
outfile = "./2013_Max_Loads.csv"
def parse_file(datafile):
workbook = xlrd.open_workbook(datafile)
sheet = workbook.sheet_by_index(0)
data = None
outputlist = []
for col in range(1, sheet.ncols):
cv = sheet.col_values(col, start_rowx=1, end_rowx=None)
header = sheet.cell_value(0,col)
maxval = max(cv)
maxpos = cv.index(maxval) + 1
maxtime = sheet.cell_value(maxpos, 0)
realtime = xlrd.xldate_as_tuple(maxtime, 0)
year = realtime[0]
month = realtime[1]
day = realtime[2]
hour = realtime[3]
data = [
'Region:', header,
'Year:', year,
'Month:', month,
'Day:', day,
'Hour:', hour,
maxpos,
maxtime,
realtime,
maxval,
]
path = "./2013_Max_Loads.csv"
return outputlist
def save_file(data, filename):
with open(filename, "wb") as f:
writer = csv.writer(f, delimiter='|')
for line in data:
writer.writerow(line)
parse_file(datafile)
save_file(parse_file(datafile),"2013_Max_Loads.csv")
You declare outfile but you don't use it
You aren't passing a directory (path) for the file to be saved in.
I also think that calling parse_file twice might be messing you up. Just pass the filename and call it from within the save_file function.
I also found that you were returning output list as a blank list.
So here, try this. I will assume your xlrd commands are correct, because I have not personally used the module.
import csv
import xlrd
def parse_file(datafile):
workbook = xlrd.open_workbook(datafile)
sheet = workbook.sheet_by_index(0)
outputlist = []
outputlist_append = outputlist.append
for col in range(1, sheet.ncols):
cv = sheet.col_values(col, start_rowx=1, end_rowx=None)
header = sheet.cell_value(0,col)
maxval = max(cv)
maxpos = cv.index(maxval) + 1
maxtime = sheet.cell_value(maxpos, 0)
realtime = xlrd.xldate_as_tuple(maxtime, 0)
year = realtime[0]
month = realtime[1]
day = realtime[2]
hour = realtime[3]
data = [
'Region:', header,
'Year:', year,
'Month:', month,
'Day:', day,
'Hour:', hour,
maxpos,
maxtime,
realtime,
maxval,
]
outputlist_append(data)
return outputlist
def save_file(data, filename):
parse_file(data)
with open(filename, 'wb') as f:
writer = csv.writer(f, delimiter='|')
for line in data:
writer.writerow(line)
return
datafile = "./2013_ERCOT_Hourly_Load_Data.xls"
outfile = "./2013_Max_Loads.csv"
save_file(datafile, outfile)
UPDATE: Edit in code in function save_file() to implement #wwii's suggestion.
Try substituting the new save_file() below:
def save_file(data, filename):
parse_file(data)
with open(filename, 'wb') as f:
wr = csv.writer(f, delimiter='|')
wr.writerows(data)
return
Also, change the variable (you used writer) to something like wr. You really want to avoid any possible conflicts with having a variable with the same name as a method, a function, or class you are calling.