This question already has answers here:
Why can't I iterate twice over the same iterator? How can I "reset" the iterator or reuse the data?
(5 answers)
Proper way to reset csv.reader for multiple iterations?
(3 answers)
Closed 3 years ago.
Super noob here, so this might be a little embarrasing.
I need to work with a csv file, and found that you can use csv.DictReader to make a list of ordered dicts. So far so good.
I can loop through the list and do stuff, but only one time.
If I want to print the dicts 2 times, it doesn´t work.
import csv
csv_file = open('untitled2.csv', mode='r')
csv_reader = csv.DictReader(csv_file, delimiter = ";")
for rows in csv_reader:
print (rows)
for rows in csv_reader:
print (rows)
This only prints the list of dicts 1 time. I need to go through the list a number of times. but I´m not able to do that.
You need to go to the begining of the file again :
csv_file.seek(0) after the first for.
Don't forget to close it when you're done.
The best way to do it is in a context :
with open('untitiled2.csv', mode='r') as csv_file:
csv_reader = csv.DictReader(csv_file, delimiter=';')
# your for
Related
This question already has answers here:
Iterating on a file doesn't work the second time [duplicate]
(4 answers)
Closed 1 year ago.
I was refactoring some code for my program and I have a mistake somewhere in the process. I am reading and writing .csv files.
In the beginning of my program I iterate through a .csv file in order to find which data from the file I need.
with open(csvPath, mode='r') as inputFile:
csvReader = csv.reader(inputFile)
potentialVals = []
paramVals = {}
for row in csvReader:
if row[3] == "Parameter":
continue
# Increment vales in dict
if row[3] not in paramVals:
paramVals[row[3]] = 1
else:
paramVals[row[3]] += 1
This iterates and works fine, the for loop gets me every row in the .csv file. I them perform some calculations and go to iterate through the same .csv file again later, and then select data to write to a new .csv file. My problem is here, when I go to iterate through a second time, it only gives me the first row of the .csv file, and nothing else.
# Write all of the information to our new csv file
with open(outputPath, mode='w') as outputFile:
csvWriter = csv.writer(outputFile, delimiter=',', quotechar='"', quoting=csv.QUOTE_ALL)
inputFile.seek(0)
rowNum = 0
for row in csvReader:
print(row)
Where the print statement is, it only prints the first line of the .csv file, and then exits the for loop. I'm not really sure what is causing this. I thought it might have been the
inputFile.seek(0)
But even if I opened a 2nd reader, the problem persisted. This for loop was working before I refactored it, all the other code is the same except the for loop I'm having trouble with, here is what it used to look like:
Edit: So I thought maybe it was a variable instance error, so I tried renaming my variables instead of reusing them and the issue persisted. Going to try a new file instance now,
Edit 2: Okay so this is interesting, when I look at the line_num value for my reader object (when I open a new one instead of using .seek) it does output 1, so I am at the beginning of my file. And when I look at the len(list(csvReader)) it is 229703, which shows that the .csv is fully there, so still not sure why it won't do anything besides the first row of the .csv
Edit 3: Just as a hail mary attempt, I tried creating a deep copy of the .csv file and iterating through that, but same results. I also tried just doing an entire separate .csv file and I also got the same issue of only getting 1 row. I guess that eliminates that it's a file issue, the information is there but there is something preventing it from reading it.
Edit 4: Here is where I'm currently at with the same issue. I might just have to rewrite this method completely haha but I'm going to lunch so I won't be able to actively respond now. Thank you for the help so far though!
# TODO: BUG HERE
with open(csvPath, mode='r') as inputFile2:
csvReader2 = csv.reader(inputFile2)
...
for row2 in csvReader2:
print("CSV Line Num: " + str(csvReader2.line_num))
print("CSV Index: " + str(rowNum))
print("CSV Length: " + str(len(list(csvReader2))))
print("CSV Row: " + str(row2))
Also incase it helps, here is csvPath:
nameOfInput = input("Please enter the file you'd like to convert: ")
csvPath = os.path.dirname(os.path.realpath(nameOfInput))
csvPath = os.path.join(csvPath, nameOfInput)
If you read the documentation carefully, it says csv reader is just a parser and all the heavy lifting is done by the underlying file object.
In your case, you are trying to read from a closed file in the second iteration and that is why it isn't working.
For csv reader to work you'll need an underlying object which supports the iterator protocol and returns a string each time its next() method is called — file objects and list objects are both suitable.
Link to the documentation: https://docs.python.org/3/library/csv.html
This question already has answers here:
How to append a new row to an old CSV file in Python?
(8 answers)
Closed 1 year ago.
I'm using selenium and beautifulsoup to iterate through a number of webpages and sort out the results. I have that working, however I want to export the results to a CSV using this block of code:
with open('finallist.csv', mode='w') as final_list:
stock_writer = csv.writer(final_list, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
stock_writer.writerow([ticker, element.get_text()])
The only issue is, with the result being multiple different things, this code as it stands just replaces the first line of the CSV every time a new result comes in. Is there any way I can have it write to a new line each time?
Per the Python documentation for the open() function, you can pass the 'a' mode to the open() function. Doing so will append any text to the end of the file, if the file already exists.
with open('finallist.csv', mode='a') as final_list:
...
This question already has answers here:
Skip first couple of lines while reading lines in Python file
(9 answers)
Closed 3 years ago.
I have somes csvs where the fields names are on the third line.
The two first line are comment and the true csv start at the third line.
What is the best way to use csv.DictReader to start at the third line not the first line ?
Regards
you can call next(f) twice (where f is the file handler) and then pass it to csv.DcitReader() constructor, e.g:
import csv
with open('so.csv') as f:
next(f)
next(f)
dict_rdr = csv.DictReader(f)
for line in dict_rdr:
print(line)
This question already has answers here:
Integers from excel files become floats?
(6 answers)
Closed 4 years ago.
I wrote some code and trying to refactor it to cut out a few steps and i cant seem to find an answer for this. Im reading an excel file and doing a bunch of column renaming and dropping columns i dont need. My end goal is to write the Excel file as an text tab delimited file and accomplished all this but in a very hacky way. I have a function convertToText() that reads the excel file and turns it into a txt file. However every single integer in the file gets a .0 appended to the end.
Ex.
Excel value 1234321
Txt File = 1234321.0
Im just doing a simple read and write using pandas, openpyxl and xlrd.
def convertToText():
with open(os.path.join(outFile, 'target2.txt'), 'wb') as myTxtfile:
wr = csv.writer(myTxtfile, delimiter="\t")
myfile = xlrd.open_workbook(outFile + fileName)
mysheet = myfile.sheet_by_index(0)
for rownum in xrange(mysheet.nrows):
wr.writerow(mysheet.row_values(rownum))
I had to write a second function just to do a find and replace on the .0 and trying to cut that step out of the process. If anyone has any ideas how to do this in the above function would be greatly appreciated!!
Is this the same thing as what you mean?
So I think your code should become, although I have no test data so I cannot try it:
def convertToText():
with open(os.path.join(outFile, 'target2.txt'), 'wb') as myTxtfile:
wr = csv.writer(myTxtfile, delimiter="\t")
myfile = xlrd.open_workbook(outFile + fileName)
mysheet = myfile.sheet_by_index(0)
for rownum in xrange(mysheet.nrows):
wr.writerow([int(i) for i in mysheet.row_values(rownum)])
This question already has answers here:
How to read specific lines from a file (by line number)?
(30 answers)
Closed 8 years ago.
I just found out it's not possible to write to a specific line in a csv file (only the end).
I have just come across another obstacle that I'm having trouble tackling, which is reading from a specific line in a csv file.
One way I have found to accomplish this is:
with open('file.csv',newline = '') as csvfile:
spamreader = csv.reader(csvfile,delimiter=',',quotechar = '"')
lines = []
for row in spamreader:
lines.append(row)
print('What line do you want to read from?')
line = lines[int(input())-1] #I think the -1 is right. since lists start at 0
However, I believe that this might be a slightly inefficient way to do this, since the more rows in the list "lines", the more RAM the program would be using.
Could someone tell me if this is actually an efficient way of doing this? Otherwise, I will just go with this.
Is there any way that I can do something like this?
spamreader.readRow(5) #I just made this up, but is there a similar function?
This is the page that I've been using, it's possible I skipped over it. https://docs.python.org/3/library/csv.html
Also, I'm not very advanced in programming, so if there is an advanced answer, can you try to keep the explanations fairly simple?
If you want to read starting from line 123:
for _ in range(122):
spamreader.next()
for row in spamreader:
...
With Python 3 it seems to be
next(spamreader)
One can also navigate in the file by moving the cursor to a specific byte using find and seek.