For a CSV file:
a,b,c,d
1,2,3,4
5,6,7,8
9,10,11,12
While the code below works fine to output the rows of the CSV:
import csv
import sys
database = {}
with open(sys.argv[1], mode='r') as csv_file:
database = csv.DictReader(csv_file)
for row in database:
print(row)
the following does not.
import csv
import sys
database = {}
with open(sys.argv[1], mode='r') as csv_file:
database = csv.DictReader(csv_file)
for row in database:
print(row)
with error
> Traceback (most recent call last): File "test.py", line 9, in
> <module>
> for row in database: File "/usr/local/lib/python3.7/csv.py", line 111, in __next__
> self.fieldnames File "/usr/local/lib/python3.7/csv.py", line 98, in fieldnames
> self._fieldnames = next(self.reader) ValueError: I/O operation on closed file.
The csv.DictReader object appears to exist but I cannot iterate over it in the 2nd snippet.
Checking various comments, they seem to say that DictReader returns an iterator - but I do not know understand if this is the reason for the error and what to change to gain access to database.
Appreciate any help. Thanks in advance!
with open is a context manager which closes the file when execution goes out of scope. As the file is closed you can't read from it.
Use the original indentation.
import csv
import sys
database = {}
with open(sys.argv[1], mode='r') as csv_file:
database = csv.DictReader(csv_file)
for row in database:
print(row)
You could also do the following:
import csv
import sys
with open(sys.argv[1], mode='r') as csv_file:
rows = list(csv.DictReader(csv_file))
for row in rows:
print(row)
The second way will pull all the data into memory.
Related
I want to open a variable number of csv files and then I would like to iterate over the csv files opened and upload 1 row of each file at a time to my sql database.
For example, loop through each file uploading the first row of each file to the database, then loop again through each file uploading the second row of each file to the database.
However, I'm stuck in having the csv files ready to be uploaded in a single object.
The error happens at 'csv_data[i] = csv.reader...'
Each file is for a different table, so I cannot append them.
import csv
import sys
i = 0
for argv in sys.argv[1:]:
csv_file = open(argv, newline='', encoding='utf-8-sig')
csv_data[i] = csv.reader(csv_file, dialect='excel', delimiter=',', quotechar='|')
csv_file.close()
i += 1
After this code, I would need something to loop through each file uploading a certain row number.
zip together the files, iterate through them:
file_handles = [open(file, newline='', encoding='utf-8-sig') for file in argv[1:]]
readers = (csv.reader(file, dialect='excel', delimiter=',', quotechar='|') for file in file_handles)
# zip here
for line_group in zip(*readers):
# line_group is a tuple of line i of each file
# don't forget to close your files
for file_handle in file_handles:
try:
file_handle.close()
except:
print("Issue closing one of the files")
I'm trying to loop through rows in a csv file. I get csv file as string from a web location. I know how to create csv.reader using with when data is stored in a file. What I don't know is, how to get rows using csv.reader without storing string to a file. I'm using Python 2.7.12.
I've tried to create StringIO object like this:
from StringIO import StringIO
csv_data = "some_string\nfor_example"
with StringIO(csv_data) as input_file:
csv_reader = reader(csv_data, delimiter=",", quotechar='"')
However, I'm getting this error:
Traceback (most recent call last):
File "scraper.py", line 228, in <module>
with StringIO(csv_data) as input_file:
AttributeError: StringIO instance has no attribute '__exit__'
I understand that StringIO class doesn't have __exit__ method which is called when when finishes doing whatever it does with this object.
My answer is how to do this correctly? I suppose I can alter StringIO class by subclassing it and adding __exit__ method, but I suspect that there is easier solution.
Update:
Also, I've tried different combinations that came to my mind:
with open(StringIO(csv_data)) as input_file:
with csv_data as input_file:
but, of course, none of those worked.
>>> import csv
>>> csv_data = "some,string\nfor,example"
>>> result = csv.reader(csv_data.splitlines())
>>> list(result)
[['some', 'string'], ['for', 'example']]
You should use the io module instead of the StringIO one, because io.BytesIO for byte string or io.StringIO for Unicode ones both support the context manager interface and can be used in with statements:
from io import BytesIO
from csv import reader
csv_data = "some_string\nfor_example"
with BytesIO(csv_data) as input_file:
csv_reader = reader(input_file, delimiter=",", quotechar='"')
for row in csv_reader:
print row
If you like context managers, you can use tempfile instead:
import tempfile
with tempfile.NamedTemporaryFile(mode='w') as t:
t.write('csv_data')
t.seek(0)
csv_reader = reader(open(t.name), delimiter=",", quotechar='"')
As an advantage to pass string splitlines directly to csv reader you can write file of any size and then safely read it in csv reader without memory issues.
This file will be closed and deleted automatically
I've got a section of code in a project that's supposed to be reading a CSV file and writing each row to an XLSX file. Right now I'm getting the error "argument 1 must be an iterator" when I run via command line.
Here is the relevant code:
import os
import openpyxl
import csv
from datetime import datetime
from openpyxl.reader.excel import load_workbook
...
plannum = 4
...
alldata_sheetname = ("All Test Data " + str(plannum))
wb = load_workbook("testingtemplate.xlsx", keep_vba=True)
...
ws_testdata = wb.get_sheet_by_name(alldata_sheetname)
...
with open("testdata.csv", 'r') as csvfile:
table = csv.reader(csvfile)
for row in table:
ws_testdata.append(row)
csv_read = csv.reader(csvfile)
...
And the specific error reads: "TypeError: argument 1 must be an iterator", and is referencing the last line of code I've provided.
Since it didn't complain about the first time I used csvfile, would it be better if I did something like csvfile = open("testdata.csv", "r") instead of using the with (and is that what I'm doing wrong here)? If that's the case, is there anything else I need to change?
Thanks to anyone who helps!!
You've closed the file by the time you get to csv_read = csv.reader(csvfile). Alternately you can keep the file open and store what you need in variables so you don't have to iterate over the file twice. E.g.:
csvfile = open("testdata.csv", "r")
table = csv.reader(csvfile)
for row in table:
ws_testdata.append(row)
# store what you need in variables
csvfile.close()
I am trying to migrate some code from Python 2 to Python 3 and cannot figure out why it is printing one character at a time as if it is reading the file as one long string.
I have been looking into it and maybe a need to use newline='' when opening the file?
But how can I do that when using urlopen()?
import csv
import urllib.request
url = "http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = ftpstream.read().decode('utf-8')
csvfile = csv.reader(csvfile, delimiter=',')
for row in csvfile:
print(row)
Try to change
csvfile = ftpstream.read().decode('utf-8')
to
csvfile = ftpstream.read().decode('utf-8').split('\r')
I have been trying to read a csv file from my desktop and have not been successful. I checked my current working directory and it is pointed to my desktop, so that doesn't seem to be the issue. Below is the module I used and the error output that I received. I am using Python 3.2.3
import csv
reader = csv.reader(open(name.csv, mode = 'r'))
for row in reader:
print (row)
Here is my result
Traceback (most recent call last):
File "C:/Users/User Name/Desktop/FileName.py", line 2,in
reader = csv.reader(open(name.csv, mode = 'r'))
NameError: name 'Beta' is not defined
Help? Thanks!
Try this...
import csv
with open('name.csv', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter=',')
for row in reader:
print row