I'm using the CSV module to read a tab delimited file. Code below:
z = csv.reader(open('/home/rv/ncbi-blast-2.2.23+/db/output.blast'), delimiter='\t')
But when I add Z.close() to end of my script i get and error stating "csv.reader' object has no attribute 'close'"
z.close()
So how do i close "Z"?
The reader is really just a parser. When you ask it for a line of data, it delegates the reading action to the underlying file object and just converts the result into a set of fields. The reader itself doesn't manage any resources that would need to be cleaned up when you're done using it, so there's no need to close it; it'd be a meaningless operation.
You should make sure to close the underlying file object, though, because that does manage a resource (an open file descriptor) that needs to be cleaned up. Here's the way to do that:
with open('/home/rv/ncbi-blast-2.2.23+/db/output.blast') as f:
z = csv.reader(f, delimiter='\t')
# do whatever you need to with z
If you're not familiar with the with statement, it's roughly equivalent to enclosing its contents in a try...finally block that closes the file in the finally part.
Hopefully this doesn't matter (and if it does, you really need to update to a newer version of Python), but the with statement was introduced in Python 2.5, and in that version you would have needed a __future__ import to enable it. If you were working with an even older version of Python, you would have had to do the closing yourself using try...finally.
Thanks to Jared for pointing out compatibility issues with the with statement.
You do not close CSV readers directly; instead you should close whatever file-like object is being used. For example, in your case, you'd say:
f = open('/home/rv/ncbi-blast-2.2.23+/db/output.blast')
z = csv.reader(f, delimiter='\t')
...
f.close()
If you are using a recent version of Python, you can use the with statement, e.g.
with open('/home/rv/ncbi-blast-2.2.23+/db/output.blast') as f:
z = csv.reader(f, delimiter='\t')
...
This has the advantage that f will be closed even if you throw an exception or otherwise return inside the with-block, whereas such a case would lead to the file remaining open in the previous example. In other words, it's basically equivalent to a try/finally block, e.g.
f = open('/home/rv/ncbi-blast-2.2.23+/db/output.blast')
try:
z = csv.reader(f, delimiter='\t')
...
finally:
f.close()
You don't close the result of the reader() method, you close the result of the open() method. So, use two statements: foo=open(...); bar=csv.reader(foo). Then you can call foo.close().
There are no bonus points awarded for doing in one line that which can be more readable and functional in two.
Related
To shorten the code a little, I used something like:
data = open(file).read()
It's impossible to close by data.close() since data it's a string, so I ask the question: if after that line is read, is the file still open? Do I need to bother to close it somehow?
In CPython the result of open(file) will have no references to it after .read() completes if there were no exceptions, so it will be instantly garbage collected which will cause it to close.
But don't do this. Make a convenience function:
def read_file(path):
with open(path) as f:
return f.read()
data = read_file(path)
Also don't fuss about saving a line of code. That attitude will lead to bad code.
It will (probably, see comments) be closed when garbage collected; however the cleanest way to handle it is within a with block:
with open(file, "r") as datafile:
data = datafile.read()
If the process is short-lived, it shouldn't cause too much trouble, but it's good practice to release the handle as soon as it becomes unnecessary.
yes, it will remain open ,my way to open data like this:
with open("file","rb") as f：
data=f.readlines()
or
datas=open("file","rb")
data=datas.readlines()
wish i could help you
This is similar or identical to csv writer not closing file but I'm not 100% sure why my behaviour is different.
def LoadCSV:
with open('test.csv', 'r') as csvfile:
targetReader = csv.reader(csvfile, delimiter=',')
for row in targetReader:
...
then finally in the function
csvfile.close()
This opens the test.csv file in the same direction as the script. Desired behaviour is for when the script has done what it's doing to the rows in the function, it renames the sheet to test.[timestamp] to archive it and watches the directory for a new sheet to arrive.
Later down the code;
os.rename('test.csv', "test." + time.strftime("%x") )
Gives an error that the file can't be renamed because a process is still using it. How do I close this file once I'm done? csvfile.close() doesn't raise an exception, and if I step through my code in interactive mode I can see that csvfile is a "closed file object." What even is that? Surely an open file is an object but a closed one isn't, how do I make my code forget this even exists so I can then do IO on the file?
NOT FOR POINTS.
Code is not valid anyway, since your function name is wrong. If that was not intentional, better edit it or to produce a pseudo-replica of your code, rather than have us guess what the issue is.
To iterate, the issues with your code:
def LoadCSV is not valid. def LoadCSV() is. Proof in following screenshot. Notice how the lack of () is showing syntax error markers.
Fixing (1) above, your next problem is using csvfile.close(). If the code is properly written, once the code is out of the scope of with, the file is closed automatically. Even if the renaming part of the code is inside the function, it shouldn't pose any problems.
Final word of warning -- using the format string %x will produce date-strings like 08/25/14, depending on locale. Obviously, this is erroneous, as a / is invalid in filenames in Windows (try renaming a file manually with this). Better to be very explicit and just use %m%d%y instead.
Finally, see the running code on my end. If your code is not structured like this, then other errors we cannot guess might arise.
Result as follows after running:
Code for reference:
import csv
import os
import time
def LoadCSV():
with open("test.csv", "r") as csvfile:
targetReader = csv.reader(csvfile, delimiter=",")
for row in targetReader:
print row
new_name = "test.%s.csv" % time.strftime("%m%d%y")
print new_name
os.rename("test.csv", new_name)
LoadCSV()
Note that on my end, there is nothing that watches my file. Antivirus is on, and no multithreading obviously is enabled. Check if one of your other scripts concurrently watches this file for changes. It's better if instead of watching the file, the file is sent as an argument post-renaming to this other function instead, as this might be the reason why it's "being used". On the one hand, and this is untested on my side, possibly better to copy the file with a new name rather than rename it.
Hope this helps.
When you are using a with block you do not need to close the file, it should be released outside the scope. If you want python to "forget" the entire filehandle you could delete it with del csvfile. But since you are using with you should not delete the variable inside the scope.
Try without the with scope instead:
csvfile = open('test.csv','r')
targetReader = csv.reader(csvfile, delimiter=',')
for row in targetReader:
....
csvfile.close()
del targetReader
os.rename('test.csv','test.'+time.strftime('%x'))
It might be the csv reader that still access the file when you are using a with block.
I find each time I open a file and read its content, it automatically close. So next time I have to open that file again to read it. I know this may be a good idea to reduce memory usage. But I need to consistently open a file due to my current task.
How should I do it?
This is my current way of reading a file
f = open('some_file', 'rU')
f.read()
After you do f.read(), the file doesn't close. In fact, it doesn't close unless you explicitly call f.close() or you use it in a with block like this:
with open('some_file') as f:
...
In which case, the file will be closed for you when the with block ends. It will also tend to be closed in any case when the file object has no more variables associated with it (ie, when f falls out of scope or gets reassigned), but this isn't guaranteed behaviour. If none of these things happen, the file is kept open.
The problem you are most likely seeing is that calling read again will get you an empty string. This doesn't happen because the file is closed - reading from a closed file gives you an error. Rather, files keep track of where you have read up to, so that if you only read part of it, you can then request the next part and it will start at the right place. To set it back to read from the start of the file again again, you can use the seek method:
with open('some_file') as f:
contents1 = f.read()
f.seek(0)
contents2 = f.read()
will give you contents1 and contents2 both containing the full contents of the file, rather than contents2 being empty. However, you probably don't want to do this unless the file could have changed in the meantime.
To make sure your file will consisently open or be consistently open, you need to be closing files or seek to 0.
while True:
f = open(...)
x = f.read()
print x
f.close()
or
f = open(...)
while True:
x = f.read()
print x
f.seek(0)
...unless you are going to write a one-liner, which will close automatically.
print open('some_file', 'rU').read()
This avoids any limit on the number of open files.
Additional thought: You can also use for line in open(...): pass, again as long as you remember to get to the beginning of the file one way or another.
What is the 'Python way' regarding working with a CSV file? If I want to run some methods on the data in a particular column, should copy the whole think into an array, or should I pass the open file into a series of methods?
I tried to return the open file and got this error:
ValueError: I/O operation on closed file
here's the code:
import sys
import os
import csv
def main():
pass
def openCSVFile(CSVFile, openMode):
with open(CSVFile, openMode) as csvfile:
zipreader = csv.reader(csvfile, delimiter=',')
return zipreader
if __name__ == '__main__':
zipfile = openCSVFile('propertyOutput.csv','rb')
numRows = sum(1 for row in zipfile)
print"Rows equals %d." % numRows
Well there are many ways you could go about manipulating csv files. It depends
largely on how big your data is and how often you will perform these operations.
I will build on the already good answers and comments to present a somewhat more
complex handling, that wouldn't be far off from a real world example.
First of all, I prefer csv.DictReader because most csv files have a header
row with the column names. csv.DictReader takes advantage of that and gives
you the opportunity to grab it's cell value by its name.
Also, most of the times you need to perform various validation and normalization
operations on said data, so we're going to associate some functions with specific
columns.
Suppose we have a csv with information about products.
e.g.
Product Name,Release Date,Price
foo product,2012/03/23,99.9
awesome product,2013/10/14,40.5
.... and so on ........
Let's write a program to parse it and normalize the values
into appropriate native python objects.
import csv
import datetime
from decimal import Decimal
def stripper(value):
# Strip any whitespace from the left and right
return value.strip()
def to_decimal(value):
return Decimal(value)
def to_date(value):
# We expect dates like: "2013/05/23"
datetime.datetime.strptime(value, '%Y/%m/%d').date()
OPERATIONS = {
'Product Name': [stripper],
'Release Date': [stripper, to_date],
'Price': [stripper, to_decimal]
}
def parse_csv(filepath):
with open(filepath, 'r') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
for column in row:
operations = OPERATIONS[column]
value = row[column]
for op in operations:
value = op(value)
# Print the cleaned value or store it somewhere
print value
Things to note:
1) We operate on the csv in a line by line basis. DictReader yields lines
one at a time and that means we can handle arbitrary sizes of csv files,
since we are not going to load the whole file into memory.
2) You can go crazy with normalizing the values of a csv, by building special
classes with magic methods or whatnot. As I said, it depends on the complexity
of your files, the quality of the data and the operations you need to perform
on them.
Have fun.
csv module provides one row at a time, understanding its content by spliting it as a list object (or dict in case of DictReader).
As Python knows how to loop on such an object, if you're just interested in some specific fields, building a list with these fields seems 'Pythonic' to me. Using an iterator is also valid if each item shall be considered separatly from the others.
You probably need to read PEP 343: The 'with' statement
Relevant quote:
Some standard Python objects now support the context management protocol and can be used with the 'with' statement. File objects are one example:
with open('/etc/passwd', 'r') as f:
for line in f:
print line
... more processing code ...
After this statement has executed, the file object in f will have been automatically closed, even if the 'for' loop raised an exception part-way through the block.
So your csvfile is closed outside with statement, and outside openCSVFile function. You need to not to use with statement,
def openCSVFile(CSVFile, openMode):
csvfile = open(CSVFile, openMode)
return csv.reader(csvfile, delimiter=',')
or move it to __main__:
def get_csv_reader(filelike):
return csv.reader(csvfile, delimiter=',')
if __name__ == '__main__':
with open('propertyOutput.csv', 'rb') as csvfile:
zipfile = get_csv_reader(csvfile)
numRows = sum(1 for row in zipfile)
print"Rows equals %d." % numRows
Firstly, the reason you're getting ValueError: I/O operation on closed file is that in the following, the with acting as a context manager is operating on an opened file which is the underlying fileobj that zipreader is then set to work on. What happens, is that as soon as the with block is exited, the file that was opened is then closed, which leaves the file unusable for zipreader to read from...
with open(CSVFile, openMode) as csvfile:
zipreader = csv.reader(csvfile, delimiter=',')
return zipreader
Generally, acquire the resource and then pass it a function if needed. So, in your main program open the file and create the csv.reader and then pass that to something and have it closed in the main program when it makes more sense that "you're done with it now".
I'm trying to read the contents of a file in a single method call.
I don't want to have to worry about opening the file, reading from the file, and then closing the file (3 method calls).
I just want the content.
In ruby, there is File.read("/path/to/file"), which returns the contents of that file and properly closes it. Is there an equivalent in Python?
You can concatenate two instructions to get the same behaviour :/. But then the file isn't properly closed.
file = open("/path/to/file","r").read()
edit:
Best option as far as I know leaves you needing 2/3 you mention. Just use the with statement so you don't have to worry about closing said file.
with open("/path/to/file","r") as file:
text = file.read()
You can use a Context Manager in Python, which is available from Python 2.5.
with open('yourfile') as f:
contents = f.read()
It will automatically, open and close the file for you. The default mode is 'r' which stands for reading.
There is no such function included with Python. It's simple enough to define one, though.
def read_whole_file(path):
with open(path) as f:
return f.read()