This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
check what files are open in Python
Hello,
Is it possible to obtain a list of all currently open file handles, I presume that they are stored somewhere in the environment.
I am interested in theis function as I would like to safely handle any files that are open when a fatal error is raised, i.e. close file handles and replace potentially corrupted files with the original files.
I have the handling working but without knowing what file handles are open, I am unable to implement this idea.
As an aside, when a file handle is initialised, can this be inherited by another imported method?
Thank you
lsof, /proc/pid/fd/
The nice way of doing this would be to modify your code to keep track of when it opens a file:
def log_open( *args, **kwargs ):
print( "Opening a file..." )
print( *args, **kwargs )
return open( *args, **kwargs )
Then, use log_open instead of open to open files. You could even do something more hacky, like modifying the File class to log itself. That's covered in the linked question above.
There's probably a disgusting, filthy hack involving the garbage collector or looking in __dict__ or something, but you don't want to do that unless you absolutely really truly seriously must.
If you're using python 2.5+ you can use the with keyword (though 2.5 needs `from future import with_statement)
with open('filename.txt', 'r') as f:
#do stuff here
pass
#here f has been closed and disposed properly - even with raised exceptions
I don't know what kind of catastrophic failure needs to bork the with statement, but I assume it's a really bad one. On WinXP, my quick unscientific test:
import time
with open('test.txt', 'w') as f:
f.write('testing\n')
while True:
time.sleep(1)
and then killing the process with Windows Task Manager still wrote the data to file.
Related
consider the python code below:
def create_fout_bt(location):
fout = open(os.path.join(location, 'The Book Thief.txt'), 'w')
# Added 'w' in Edit 1. Can't believe none of you guys noticed it! :P
return fout
def main():
location = r'E:\Books\Fiction'
fout_bt = create_fout_bt(location)
fout_bt.write('Author: Markus Zusak\n')
fout_bt.close()
main()
In this code, the fileobject named fout is created inside the function create_fout_bt, but not closed within the same function. What I understand is that we have to close every fileobject we create; so is this ok? In practice, the code works fine and the output file is generated with the content I wrote to it, but just wondering if a fileobject is dangling somewhere out there.
Thanks for your time.
Edit 1:
Thank you for introducing me to the python with statement. Hopefully I'll use it in the future.
Also, let me clarify that the code I mentioned here is a generic, simple case. Of course it doesn't make sense to define a function just to create a fileobject! In the real scenario, I will be writing to many different files concurrently. For example:
fout1.write('%s: %f' %('Magnetic Field', magnetic_field))
fout2.write('%s: %f' %('Power', power))
fout3.write('%s: %f' %('Cadence', cadence))
Now this requires creating the fileobjects fout1, fout2, fout3:
fout1 = open(os.path.join(rootPath, 'filename1.txt'), 'w')
fout2 = open(os.path.join(rootPath, 'filename2.txt'), 'w')
fout3 = open(os.path.join(rootPath, 'filename3.txt'), 'w')
Since there are many of them, I wanted to put them inside a function to make it look better - now a single function call will get me all the fileobjects:
fout1, fout2, fout3 = create_file_objects(rootPath)
Moreover, in the real scenario, I have to write into a file at multiple locations in the program. From what I have understood, if I'm using 'with', I'll have to open the file in append mode each time I have to write into it (making the code look cluttered); compared to using an 'open()' function which will keep the file open till I use the close() function.
Like deceze commented, the problem I'm worried about is spreading the responsibility of the fileobject to multiple functions. In my first example,
'fout' is the variable created inside the function 'create_fout_bt' and 'fout_bt' is the variable to which that value is assigned by the latter. Now, I know 'fout_bt' is taken care of with the statement 'fout_bt.close()', but what about 'fout' inside the function 'create_fout_bt'? Will it be disposed off when the function 'create_fout_bt' returns?
Hope my doubt is more clear. Do let me know if I just missed something obvious. Any comments on how to make my future posts more palatable will also be much appreciated. :)
Your code works fine, I try #Sujay 's suggestion, it raises an error I/O operation on closed file after fout_bt.close()
If you afraid of your code style, you can use with to do it.
code:
def create_fout_bt(location):
fout = open(os.path.join(location, 'The Book Thief.txt'),"a")
return fout
def main():
location = r'E:\Books\Fiction'
with create_fout_bt(location) as fout_bt:
fout_bt.write('Author: Markus Zusak\n')
main()
The only thing is that the code that opens the file (create_fout_bt) cannot guarantee that the file will also be closed. Which isn't an issue per se, but it spreads that responsibility around and may lead to situations in which the file isn't closed, because the caller doesn't handle the returned file handle correctly. It's still fine to do this, you just need to be diligent. One way this could be improved is with this:
with create_fout_bt(location) as fout_bt:
fout_bt.write('Author: Markus Zusak\n')
Using a with context manager on the file object, regardless of whether directly created with open or "indirectly" via create_fout_bt, guarantees that the file will be closed, regardless of errors happening in your code.
you can use 'with'.
with 'with' you don't need to close your files anymore and it automatically close it self.
do it like this :
with create_fout_bt(location) as fout_bt:
fout_bt.write('Author: Markus Zusak\n')
This question already has answers here:
Why is `with open()` better for opening files in Python?
(4 answers)
File read using "open()" vs "with open()" [duplicate]
(1 answer)
What is the python "with" statement designed for?
(11 answers)
Closed 3 years ago.
For the sake of a small example. Let's say that I want to read each line from this file into a list:
First line
Second line
Third line
Fourth line
and it's called example.txt and in my cwd.
I've always been told that I should do it using a with statement like these:
# method 1
lines = []
with open("example.txt") as handle:
for line in handle:
lines.append(line.strip("\n"))
# method 2
with open("example.txt") as handle:
lines = [line.strip("\n") for line in handle]
There's a lot of info out there on this.. I found some good stuff in
What is the python keyword "with" used for?. So it seems this is/was recommended so that the file was properly closed, especially in the cases of exceptions being thrown during processing.
# method 3
handle = open("example.txt")
lines = [line.strip("\n") for line in handle]
handle.close()
# method 4
lines = [line.strip("\n") for line in open("example.txt")]
For that reason the above two methods would be frowned upon. But all of the information I could find on the subject is extremely old, most of it being aimed at python 2.x. sometimes using with instead of these methods within more complex code is just impractical.
I know that in python3 at some point variables within a list comprehension were limited to their own scope (so that the variables would not leak out of the comprehension).. could that take care of the issue in method 4? Would I still experience issues when using method 3 if an exception was thrown before I explicity called the .close() statement?
Is the insistence on always using with to open file objects just old habits dying hard?
The with keyword is a context manager. It is not required to work with a file. However, it is good practice.
It is good practice to use the with keyword when dealing with file
objects. The advantage is that the file is properly closed after its
suite finishes, even if an exception is raised at some point. Using
with is also much shorter than writing equivalent try-finally blocks...
Python Tutorial Section 7.2
Using it is syntactic sugar for a try/except block similar to:
f = open('example.txt', 'w')
try:
f.write('hello, world')
finally:
f.close()
You could explicitly write that block out whenever you want to work with a file. In my opinion it is more pythonic, and easier to read, to use the context manager syntax.
with open('example.txt', 'w') as f:
f.write('hello, world')
If you would like to learn a little more about context managers check out this blog post and of course the Python Documentation on contextlib.
There is also a 5th way:
handle = None
try:
handle = open("example.txt")
except:
pass
finally:
if handle is not None:
handle.close()
Note, this would not work, in case open would throw an exception, handle.close() would crash the program.:
try:
handle = open("example.txt")
except:
pass
finally:
handle.close()
Yes. Closing files is still needed. This concept is present in all programming languages and is caused by buffering, which improves performance of working with IO. To avoid the need of closing file handles you would need to find a way of reading files without buffering.
Creating code in 2.7 (for a Python class I am taking) and cross-testing it in 3.x (for me). This code is not required. I decided to build objects that might be useful for the future instead of simply answering the original question directly.
Since 3.x has no file object to inherit, my object uses file instances within it instead of inheriting from it directly. During testing of the code which follows, I encountered an oddity. I can close a file and then close it again and this does not trigger any kind of error. To sanity checking this code is being built right, two questions about this:
* why can I close the same file twice
* the way I have written this, when I close the file, am I really closing it and freeing up the memory the file object is taking up?
Here is the code:
class FileData(object):
def __init__(self, xfilename, xmode):
self.filename = xfilename
self.mode = xmode
self.f = open(xfilename, 'r')
def getAllFileData(self):
self.lines = f.readlines()
self.f.close()
def getLineFromFile(self):
if f.closed:
self.f = open(xfilename, 'r')
self.f.readline()
def fileHead(self, numRows):
if self.f.closed:
self.f = open(xfilename, 'r')
for i in range(numRows):
print(self.f.readline())
self.f.close()
Then I ran these test lines and accidentally re-ran the line to close the file multiple times. To make sure I wasn't missing something, I later organized these lines in a Jupyter cell and ran them together.
chatLog = FileData("script/record.txt", "r")
chatLog.fileHead(15)
chatLog.f.close()
chatLog.f.close()
Note that fileHead() also closes the file when done, so really above code should have tried to close the same file 3 times.
Why no error? And is this closing the file the way I think it is? Learning the language so any input would be appreciated.
f.close() won't do anything if the file is already closed as you noticed yourself.
You could protect against it but seeing how complex it is I wouldn't advise to do it:
import _io
def override_close(f):
if f.closed:
raise IOError("already closed error")
else:
_io.TextIOWrapper.close(f)
f = open("foo.c")
f.close = lambda : override_close(f)
print("closing")
f.close()
print("protected")
f.close() # raises IOError
I have overridden the close method for the f object so it checks against already closed file and raises an exception.
For your example, the best way would be to hide f (and other data) from the outside by making them not directly invisible from the outside:
self.__f = open(...)
so callers cannot mess (easily :)) with your file handle anymore.
Because using .close() repeatedly is okay. Internally, it might be checking .closed to be True then do nothing, else actually close the file.
I would like to insert the following function into a Python script, concerning a file, and I haven't found how to write it yet:
if the file is opened, close it
if it is already closed, do nothing
Any help will be really much appreciated. Thank you in advance!
You write this as f.close().
In Python 3.x, IOBase.close says:
Flush and close this stream. This method has no effect if the file is already closed.
Likewise, in Python 2.x, file.close says:
Close the file … Calling close() more than once is allowed.
Of course if you read the docs, you'll notice that files (whether 3.x IOBase or 2.x file) also have a closed attribute, so if you really wanted to write what you were asking for explicitly, you could:
if not f.closed:
f.close()
But that has no benefit over just calling f.close(). Unless you want to make sure that some file-like objects that aren't 100% file-like raise an inscrutable AttributeError instead of just working as you'd like, but I doubt you want that.
f.close() is the way to close a file. Python details how to use file operations Here in python 2 and here in Python3
This question already has an answer here:
In the Inline "open and write file" is the close() implicit?
(1 answer)
Closed 9 years ago.
I've seen this idiom in Dive Into Python 3:
l = list(open('strcpy.c'))
The question is, how can I close the file?
Is something happening behind the scenes?
I couldn't find this information in the book.
The file will be closed when its object is garbage-collected. In CPython, this happens pretty much immediately after that line is executed, because the file is never assigned to a variable. In other Pythons, such as Jython or IronPython, this may not happen right away (or at all), though all open files are always closed when the process exits.
For this reason, a better approach is to close the file explicitly using 'with':
with open("strcpy.c") as infile:
l = list(infile)
An advantage of this is that the file will be properly closed even if an exception occurs in reading it; you don't have to manually write code for this case using a try/except block.
A with statement can be written on one line if you want to stick with the concise one-liner. :-)
That said, I do sometimes use this idiom myself in short-running scripts where having the file open a wee bit longer than it strictly needs to be isn't a big deal. An advantage is that you don't clutter things up with a variable (infile in this case) pointing to a closed file.
From doc:
It is good practice to use the with keyword when dealing with file
objects. This has the advantage that the file is properly closed after
its suite finishes, even if an exception is raised on the way.
You can use it like this:
with open('strcpy.c') as f:
l = list(f)
Personally, I would never open a file without a context manager:
with open('strcpy.c') as myfile:
l = list(myfile)
# do stuff with l here
When using the context manager, the file is automagically closed at the end of the indentation block.