I am on windows with Python 2.5. I have an open file for writing. I write some data. Call file close. When I try to delete the file from the folder using Windows Explorer, it errors, saying that a process still holds a handle to the file.
If I shutdown python, and try again, it succeeds.
It does close them.
Are you sure f.close() is getting called?
I just tested the same scenario and windows deletes the file for me.
Are you handling any exceptions around the file object? If so, make sure the error handling looks something like this:
f = open("hello.txt")
try:
for line in f:
print line
finally:
f.close()
In considering why you should do this, consider the following lines of code:
f = open('hello.txt')
try:
perform_an_operation_that_causes_f_to_raise_an_exception()
f.close()
except IOError:
pass
As you can see, f.close will never be called in the above code. The problem is that the above code will also cause f to not get garbage collected. The reason is that f will still be referenced in sys.traceback, in which case the only solution is to manually call close on f in a finally block or set sys.traceback to None (and I strongly recommend the former).
Explained in the tutorial:
with open('/tmp/workfile', 'r') as f:
read_data = f.read()
It works when you writing or pickling/unpickling, too
It's not really necessary that try finally block: Java way of doing things, not Python
I was looking for this, because the same thing happened to me. The question didn't help me, but I think I figured out what happened.
In the original version of the script I wrote, I neglected to add in a 'finally' clause to the file in case of an exception.
I was testing the script from the interactive prompt and got an exception while the file was open. What I didn't realize was that the file object wasn't immediately garbage-collected. After that, when I ran the script (still from the same interactive session), even though the new file objects were being closed, the first one still hadn't been, and so the file handle was still in use, from the perspective of the operating system.
Once I closed the interactive prompt, the problem went away, at which I remembered that exception occurring while the file was open and realized what had been going on. (Moral: Don't try to program on insufficient sleep. : ) )
Naturally, I have no idea if this is what happened in the case of the original poster, and even if the original poster is still around, they may not remember the specific circumstances, but the symptoms are similar, so I thought I'd add this as something to check for, for anyone caught in the same situation and looking for an answer.
I did it using intermediate file:
import os
f = open("report.tmp","w")
f.write("{}".format("Hello"))
f.close()
os.system("move report.tmp report.html") #this line is for Windows users
Related
consider the python code below:
def create_fout_bt(location):
fout = open(os.path.join(location, 'The Book Thief.txt'), 'w')
# Added 'w' in Edit 1. Can't believe none of you guys noticed it! :P
return fout
def main():
location = r'E:\Books\Fiction'
fout_bt = create_fout_bt(location)
fout_bt.write('Author: Markus Zusak\n')
fout_bt.close()
main()
In this code, the fileobject named fout is created inside the function create_fout_bt, but not closed within the same function. What I understand is that we have to close every fileobject we create; so is this ok? In practice, the code works fine and the output file is generated with the content I wrote to it, but just wondering if a fileobject is dangling somewhere out there.
Thanks for your time.
Edit 1:
Thank you for introducing me to the python with statement. Hopefully I'll use it in the future.
Also, let me clarify that the code I mentioned here is a generic, simple case. Of course it doesn't make sense to define a function just to create a fileobject! In the real scenario, I will be writing to many different files concurrently. For example:
fout1.write('%s: %f' %('Magnetic Field', magnetic_field))
fout2.write('%s: %f' %('Power', power))
fout3.write('%s: %f' %('Cadence', cadence))
Now this requires creating the fileobjects fout1, fout2, fout3:
fout1 = open(os.path.join(rootPath, 'filename1.txt'), 'w')
fout2 = open(os.path.join(rootPath, 'filename2.txt'), 'w')
fout3 = open(os.path.join(rootPath, 'filename3.txt'), 'w')
Since there are many of them, I wanted to put them inside a function to make it look better - now a single function call will get me all the fileobjects:
fout1, fout2, fout3 = create_file_objects(rootPath)
Moreover, in the real scenario, I have to write into a file at multiple locations in the program. From what I have understood, if I'm using 'with', I'll have to open the file in append mode each time I have to write into it (making the code look cluttered); compared to using an 'open()' function which will keep the file open till I use the close() function.
Like deceze commented, the problem I'm worried about is spreading the responsibility of the fileobject to multiple functions. In my first example,
'fout' is the variable created inside the function 'create_fout_bt' and 'fout_bt' is the variable to which that value is assigned by the latter. Now, I know 'fout_bt' is taken care of with the statement 'fout_bt.close()', but what about 'fout' inside the function 'create_fout_bt'? Will it be disposed off when the function 'create_fout_bt' returns?
Hope my doubt is more clear. Do let me know if I just missed something obvious. Any comments on how to make my future posts more palatable will also be much appreciated. :)
Your code works fine, I try #Sujay 's suggestion, it raises an error I/O operation on closed file after fout_bt.close()
If you afraid of your code style, you can use with to do it.
code:
def create_fout_bt(location):
fout = open(os.path.join(location, 'The Book Thief.txt'),"a")
return fout
def main():
location = r'E:\Books\Fiction'
with create_fout_bt(location) as fout_bt:
fout_bt.write('Author: Markus Zusak\n')
main()
The only thing is that the code that opens the file (create_fout_bt) cannot guarantee that the file will also be closed. Which isn't an issue per se, but it spreads that responsibility around and may lead to situations in which the file isn't closed, because the caller doesn't handle the returned file handle correctly. It's still fine to do this, you just need to be diligent. One way this could be improved is with this:
with create_fout_bt(location) as fout_bt:
fout_bt.write('Author: Markus Zusak\n')
Using a with context manager on the file object, regardless of whether directly created with open or "indirectly" via create_fout_bt, guarantees that the file will be closed, regardless of errors happening in your code.
you can use 'with'.
with 'with' you don't need to close your files anymore and it automatically close it self.
do it like this :
with create_fout_bt(location) as fout_bt:
fout_bt.write('Author: Markus Zusak\n')
I'm doing a calculation on EC2 using python and when I try to dump a pickle file containing a dictionary, the program just exists, no error notice or anything. The file is large, around 1 gig, but everything works fine on my laptop, just not on EC2. I'm using an m3.large instance with an attached EBS volume with plenty of space. In the following code snippet, it prints out "dumping now" and then.....nothing, no "dumping complete". No error is caught by 'except'. When I try to load the pickle file I get an EOF Error.
Thanks for any advice!
try:
fp = open(pickleFile,"wb")
print 'dumping now'
pickle.dump(dataDict, fp)
print 'dumping complete'
fp.close()
except:
fp = open('/Users/thisUser/Data/report.txt','w')
fp.write('error writing pickle file')
fp.close()
Especially if you're debugging file IO, try not to have your debugging depend on said file IO. If there are any permissions or disk issues, you'll never know since the error message won't be able to write either.
Try running watch "ls -lh /your/file", then running your script, to see if the file writing stops suddenly, or is just continuing at a slow pace.
Try Ctrl-C if your script is hung and your watch looks static, and check out where in the traceback your function is spending its time.
Try pickling elements/keys of the dictionary individually--the pickling may be crashing on an individual complex element.
Try cPickle
Compare your versions at home and on ec2: import cPickle; print cPickle.__version__
Try different pickling versions.
I'm not sure which of these might help, but one of them might.
My problem is that logging stops for a python program when the log is rotated.
I have tracked it down to the stream itself. I don't see any way to tell if the stream is broken from python. After the file is deleted it still accepts writes without any issue.
import os
FILE = 'testing.txt'
fs = open(FILE, 'a')
fs.write('word')
os.remove(FILE)
fs.write('Nothing....') # Nothing breaks
print(fs.errors) # No errors
So, how can I find out if the file stream is still valid?
And checking to see if the file exists will not help since the file will always exist regardless of whether or not the stream is still valid.
Upon much more inspection, I found the solution. It is an OS specific problem. When the file is deleted in Linux (or Macintosh) it just unlinks it. (I was not aware of this)
So if you run lsof on the machine, it still shows the file as open.
[user#machine]$ lsof | grep --color -i "testing.txt"
python26 26495 user 8w REG 8,33 23474 671920 /home/user/temp/testing.txt (deleted)
The solution is to stat the stream in python.
stat = os.fstat(fs.fileno())
Which will give you the number of links it has.
if stat.st_nlink < 1:
#has been deleted
And there you go. Now you know if you should reload it or not. Hopefully this helps someone else.
Try Exception handling:
import os
FILE = 'testing.txt'
try:
fs = open(FILE, 'a')
fs.write('word')
os.remove(FILE)
fs.write('Nothing....') # Nothing breaks
except Exception, e:
print "Error:", e
print(fs.errors) # No errors
There are python bindings for ionotify if you need more intelligence than just an try: except: clause. But I think its only pertinent to Linux (im not sure of your platform)
Another solution I found is to add the "copytruncate" flag into the logrotate config.
See "man logrotate" for more info.
edit:OK, I could swear that the way I'd tested it showed that the getcwd was also causing the exception, but now it appears it's just the file creation. When I move the try-except blocks to their it actually does catch it like you'd think it would. So chalk that up to user error.
Original Question:
I have a script I'm working on that I want to be able to drop a file on it to have it run with that file as an argument. I checked in this question, and I already have the mentioned registry keys (apparently the Python 2.6 installer takes care of it.) However, it's throwing an exception that I can't catch. Running it from the console works correctly, but when I drop a file on it, it throws an exception then closes the console window. I tried to have it redirect standard error to a file, but it threw the exception before the redirection occurred in the script. With a little testing, and some quick eyesight I saw that it was throwing an IOError when I tried to create the file to write the error to.
import sys
import os
#os.chdir("C:/Python26/Projects/arguments")
try:
print sys.argv
raw_input()
os.getcwd()
except Exception,e:
print sys.argv + '\n'
print e
f = open("./myfile.txt", "w")
If I run this from the console with any or no arguments, it behaves as one would expect. If I run it by dropping a file on it, for instance test.txt, it runs, prints the arguments correctly, then when os.getcwd() is called, it throws the exception, and does not perform any of the stuff from the except: block, making it difficult for me to find any way to actually get the exception text to stay on screen. If I uncomment the os.chdir(), the script doesn't fail. If I move that line to within the except block, it's never executed.
I'm guessing running by dropping the file on it, which according to the other linked question, uses the WSH, is somehow messing up its permissions or the cwd, but I don't know how to work around it.
Seeing as this is probably not Python related, but a Windows problem (I for one could not reproduce the error given your code), I'd suggest attaching a debugger to the Python interpreter when it is started. Since you start the interpreter implicitly by a drag&drop action, you need to configure Windows to auto-attach a debugger each time Python starts. If I remember correctly, this article has the needed information to do that (you can substitute another debugger if you are not using Visual Studio).
Apart from that, I would take a snapshot with ProcMon while dragging a file onto your script, to get an idea of what is going on.
As pointed out in my edit above, the errors were caused by the working directory changing to C:\windows\system32, where the script isn't allowed to create files. I don't know how to get it to not change the working directory when started that way, but was able to work around it like this.
if len(sys.argv) == 1:
files = [filename for filename in os.listdir(os.getcwd())
if filename.endswith(".txt")]
else:
files = [filename for filename in sys.argv[1:]]
Fixing the working directory can be managed this way I guess.
exepath = sys.argv[0]
os.chdir(exepath[:exepath.rfind('\\')])
It turns out that "with" is a funny word to search for on the internet.
Does anyone knows what the deal is with nesting with statements in python?
I've been tracking down a very slippery bug in a script I've been writing and I suspect that it's because I'm doing this:
with open(file1) as fsock1:
with open(file2, 'a') as fsock2:
fstring1 = fsock1.read()
fstring2 = fsock2.read()
Python throws up when I try to read() from fsock2. Upon inspection in the debugger, this is because it thinks the file is empty. This wouldn't be worrisome except for the fact that running the exact same code in the debugging interperter not in a with statement shows me that the file is, in fact, quite full of text...
I'm going to proceed on the assumption that for now nesting with statements is a no-no, but if anyone who knows more has a different opinion, I'd love to hear it.
I found the solution in python's doc. You may want to have a look at this (Python 3) or this (Python 2)
If you are running python 2.7+ you can use it like this:
with open(file1) as fsock1, open(file2, 'a') as fsock2:
fstring1 = fsock1.read()
fstring2 = fsock2.read()
This way you avoid unnecessary indentation.
AFAIK you can't read a file open with append mode 'a'.
Upon inspection in the debugger, this is because it thinks the file is empty.
I think that happens because it can't actually read anything. Even if it could, when you append to a file, the seek pointer is moved to the end of the file in preparation for writing to occur.
These with statements work just fine for me:
with open(file1) as f:
with open(file2, 'r') as g: # Read, not append.
fstring1 = f.read()
fstring2 = g.read()
Note that use of contextlib.nested, as another poster suggested, is potentially fraught with peril here. Let's say you do this:
with contextlib.nested(open(file1, "wt"), open(file2)) as (f_out, f_in):
...
The context managers here get created one at a time. That means that if the opening of file2 fails (say, because it doesn't exist), then you won't be able to properly finalize file1 and you'll have to leave it up to the garbage collector. That's potentially a Very Bad Thing.
There is no problem with nesting with statements -- rather, you're opening file2 for append, so you can't read from it.
If you do dislike nesting with statements, for whatever reason, you can often avoid that with the contextlib.nested function. However, it won't make broken code (e.g., code that opens a file for append and then tries to read it instead) work, nor will lexically nesting with statements break code that's otherwise good.
As of python 3.10 you can do it like this
with (
Something() as example1,
SomethingElse() as example2,
YetSomethingMore() as example3,
):
...
this can be helpful in pytests when you want to do nested patches in some autouse fixture like so
from unittest.mock import patch
import pytest
#pytest.fixture(scope="session", autouse=True)
def setup():
with (
patch("something.Slow", MagicMock()) as slow_mock,
patch("something.Expensive") as expensive_mock,
patch("other.ThirdParty", as third_party_mock,
):
yield
As for searching for "with", prefixing a word with '+' will prevent google from ignoring it.