Develop Reference

Wrong result with a loop in a loop [duplicate]

This question already has answers here:
Why can't I call read() twice on an open file?
(7 answers)
Closed 7 months ago.
I have a problem with iterating on a file. Here's what I type on the interpreter and the result:
>>> f = open('baby1990.html', 'rU')
>>> for line in f.readlines():
... print(line)
...
# ... all the lines from the file appear here ...
When I try to iterate on the same open file again I get nothing!
>>> for line in f.readlines():
... print(line)
...
>>>
There is no output at all. To solve this I have to close() the file then open it again for reading! Is that normal behavior?

Yes, that is normal behavior. You basically read to the end of the file the first time (you can sort of picture it as reading a tape), so you can't read any more from it unless you reset it, by either using f.seek(0) to reposition to the start of the file, or to close it and then open it again which will start from the beginning of the file.
If you prefer you can use the with syntax instead which will automatically close the file for you.
e.g.,
with open('baby1990.html', 'rU') as f:
for line in f:
print line
once this block is finished executing, the file is automatically closed for you, so you could execute this block repeatedly without explicitly closing the file yourself and read the file this way over again.

As the file object reads the file, it uses a pointer to keep track of where it is. If you read part of the file, then go back to it later it will pick up where you left off. If you read the whole file, and go back to the same file object, it will be like reading an empty file because the pointer is at the end of the file and there is nothing left to read. You can use file.tell() to see where in the file the pointer is and file.seek to set the pointer. For example:
>>> file = open('myfile.txt')
>>> file.tell()
0
>>> file.readline()
'one\n'
>>> file.tell()
4L
>>> file.readline()
'2\n'
>>> file.tell()
6L
>>> file.seek(4)
>>> file.readline()
'2\n'
Also, you should know that file.readlines() reads the whole file and stores it as a list. That's useful to know because you can replace:
for line in file.readlines():
#do stuff
file.seek(0)
for line in file.readlines():
#do more stuff
with:
lines = file.readlines()
for each_line in lines:
#do stuff
for each_line in lines:
#do more stuff
You can also iterate over a file, one line at a time, without holding the whole file in memory (this can be very useful for very large files) by doing:
for line in file:
#do stuff

The file object is a buffer. When you read from the buffer, that portion that you read is consumed (the read position is shifted forward). When you read through the entire file, the read position is at the end of the file (EOF), so it returns nothing because there is nothing left to read.
If you have to reset the read position on a file object for some reason, you can do:
f.seek(0)

Of course.
That is normal and sane behaviour.
Instead of closing and re-opening, you could rewind the file.

Copy file data using python [duplicate]

How do I append to a file instead of overwriting it?

Set the mode in open() to "a" (append) instead of "w" (write):
with open("test.txt", "a") as myfile:
myfile.write("appended text")
The documentation lists all the available modes.

You need to open the file in append mode, by setting "a" or "ab" as the mode. See open().
When you open with "a" mode, the write position will always be at the end of the file (an append). You can open with "a+" to allow reading, seek backwards and read (but all writes will still be at the end of the file!).
Example:
>>> with open('test1','wb') as f:
f.write('test')
>>> with open('test1','ab') as f:
f.write('koko')
>>> with open('test1','rb') as f:
f.read()
'testkoko'
Note: Using 'a' is not the same as opening with 'w' and seeking to the end of the file - consider what might happen if another program opened the file and started writing between the seek and the write. On some operating systems, opening the file with 'a' guarantees that all your following writes will be appended atomically to the end of the file (even as the file grows by other writes).
A few more details about how the "a" mode operates (tested on Linux only). Even if you seek back, every write will append to the end of the file:
>>> f = open('test','a+') # Not using 'with' just to simplify the example REPL session
>>> f.write('hi')
>>> f.seek(0)
>>> f.read()
'hi'
>>> f.seek(0)
>>> f.write('bye') # Will still append despite the seek(0)!
>>> f.seek(0)
>>> f.read()
'hibye'
In fact, the fopen manpage states:
Opening a file in append mode (a as the first character of mode)
causes all subsequent write operations to this stream to occur at
end-of-file, as if preceded the call:
fseek(stream, 0, SEEK_END);
Old simplified answer (not using with):
Example: (in a real program use with to close the file - see the documentation)
>>> open("test","wb").write("test")
>>> open("test","a+b").write("koko")
>>> open("test","rb").read()
'testkoko'

I always do this,
f = open('filename.txt', 'a')
f.write("stuff")
f.close()
It's simple, but very useful.

Python has many variations off of the main three modes, these three modes are:
'w' write text
'r' read text
'a' append text
So to append to a file it's as easy as:
f = open('filename.txt', 'a')
f.write('whatever you want to write here (in append mode) here.')
Then there are the modes that just make your code fewer lines:
'r+' read + write text
'w+' read + write text
'a+' append + read text
Finally, there are the modes of reading/writing in binary format:
'rb' read binary
'wb' write binary
'ab' append binary
'rb+' read + write binary
'wb+' read + write binary
'ab+' append + read binary

You probably want to pass "a" as the mode argument. See the docs for open().
with open("foo", "a") as f:
f.write("cool beans...")
There are other permutations of the mode argument for updating (+), truncating (w) and binary (b) mode but starting with just "a" is your best bet.

You can also do it with print instead of write:
with open('test.txt', 'a') as f:
print('appended text', file=f)
If test.txt doesn't exist, it will be created...

when we using this line open(filename, "a"), that a indicates the appending the file, that means allow to insert extra data to the existing file.
You can just use this following lines to append the text in your file
def FileSave(filename,content):
with open(filename, "a") as myfile:
myfile.write(content)
FileSave("test.txt","test1 \n")
FileSave("test.txt","test2 \n")

The 'a' parameter signifies append mode. If you don't want to use with open each time, you can easily write a function to do it for you:
def append(txt='\nFunction Successfully Executed', file):
with open(file, 'a') as f:
f.write(txt)
If you want to write somewhere else other than the end, you can use 'r+'†:
import os
with open(file, 'r+') as f:
f.seek(0, os.SEEK_END)
f.write("text to add")
Finally, the 'w+' parameter grants even more freedom. Specifically, it allows you to create the file if it doesn't exist, as well as empty the contents of a file that currently exists.
† Credit for this function goes to #Primusa

You can also open the file in r+ mode and then set the file position to the end of the file.
import os
with open('text.txt', 'r+') as f:
f.seek(0, os.SEEK_END)
f.write("text to add")
Opening the file in r+ mode will let you write to other file positions besides the end, while a and a+ force writing to the end.

if you want to append to a file
with open("test.txt", "a") as myfile:
myfile.write("append me")
We declared the variable myfile to open a file named test.txt. Open takes 2 arguments, the file that we want to open and a string that represents the kinds of permission or operation we want to do on the file
here is file mode options
Mode Description
'r' This is the default mode. It Opens file for reading.
'w' This Mode Opens file for writing.
If file does not exist, it creates a new file.
If file exists it truncates the file.
'x' Creates a new file. If file already exists, the operation fails.
'a' Open file in append mode.
If file does not exist, it creates a new file.
't' This is the default mode. It opens in text mode.
'b' This opens in binary mode.
'+' This will open a file for reading and writing (updating)

If multiple processes are writing to the file, you must use append mode or the data will be scrambled. Append mode will make the operating system put every write, at the end of the file irrespective of where the writer thinks his position in the file is. This is a common issue for multi-process services like nginx or apache where multiple instances of the same process, are writing to the same log
file. Consider what happens if you try to seek, then write:
Example does not work well with multiple processes:
f = open("logfile", "w"); f.seek(0, os.SEEK_END); f.write("data to write");
writer1: seek to end of file. position 1000 (for example)
writer2: seek to end of file. position 1000
writer2: write data at position 1000 end of file is now 1000 + length of data.
writer1: write data at position 1000 writer1's data overwrites writer2's data.
By using append mode, the operating system will place any write at the end of the file.
f = open("logfile", "a"); f.seek(0, os.SEEK_END); f.write("data to write");
Append most does not mean, "open file, go to end of the file once after opening it". It means, "open file, every write I do will be at the end of the file".
WARNING: For this to work you must write all your record in one shot, in one write call. If you split the data between multiple writes, other writers can and will get their writes in between yours and mangle your data.

Sometimes, beginners have this problem because they attempt to open and write to a file in a loop:
for item in my_data:
with open('results.txt', 'w') as f:
f.write(some_calculation(item))
The problem is that every time the file is opened for writing, it will be truncated (cleared out).
We can solve this by opening in append mode instead; but in cases like this, it will normally be better to solve the problem by inverting the logic. If the file is opened only once, then it won't get overwritten each time; and we can keep writing to it as long as it is open - we don't have to re-open it for each write (it would be pointless for Python to make things work that way, since it would add to the required code for no benefit).
Thus:
with open('results.txt', 'w') as f:
for item in my_data:
f.write(some_calculation(item))

The simplest way to append more text to the end of a file would be to use:
with open('/path/to/file', 'a+') as file:
file.write("Additions to file")
file.close()
The a+ in the open(...) statement instructs to open the file in append mode and allows read and write access.
It is also always good practice to use file.close() to close any files that you have opened once you are done using them.

Write to file and save to ftp with python 2.6

I'm trying to store a file I create on an ftp server.
I've been able to create the temp file and store it as an empty file, but I haven't been able to write any data to the file before storing it.
Here is the partially working code:
#Loggin to server.
ftp = FTP(Integrate.ftp_site)
ftp.login(paths[0], paths[1])
ftp.cwd(paths[3])
f = tempfile.SpooledTemporaryFile()
# Throws error.
f.write(bytes("hello", 'UTF-8'))
#No error, doesn't work.
#f.write("hello")
#Also, doesn't throw error, and doesn't write anything to the file.
# f.write("hello".encode('UTF-8'))
file_name = "test.txt"
ftp.storlines("Stor " + file_name, f)
#Done.
f.close()
ftp.quit()
What am I doing wrong?
Thanks

Seeking!
To know where to read or write in the file (or file-like object), Python keeps a pointer to a location in the file. The documentation simply calls it "the file's current position". So, if you have a filed with these lines in it:
hello world
how are you
You can read it with Python like in the following code. Note that the tell() function tells you the file's position.
>>> f = open('file.txt', 'r')
>>> f.tell()
0
>>> f.readline()
'hello world\n'
>>> f.tell()
12
Python is now twelve characters "into" the file. If you'd count the characters, that means it's right after the newline character (\n is a single character). Continuing to read from the file with readlines() or any other reading function will use this position to know where to start reading.
Writing to the file will also use and increment the position. This means that if, after writing to the file you read from the file, Python will start reading at the position it has saved (which is right after whatever you just wrote), not the beginning of the file.
The ftp.storlines() function uses the same readlines() function, which only starts reading at the file's position, so after whatever you wrote. You can solve this by seeking back to the start of the file before calling ftp.storlines(). Use f.seek(0) to reset the file position to the very start of the file.

How to confirm that a file object is empty? [Python]

in a py module, I write:
outFile = open(fileName, mode='w')
if A:
outFile.write(...)
if B:
outFile.write(...)
and in these lines, I didn't use flush or close method.
Then after these lines, I want to check whether this "outFile" object is empty or not. How can I do with it?

There are a few problems with your code.
You can't .write to a file that you opened with 'r'. You need to open(fileName, 'w').
If A or B then you've certainly written to the file, so it's not empty!
Barring those. you can get the length of a file with
os.stat(outFile.fileno())
EDIT: I'll explain what flush does. Python is often used to do quite large amounts of file reads and writes, which can be slow. It is thus tweaked to make them as fast as possible. One way that is does so is to "buffer" such writes and then do them all in one big block: when you write a small string, Python will remember it but won't actually write it to the file until it thinks it should.
This means that if you want to tell whether you have written data to the file by inspecting the file, you have to tell Python to write all the data it's remembering first, or else you might not see it. flush is the command to write all the buffered data.
Of course, if you ask Python whether it's written anything to the file, say by inspecting the position in the file (.tell()), then it will know about the buffering.

If you've already written to the file, you can use .tell() to check if the current file position is nonzero:
>>> handle = open('/tmp/file.txt', 'w')
>>> handle.write('foo')
>>> handle.tell()
3
This won't work if you .seek() back to the beginning of the file.

You can use os.stat to get file info:
import os
fileSize = os.stat(fileName).st_size

with open("filename.txt", "r+") as f:
if f.read():
# file isn't empty
f.write("something")
# uncomment this line if you want to delete everything else in the file
# f.truncate()
else:
# file is empty
f.write("somethingelse")
"r+" mode always you to read & write.
"with" will automatically close file

Why the second time I run "readlines" on the same file nothing is returned?

>>> f = open('/tmp/version.txt', 'r')
>>> f
<open file '/tmp/version.txt', mode 'r' at 0xb788e2e0>
>>> f.readlines()
['2.3.4\n']
>>> f.readlines()
[]
>>>
I've tried this in Python's interpreter. Why does this happen?

You need to seek to the beginning of the file. Use f.seek(0) to return to the begining:
>>> f = open('/tmp/version.txt', 'r')
>>> f
<open file '/tmp/version.txt', mode 'r' at 0xb788e2e0>
>>> f.readlines()
['2.3.4\n']
>>> f.seek(0)
>>> f.readlines()
['2.3.4\n']
>>>

Python keeps track of where you are in the file. When you're at the end, it doesn't automatically roll back over. Try f.seek(0).

The important part to understand that some of the other posters don't explicitly state is that files are read with a cursor that marks the current position in the file. So on the first readlines() call the cursor is at the beginning of your file, and is progressed all the way to the end of the file since all the files data was returned. On the second readlines call the cursor is at the end of the file, so when it reads to the end of the file, it doesn't move at all, and no data is returned. For educational purposes, you could write a quick bit of code that would open a file, read a few bytes or lines out, and then call readlines(), you will see that the output of the readlines() call begins where you left off with your previous reads, and continues until the end of the file.
The seek(0) call mentioned by other will allow you to reset the cursor at the beginning of the file to start over with the reads.

In addition to seeking to the beginning of the file, you can also store the value as something that you can reuse later if you just need them in memory. Something like this:
with open('tmp/version.txt', 'r') as f:
lines = f.readlines()
The with statement is new in 2.6 I believe, in prior versions you'd need to import it from future.